Bisimulation metrics for continuous Markov decision processes

SIAM Journal on Computing

Volumn 40, Issue 6, 2011, Pages 1662-1714

  Ferns, Norm ^a   Panangaden, Prakash ^b   Precup, Doina ^b  

^a McGill University ^*  (Australia)

^b MCGILL UNIVERSITY   (Canada)

Author keywords

Bisimulation; Continuous; Markov decision process; Metrics; Reinforcement learning

Indexed keywords

BEHAVIORAL RESEARCH; LINEAR PROGRAMMING; MARKOV PROCESSES; PRESSES (MACHINE TOOLS); ROBOT PROGRAMMING; ROBOTS; SAMPLING; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

BEHAVIORAL SIMILARITIES; BISIMULATIONS; CONTINUOUS; FINITE MARKOV DECISION PROCESS; MARKOV DECISION PROCESSES; METRICS; OPTIMAL VALUE FUNCTIONS; PROBABILISTIC TRANSITION SYSTEM;

REINFORCEMENT LEARNING;

EID: 84855578361     PISSN: 00975397     EISSN: None     Source Type: Journal    
DOI: 10.1137/10080484X     Document Type: Conference Paper

References (64)

1. N. ALON, S. BEN-DAVID, N. CESA-BIANCHI, AND D. HAUSSLER, Scale-sensitive dimensions, uniform convergence, and learnability, J. Assoc. Comput. Mach., 44(1997), pp. 615-631. (Pubitemid 127617707)
2. M. ANTHONY, Uniform Glivenko-Cantelli Theorems and Concentration of Measure in the Mathematical Modelling of Learning, Technical Report LSE-CDAM-2002-07, Centre for Discrete and Applicable Mathematics, London, UK, 2002; available online at http://www.maths.lse.ac.uk/Personal/martin/mresearch. html.
3. M. BERNARDO AND M. BRAVETTI, Performance measure sensitive congruences for Markovian process algebras, Theoret. Comput. Sci., 290(2003), pp. 117-160. (Pubitemid 35264420)
4. P. BILLINGSLEY, Convergence of Probability Measures, John Wiley, New York, 1968.
5. R. BLUTE, J. DESHARNAIS, A. EDALAT, AND P. PANANGADEN, Bisimulation for labelled Markov processes, in LICS '97: Proceedings of the 12th Annual IEEE Symposium on Logic in Computer Science, IEEE Computer Society, Washington, DC, 1997, pp. 149-159.
6. A. BOUCHARD-CÔTÉ, N. FERNS, P. PANANGADEN, AND D. PRECUP, An approximation algorithm for labelled Markov processes: Towards realistic approximation, in QEST '05: Proceedings of the Second International Conference on the Quantitative Evaluation of Systems, IEEE Computer Society, Washington, DC, 2005, pp. 54-61.
7. C. BOUTILIER, T. DEAN, AND S. HANKS, Decision-theoretic planning: Structural assumptions and computational leverage, J. Artificial Intell. Res., 11(1999), pp. 1-94. (Pubitemid 129628760)
8. C. BOUTILIER, R. DEARDEN, AND M. GOLDSZMIDT, Exploiting structure in policy construction, in Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, Chris Mellish, ed., Morgan Kaufmann, San Francisco, 1995, pp. 1104-1111.
9. C. BOUTILIER, R. DEARDEN, AND M. GOLDSZMIDT, Stochastic dynamic programming with factored representations, Artificial Intell., 121(2000), pp. 49-107.
10. K. L. CLARKSON, Nearest-neighbor searching and metric space dimensions, in Nearest-Neighbor Methods for Learning and Vision: Theory and Practice, G. Shakhnarovich, T. Darrell, and P. Indyk, eds., MIT Press, Cambridge, MA, 2006, pp. 15-59.
11. T. DEAN AND R. GIVAN, Model minimization in Markov decision processes, in Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI) /Innovative Applications of Artificial Intelligence IAAI, AAAI Press, Palo Alto, CA, 1997, pp. 106-111.
12. T. DEAN, R. GIVAN, AND S. LEACH, Model reduction techniques for computing approximately optimal solutions for Markov decision processes, in Proceedings of the 13th Annual Conference on Uncertainty in Artificial Intelligence (UAI-97), Morgan Kaufmann, San Francisco, 1997, pp. 124-131.
13. J. DEDECKER, C. PRIEUR, AND P. RAYNAUD DE FITTE, Parametrized Kantorovich-Rubinstein Theorem and Application to the Coupling of Random Variables, preprint, arXiv:math/041005v2[math. PR], 2004.
14. J. DESHARNAIS, Label led Markov Processes, Ph. D. thesis, McGill University, Montreal, Canada, 2000.
15. J. DESHARNAIS, A. EDALAT, AND P. PANANGADEN, Bisimulation for labeled Markov processes, Informat. Comput., 179(2002), pp. 163-193.
16. J. DESHARNAIS, V. GUPTA, R. JAGADEESAN, AND P. PANANGADEN, Metrics for labeled Markov systems, in CONCUR '99: Proceedings of the 10th International Conference on Concurrency Theory, Springer-Verlag, London, 1999, pp. 258-273.
17. J. DESHARNAIS, V. GUPTA, R. JAGADEESAN, AND P. PANANGADEN, Metrics for label led Markov processes, Theoret. Comput. Sci., 318(2004), pp. 323-354.
18. J. DESHARNAIS, R. JAGADEESAN, V. GUPTA, AND P. PANANGADEN, The metric analogue of weak bisimulation for probabilistic processes, in LICS '02: Proceedings of the 17th Annual IEEE Symposium on Logic in Computer Science (Copenhagen, Denmark), IEEE Computer Society, Washington, DC, 2002, pp. 413-422.
19. R. M. DUDLEY, Notes on Empirical Processes, Lecture notes for a course given at Aarhus University, Aarhus, Denmark, 1999.
20. R. M. DUDLEY, Real Analysis and Probability, Cambridge University Press, Cambridge, UK, 2002.
21. R. M. DUDLEY, E. GINÉ, AND J. ZINN, Uniform and universal Glivenko-Cantelli classes, J. Theoret. Probab., 4(1991), pp. 485-510.
22. A. EDALAT, When Scott is weak on the top, Math. Struct. Comput. Sci., 7(1997), pp. 401-417.
23. E. EVEN-DAR AND Y. MANSOUR, Approximate equivalence of Markov decision processes, in Proceedings of the 16th Annual Conference on Computational Learning Theory and 7th Kernel Workshop, (COLT/Kernel, Washington, DC), Lecture Notes in Comput. Sci. 2777, Springer, Berlin, New York, 2003, pp. 581-594. (Pubitemid 37053231)
24. N. FERNS, Metrics for Markov Decision Processes, Master's thesis, McGill University, Montreal, Canada, 2003.
25. N. FERNS, State-Similarity Metrics for Continuous Markov Decision Processes, Ph. D. Thesis, McGill University, Montreal, Canada, 2008.
26. N. FERNS, P. S. CASTRO, D. PRECUP, AND P. PANANGADEN, Methods for computing state similarity in Markov decision processes, in Proceedings of the 22nd Annual Conference on Uncertainty in Artificial Intelligence (UAI-06), AUAI Press, Arlington, VA, 2006, pp. 174-181.
27. N. FERNS, P. PANANGADEN, AND D. PRECUP, Metrics for finite Markov decision processes, Proceedings of the 20th Annual Conference on Uncertainty in Artificial Intelligence (UAI-04), AUAI Press, Arlington, VA, 2004, pp. 162-169.
28. N. FERNS, P. PANANGADEN, AND D. PRECUP, Metrics for Markov decision processes with infinite state spaces, in Proceedings of the 21st Annual Conference on Uncertainty in Artificial Intelligence (UAI-05), AUAI Press, Arlington, VA, 2005, pp. 201-208.
29. G. B. FOLLAND, Real Analysis: Modern Techniques and Their Applications, 2nd ed., Wiley-Interscience, New York, 1999.
30. A. FRANGIONI AND A. MANCA, A computational study of cost reoptimization for min-cost flow problems, INFORMS J. Comput., 18(2006), pp. 61-70. (Pubitemid 43515946)
31. A. L. GIBBS AND F. E. SU, On choosing and bounding probability metrics, Internat. Statist. Rev., 70(2002), pp. 419-435.
32. R. GIVAN, T. DEAN, AND M. GREIG, Equivalence notions and model minimization in Markov decision processes, Artificial Intell., 147(2003), pp. 163-223.
33. T. F. GONZALEZ, Clustering to minimize the maximum intercluster distance, Theoret. Comput. Sci., 38(1985), pp. 293-306.
34. J. H. GOTO, M. E. LEWIS, AND M. L. PUTERMAN, Coffee, tea, or...?: A Markov decision process model for airline meal provisioning, Trans. Sci., 38(2004), pp. 107-118.
35. J. GOUBAULT-LARRECQ, Simulation hemi-metrics between infinite-state stochastic games, in Proceedings of the Theory and Practice of Software, 11th International Conference on Foundations of Software Science and Computational Structures (FOSSACS'08/ETAPS'08), Springer-Verlag, Berlin, Heidelberg, 2008, pp. 50-65.
36. M. HENNESSY AND R. MILNER, Algebraic laws for nondeterminism and concurrency, J. Assoc. Comput. Mach., 32(1985), pp. 137-161.
37. L. PACK KAELBLING, M. L. LITTMAN, AND A. R. CASSANDRA, Planning and acting in partially observable stochastic domains, Artificial Intell., 101(1998), pp. 99-134. (Pubitemid 128387390)
38. J. G. KEMENY AND J. L. SNELL, Finite Markov Chains, Van Nostrand, New York, 1960.
39. D. KOZEN, A probabilistic PDL, in STOC '83: Proceedings of the Fifteenth Annual ACM Symposium on Theory of Computing, ACM, New York, 1983, pp. 291-297.
40. T. LANE AND L. PACK KAELBLING, Approaches to macro decompositions of large Markov decision process planning problems, in Proceedings of the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference on Mobile Robots XVI, vol. 4573, H. M. Gage and D. W. Choset, eds., SPIE, Newton, MA, 2002, pp. 104-113.
41. K. G. LARSEN AND A. SKOU, Bisimulation through probabilistic testing, Inform. Comput., 94(1991), pp. 1-28. (Pubitemid 21694897)
42. L. LI, T. J. WALSH, AND M. L. LITTMAN, Towards a unified theory of state abstraction for MDPs, in AI & MATH '06: Proceedings of the Ninth International Symposium on Artificial Intelligence and Mathematics, 2006, pp. 531-539.
43. M. LIKHACHEV, G. GORDON, AND S. THRUN, Planning for Markov decision processes with sparse stochasticity, in Advances in Neural Information Processing Systems 17, L. K. Saul, Y. Weiss, and L. Bottou, eds., MIT Press, Cambridge, MA, 2005, pp. 785-792.
44. R. MILNER, A calculus of Communicating Systems, Lecture Notes in Comput. Sci. 92, Springer-Verlag, New York, 1980.
45. R. MILNER, Communication and Concurrency, Prentice-Hall International, Englewood Cliffs, NJ, 1989.
46. A. MÜLLER, How does the value function of a Markov decision process depend on the transition probabilities?, Math. Oper. Res., 22(1997), pp. 872-885. (Pubitemid 127653365)
47. J. ORLIN, A faster strongly polynomial minimum cost flow algorithm, in STOC '88: Proceedings of the Twentieth Annual ACM Symposium on Theory of Computing, ACM, New York, 1988, pp. 377-387.
48. R. ORTNER, Pseudometrics for state aggregation in average reward Markov decision processes, in Proceedings of Algorithmic Learning Theory: 18th International Conference (ALT), Sendai, Japan, Lecture Notes in Comput. Sci. 4754, M. Hutter, R. A. Servedio, and E. Takimoto, eds., Springer-Verlag, Berlin, New York, 2007, pp. 373-387.
49. P. PANANGADEN, Labelled Markov Processes, Imperial College Press, 2009.
50. D. PARK, Concurrency and automata on infinite sequences, in Proceedings of the 5th GI-Conference on Theoretical Computer Science, Springer-Verlag, London, 1981, pp. 167-183.
51. K. R. PARTHASARATHY, Probability Measures on Metric Spaces, Academic Press, New York, 1967.
52. C. PHILLIPS, Knowledge Transfer in Markov Decision Processes, Final report for Canadian Distributed Mentors Project scholarship, http://www.cra.org/Activities/craw-archive/cdmp/awards/2006/Phillips/summary. pdf, (2006).
53. M. L. PUTERMAN, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley, New York, 1994.
54. S. T. RACHEV AND L. RUESCHENDORF, Mass Transportation Problems. Vol. 1: Theory. Vol. 2: Applications, Springer Ser. Statis. Probab. Appl., Springer, New York, 1998.
55. W. RUDIN, Principles of Mathematical Analysis, 3rd ed., McGraw-Hill, New York, 1976.
56. R. S. SUTTON AND A. G. BARTO, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
57. F. VAN BREUGEL, B. SHARMA, AND J. WORRELL, Approximating a behavioural pseudometric without discount for probabilistic systems, in Proceedings of Foundations of Software Science and Computational Structures, 10th International Conference (FOSSACS), Held as Part of the Joint European Conferences on Theory and Practice of Software (ETAPS Braga, Portugal), Lecture Notes in Comput. Sci. 4423, Helmut Seidl, ed., Springer, Berlin, New York, 2007, pp. 123-137. (Pubitemid 350259700)
58. F. VAN BREUGEL AND J. WORRELL, Towards quantitative verification of probabilistic transition systems, in ICALP '01: Proceedings of the 28th International Colloquium on Automata, Languages and Programming, Springer-Verlag, London, UK, 2001, pp. 421-432. (Pubitemid 33300818)
59. F. VAN BREUGEL AND J. WORRELL, An algorithm for quantitative verification of probabilistic transition systems, in CONCUR '01: Proceedings of the 12th International Conference on Concurrency Theory, Springer-Verlag, London, UK, 2001, pp. 336-350. (Pubitemid 33326666)
60. C. VILLANI, Topics in Optimal Transportation, Grad. Stud. Math. 58, AMS, Providence, RI, 2003.
61. J. VYGEN, On dual minimum cost flow algorithms (extended abstract), in STOC '00: Proceedings of the Thirty-Second Annual ACM Symposium on Theory of Computing, ACM, New York, 2000, pp. 117-125.
62. J. WEBSTER, Finite approximation of measure and integration, Ann. Pure Appl. Logic, 137(2006), pp. 439-449. (Pubitemid 41614400)
63. G. WINSKEL, The Formal Semantics of Programming Languages, Found. Comput. Sci. Ser., MIT Press, Cambridge, MA, 1993.
64. L. ZHANG, H. HERMANNS, F. EISENBRAND, AND D. N. JANSEN, Flow faster: Efficient decision algorithms for probabilistic simulations, in Proceedings of the 13th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'07), Springer-Verlag, Berlin, Heidelberg, 2007, pp. 155-169. (Pubitemid 350259292)