Practical solution techniques for first-order MDPs

Artificial Intelligence

Volumn 173, Issue 5-6, 2009, Pages 748-788

  Sanner, Scott ^a   Boutilier, Craig ^b  

^a CSIRO   (Australia)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

First order logic; MDPs; Planning

Indexed keywords

FORMAL LOGIC; LINEARIZATION; SPECIFICATIONS;

APPROXIMATE LINEAR PROGRAMMING; DECISION-THEORETIC; FIRST ORDERS; FIRST-ORDER LOGIC; INTERNATIONAL PLANNING COMPETITIONS; MARKOV DECISION PROCESS; MDPS; PLANNING PROBLEMS; PRACTICAL SOLUTIONS; PROBABILISTIC PLANNING; PROBLEM SPECIFICATIONS; PROOF OF CONCEPTS; SOLUTION APPROACHES; SPACE AND TIME COMPLEXITY;

PLANNING;

EID: 60549103706     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2008.11.003     Document Type: Article

References (91)

1. D. Andre, S. Russell, Programmable reinforcement learning agents, in: Advances in Neural Information Processing Systems (NIPS-01), vol. 13, 2001, pp. 78-85
2. F. Bacchus, J.Y. Halpern, H.J. Levesque, Reasoning about noisy sensors in the situation calculus, in: International Joint Conference on Artificial Intelligence (IJCAI-95), Montreal, 1995, pp. 1933-1940
3. Bacchus F., and Kabanza F. Using temporal logics to express search control knowledge for planning. Artificial Intelligence 116 1-2 (2000) 123-191
4. R.I. Bahar, E. Frohm, C. Gaona, G. Hachtel, E. Macii, A. Pardo, F. Somenzi, Algebraic decision diagrams and their applications, in: IEEE/ACM International Conference on CAD, 1993, pp. 428-432
5. A.G. Barto, S.J. Bradtke, S.P. Singh, Learning to act using real-time dynamic programming, Tech. Rep. UM-CS-1993-002, U. Mass. Amherst, 1993
6. Bellman R.E. Dynamic Programming (1957), Princeton University Press, Princeton, NJ
7. Bertsekas D.P. Dynamic Programming (1987), Prentice Hall, Englewood Cliffs, NJ
8. Bertsekas D.P., and Tsitsiklis J.N. Neuro-Dynamic Programming (1996), Athena Scientific, Belmont, MA
9. A.L. Blum, M.L. Furst, Fast planning through graph analysis, in: IJCAI 95, Montreal, 1995, pp. 1636-1642
10. B. Bonet, H. Geffner, mGPT: A probabilistic planner based on heuristic search, in: Online Proceedings for The Probabilistic Planning Track of IPC-04: http://www.cs.rutgers.edu/~mlittman/topics/ipc04-pt/proceedings/, 2004
11. C. Boutilier, R.I. Brafman, C. Geib, Prioritized goal decomposition of Markov decision processes: Toward a synthesis of classical and decision theoretic planning, in: International Joint Conference on Artificial Intelligence (IJCAI-97). Nagoya, 1997, pp. 1156-1162
12. Boutilier C., Dean T., and Hanks S. Decision-theoretic planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research (JAIR) 11 (1999) 1-94
13. C. Boutilier, N. Friedman, M. Goldszmidt, D. Koller, Context-specific independence in Bayesian networks, in: Uncertainty in Artificial Intelligence (UAI-96), Portland, OR, 1996, pp. 115-123
14. C. Boutilier, R. Reiter, B. Price, Symbolic dynamic programming for first-order MDPs, in: International Joint Conference on Artificial Intelligence (IJCAI-01), Seattle, 2001, pp. 690-697
15. C. Boutilier, R. Reiter, M. Soutchanski, S. Thrun, Decision-theoretic, high-level agent programming in the situation calculus, in: AAAI-00, Austin, TX, 2000, pp. 355-362
16. Brachman R., and Levesque H. Knowledge Representation and Reasoning (2004), Morgan Kaufmann Publishers Inc., San Francisco, CA
17. O. Buffet, D. Aberdeen, The factored policy gradient planner (ipc-06 version), in: Proceedings of the Fifth International Planning Competition, 2006
18. Buntine W. Generalized subsumption and its application to induction and redundancy. Artificial Intelligence 36 (1988) 375-399
19. de Farias D., and Roy B.V. The linear programming approach to approximate dynamic programming. Operations Research 51 6 (2003) 850-865
20. R. de Salvo Braz, E. Amir, D. Roth, Lifted first-order probabilistic inference, in: 19th International Joint Conference on Artificial Intelligence (IJCAI-2005), Edinburgh, UK, 2005, pp. 1319-1325
21. R. de Salvo Braz, E. Amir, D. Roth, MPE and partial inversion in lifted probabilistic variable elimination, in: National Conference on Artificial Intelligence (AAAI-06), Boston, USA, 2006
22. Dearden R., and Boutilier C. Abstraction and approximate decision-theoretic planning. Artificial Intelligence 89 12 (1997) 219-283
23. Dechter R. Bucket elimination: A unifying framework for reasoning. Artificial Intelligence 113 (1999) 41-85
24. K. Driessens, S. Dzeroski, Integrating experimentation and guidance in relational reinforcement learning, in: International Conference on Machine Learning (ICML), 2002, pp. 115-122
25. Dzeroski S., DeRaedt L., and Driessens K. Relational reinforcement learning. Machine Learning Journal (MLJ) 43 (2001) 7-52
26. A. Fern, S. Yoon, R. Givan, Approximate policy iteration with a policy language bias, in: Advances in Neural Information Processing Systems 16 (NIPS-03), December 2003
27. A. Fern, S. Yoon, R. Givan, Learning domain-specific control knowledge from random walks, in: International Conference on Planning and Scheduling (ICAPS-04), June 2004, pp. 191-199
28. A. Ferrein, C. Fritz, G. Lakemeyer, Extending DTGolog with options, in: 18th International Joint Conference on Artificial Intelligence (IJCAI-2003), Acapulco, Mexico, 2003, pp. 144-151
29. Fikes R.E., and Nilsson N.J. STRIPS: A new approach to the application of theorem proving to problem solving. AI Journal 2 (1971) 189-208
30. N.H. Gardiol, L.P. Kaelbling, Envelope-based planning in relational MDPs, in: Advances in Neural Information Processing Systems 16 (NIPS-03), Vancouver, CA, 2004, pp. 1040-1046
31. Gartner T., Driessens K., and Ramon J. Graph kernels and Gaussian processes for relational reinforcement learning. Machine Learning Journal (MLJ) 64 (2006) 91-119
32. A. Gerevini, B. Bonet, B. Givan (Eds.), Online Proceedings for The Fifth International Planning Competition IPC-05: http://www.ldc.usb.ve/bonet/ipc5/docs/ipc-2006-booklet.pdf.gz, Lake District, UK, 2006
33. C. Gretton, S. Thiebaux, Exploiting first-order regression in inductive policy selection, in: Uncertainty in Artificial Intelligence (UAI-04), Banff, Canada, 2004, pp. 217-225
34. C. Guestrin, M. Hauskrecht, B. Kveton, Solving factored MDPs with continuous and discrete variables, in: 20th Conference on Uncertainty in Artificial Intelligence, 2004, pp. 235-242
35. C. Guestrin, D. Koller, C. Gearhart, N. Kanodia, Generalizing plans to new environments in relational MDPs, in: 18th International Joint Conference on Artificial Intelligence (IJCAI-2003), Acapulco, Mexico, 2003, pp. 1003-1010
36. Guestrin C., Koller D., Parr R., and Venktaraman S. Efficient solution methods for factored MDPs. Journal of Artificial Intelligence Research (JAIR) 19 (2002) 399-468
37. M. Hauskrecht, B. Kveton, Linear program approximations for factored continuous-state Markov decision processes, in: Advances in Neural Information Processing Systems 16, 2004, pp. 895-902
38. J. Hoey, R. St-Aubin, A. Hu, C. Boutilier, SPUDD: Stochastic planning using decision diagrams, in: Uncertainty in Artificial Intelligence (UAI-99), Stockholm, 1999, pp. 279-288
39. Hoffmann J., and Nebel B. The FF planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research (JAIR) 14 (2001) 253-302
40. Hölldobler S., Karabaev E., and Skvortsova O. FluCaP: A heuristic search planner for first-order mdps. Journal of Artificial Intelligence Research (JAIR) 27 (2006) 419-439
41. Howard R.A. Dynamic Programming and Markov Processes (1960), MIT Press
42. E. Karabaev, O. Skvortsova, A heuristic search algorithm for solving first-order MDPs, in: Uncertainty in Artificial Intelligence (UAI-05), Edinburgh, Scotland, 2005, pp. 292-299
43. K. Kersting, M. van Otterlo, L. de Raedt, Bellman goes relational, in: International Conference on Machine Learning (ICML-04), ACM Press, 2004, pp. 465-472
44. Khardon R. Learning action strategies for planning domains. Artificial Intelligence 113 1-2 (1999) 125-148
45. Khardon R. Learning to take actions. Machine Learning 35 1 (1999) 57-90
46. D. Koller, R. Parr, Computing factored value functions for policies in structured MDPs, in: International Joint Conference on Artificial Intelligence (IJCAI-99), Stockholm, 1999, pp. 1332-1339
47. D. Koller, R. Parr, Policy iteration for factored MDPs, in: Uncertainty in Artificial Intelligence (UAI-00), Stockholm, 2000, pp. 326-334
48. Levesque H.J., Reiter R., Lespérance Y., Lin F., and Scherl R. GOLOG: A logic programming language for dynamic domains. Journal of Logic Programming 31 1-3 (1997) 59-83
49. I. Little, Paragraph: A Graphplan-based probabilistic planner, in: Proceedings of the Fifth International Planning Competition, 2006
50. M.L. Littman, H.L.S. Younes (Eds.), Online Proceedings for The Probabilistic Planning Track of IPC-04: http://www.cs.rutgers.edu/mlittman/topics/ipc04-pt/proceedings/. Vancouver, Canada, 2004
51. S. Mahadevan, Samuel meets Amarel: Automating value function approximation using global state space analysis, in: National Conference on Artificial Intelligence (AAAI-05), Pittsburgh, 2005, pp. 1000-1005
52. McCarthy J. Situations, actions and causal laws, Tech. rep., Stanford University, 1963, reprinted. In: Minsky M. (Ed). Semantic Information Processing (1968), MIT Press, Cambridge, MA 410-417
53. N. Meuleau, M. Hauskrecht, K.-E. Kim, L. Peshkin, L.P. Kaelbling, T. Dean, C. Boutilier, Solving very large weakly coupled Markov decision processes, in: National Conference on Artificial Intelligence (AAAI-98), Madison, WI, 1998, pp. 165-172
54. B. Motik, Reasoning in description logics using resolution and deductive databases, Ph.D. thesis, Univesität Karlsruhe (TH), Karlsruhe, Germany, January 2006
55. Ng A.Y., Harada D., and Russell S. Policy invariance under reward transformations: theory and application to reward shaping. Proc. 16th International Conf. on Machine Learning (1999), Morgan Kaufmann, San Francisco, CA 278-287
56. Parr R., and Russell S. Reinforcement learning with hierarchies of machines. In: Jordan M.M.K., and Solla S. (Eds). Advances in Neural Information Processing Systems 10 (1998), MIT Press, Cambridge, MA 1043-1049
57. R. Patrascu, P. Poupart, D. Schuurmans, C. Boutilier, C. Guestrin, Greedy linear value-approximation for factored Markov decision processes, in: National Conference on Artificial Intelligence (AAAI-02), Edmonton, 2002, pp. 285-291
58. E.P.D. Pednault, ADL: Exploring the middle ground between STRIPS and the situation calculus, in: KR, 1989, pp. 324-332
59. Poole D. The independent choice logic for modelling multiple agents under uncertainty. Artificial Intelligence 94 1-2 (1997) 7-56
60. D. Poole, First-order probabilistic inference, in: IJCAI, 2003, pp. 985-991
61. P. Poupart, C. Boutilier, R. Patrascu, D. Schuurmans, Piecewise linear value function approximation for factored MDPs, in: National Conference on Artificial Intelligence (AAAI-02), Edmonton, 2002, pp. 292-299
62. Puterman M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming (1994), Wiley, New York
63. Reiter R. The frame problem in the situation calculus: A simple solution (sometimes) and a completeness result for goal regression. In: Lifschitz V. (Ed). Artificial Intelligence and Mathematical Theory of Computation (Papers in Honor of John McCarthy) (1991), Academic Press, San Diego, CA 359-380
64. Reiter R. Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems (2001), MIT Press
65. Riazanov A., and Voronkov A. The design and implementation of vampire. AI Communications 15 2 (2002) 91-110
66. J. Rintanen, Expressive equivalence of formalisms for planning with sensing, in: 13th International Conference on Automated Planning and Scheduling, 2003, pp. 185-194
67. S. Sanner, First-order decision-theoretic planning in structured relational environments, Ph.D. thesis, University of Toronto, Toronto, ON, Canada, March 2008
68. S. Sanner, C. Boutilier, Approximate linear programming for first-order MDPs, in: Uncertainty in Artificial Intelligence (UAI-05), Edinburgh, Scotland, 2005, pp. 509-517
69. S. Sanner, C. Boutilier, Practical linear evaluation techniques for first-order MDPs, in: Uncertainty in Artificial Intelligence (UAI-06), Boston, MA, 2006
70. S. Sanner, C. Boutilier, Approximate solution techniques for factored first-order MDPs, in: 17th International Conference on Automated Planning and Scheduling (ICAPS-07), 2007, pp. 288-295
71. D. Schuurmans, R. Patrascu, Direct value approximation for factored MDPs, in: Advances in Neural Information Processing 14 (NIPS-01), Vancouver, 2001, pp. 1579-1586
72. Schweitzer P., and Seidmann A. Generalized polynomial approximations in Markovian decision processes. Journal of Mathematical Analysis and Applications 110 (1985) 568-582
73. Shapley L.S. Stochastic games. Proceedings of the National Academy of Sciences 39 (1953) 327-332
74. Singh S.P., and Cohn D. How to dynamically merge Markov decision processes. Advances in Neural Information Processing Systems (NIPS-98) (1998), MIT Press, Cambridge, MA 1057-1063
75. R. St-Aubin, J. Hoey, C. Boutilier, APRICODD: Approximate policy construction using decision diagrams, in: Advances in Neural Information Processing 13 (NIPS-00), Denver, 2000, pp. 1089-1095
76. F. Teichteil, P. Fabiani, Symbolic stochastic focused dynamic programming with decision diagrams, in: Proceedings of the Fifth International Planning Competition, 2006
77. Thiebaux S., Gretton C., Slaney J., Price D., and Kabanza F. Decision-theoretic planning with non-Markovian rewards. Journal of Artificial Intelligence Research 25 (January 2006) 17-74
78. Tsitsiklis J.N., and Van Roy B. Feature-based methods for large scale dynamic programming. Machine Learning 22 (1996) 59-94
79. M. Veloso, Learning by analogical reasoning in general problem solving, Ph.D. thesis, Carnegie Mellon University, August 1992
80. C. Wang, S. Joshi, R. Khardon, First order decision diagrams for relational MDPs, in: Twentieth International Joint Conference on Artificial Intelligence (IJCAI-07), Hyderabad, India, 2007, pp. 1095-1100
81. Wang C., Joshi S., and Khardon R. First order decision diagrams for relational MDPs. Journal of Artificial Intelligence Research (JAIR) 31 (2008) 431-472
82. C. Wang, R. Khardon, Policy iteration for relational MDPs, in: Uncertainty in Artificial Intelligence (UAI-07), Vancouver, Canada, 2007
83. J. Wu, R. Givan, Discovering relational domain features for probabilistic planning, in: 17th International Conference on Automated Planning and Scheduling (ICAPS 2007), 2007, pp. 344-351
84. S. Yoon, A. Fern, R. Givan, Inductive policy selection for first-order Markov decision processes, in: Uncertainty in Artificial Intelligence (UAI-02), Edmonton, 2002, pp. 569-576
85. S. Yoon, A. Fern, R. Givan, Learning reactive policies for probabilistic planning domains, in: Online Proceedings for The Probabilistic Planning Track of IPC-04: http://www.cs.rutgers.edu/mlittman/topics/ipc04-pt/proceedings/, 2004
86. S. Yoon, A. Fern, R. Givan, Learning measures of progress for planning domains, in: 20th National Conference on Artificial Intelligence, July 2005, pp. 1217-1222
87. Yoon S., Fern A., and Givan R. Approximate policy iteration with a policy language bias: Learning to solve relational Markov decision processes. Journal of Artificial Intelligence Research (JAIR) 25 (2006) 85-118
88. S. Yoon, A. Fern, R. Givan, FF-Replan: A baseline for probabilistic planning, in: 17th International Conference on Automated Planning and Scheduling (ICAPS-07), 2007, pp. 352-359
89. Younes H.L.S., Littman M.L., Weissman D., and Asmuth J. The first probabilistic track of the international planning competition. Journal of Artificial Intelligence Research (JAIR) 24 (2005) 851-887
90. N.L. Zhang, D. Poole, A simple approach to Bayesian network computations, in: Proc. of the Tenth Canadian Conference on Artificial Intelligence, 1994, pp. 171-178
91. Zhang N.L., and Poole D. Exploiting causal independence in Bayesian network inference. Journal of Artificial Intelligence Research (JAIR) 5 (1996) 301-328