Decision-Theoretic Planning: Structural Assumptions and Computational Leverage

Journal of Artificial Intelligence Research

Volumn 11, Issue , 1999, Pages 1-94

  Boutilier, Craig ^a   Dean, Thomas ^b   Hanks, Steve ^c  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b Brown Univerity   (United States)

^c University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0346942368     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.575     Document Type: Article

References (158)

1. Allen, J., Hendler, J., & Tate, A. (Eds.). (1990). Readings in Planning. Morgan-Kaufmann, San Mateo.
2. Aström, K. J. (1965). Optimal control of Markov decision processes with incomplete state estimation. J. Math. Anal. Appl., 10, 174-205.
3. Bacchus, F., Boutilier, C., & Grove, A. (1996). Rewarding behaviors. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pp. 1160-1167 Portland, OR.
4. Bacchus, F., Boutilier, C., & Grove, A. (1997). Structured solution methods for non-Markovian decision processes. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 112-117 Providence, RI.
5. Bacchus, F., & Kabanza, F. (1995). Using temporal logic to control search in a forward chaining planner. In Proceedings of the Third European Workshop on Planning (EWSP'95) Assisi, Italy. Available via the URL ftp://logos.uwaterloo.ca:/pub/tlplan/tlplan.ps.Z.
6. Bacchus, F., & Teh, Y. W. (1998). Making forward chaining relevant. In Proceedings of the Fourth International Conference on AI Planning Systems, pp. 54-61 Pittsburgh, PA.
7. Bahar, R. L, Frohm, E. A., Gaona, C. M., Hachtel, G. D., Macii, E., Pardo, A., & Somenzi, F. (1993). Algebraic decision diagrams and their applications. In International Conference on Camputer-Aided Design, pp. 188-191. IEEE.
8. Baker, A. B. (1991). Nonmonotonic reasoning in the framework of the situation calculus. Artificial Intelligence, 49, 5-23.
9. Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence, 72(1-2), 81-138.
10. Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ.
11. Bertsekas, D. P., & Castanon, D. A. (1989). Adaptive aggregation for infinite horizon dynamic programming. IEEE Transactions on Automatic Control, 34(6), 589-598.
12. Bertsekas, D. P. (1987). Dynamic Programming. Prentice-Hall, Englewood Cliffs, NJ.
13. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic Programming. Athena, Belmont, MA.
14. Blackwell, D. (1962). Discrete dynamic programming. Annals of Mathematical Statistics, 33, 719-726.
15. Blum, A. L., & Furst, M. L. (1995). Fast planning through graph analysis. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1636-1642 Montreal, Canada.
16. Bonet, B., & Geffner, H. (1998). Learning sorting and decision trees with POMDPs. In Proceedings of the Fifteenth International Conference on Machine Learning, pp. 73-81 Madison, WI.
17. Bonet, B., Loerincs, G., & Geffner, H. (1997). A robust and fast action selection mechanism. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 714-719 Providence, RI.
18. Boutilier, C. (1997). Correlated action effects in decision theoretic regression. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pp. 30-37 Providence, RI.
19. Boutilier, C., Brafman, R. I., & Geib, C. (1997). Prioritized goal decomposition of Markov decision processes: Toward a synthesis of classical and decision theoretic planning. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 1156-1162 Nagoya, Japan.
20. Boutilier, C., Brafman, R. I., & Geib, C. (1998). Structured reachability analysis for Markov decision processes. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 24-32 Madison, WI.
21. Boutilier, C., & Dearden, R. (1994). Using abstractions for decision-theoretic planning with time constraints. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pp. 1016-1022 Seattle, WA.
22. Boutilier, C., & Dearden, R. (1996). Approximating value trees in structured dynamic programming. In Proceedings of the Thirteenth International Conference on Machine Learning, pp. 54-62 Bari, Italy.
23. Boutilier, C., Dearden, R., & Goldszmidt, M. (1995). Exploiting structure in policy construction. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1104-1111 Montreal, Canada.
24. Boutilier, C., Dearden, R., & Goldszmidt, M. (1999). Stochastic dynamic programming with factored representations, (manuscript).
25. Boutilier, C., Friedman, N., Goldszmidt, M., & Koller, D. (1996). Context-specific independence in Bayesian networks. In Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence, pp. 115-123 Portland, OR.
26. Boutilier, C., & Goldszmidt, M. (1996). The frame problem and Bayesian network action representations. In Proceedings of the Eleventh Biennial Canadian Conference on Artificial Intelligence, pp. 69-83 Toronto.
27. Boutilier, C., & Poole, D. (1996). Computing optimal policies for partially observable decision processes using compact representations. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pp. 1168-1175 Portland, OR.
28. Boutilier, C., & Puterman, M. L. (1995). Process-oriented planning and average-reward optimality. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1096-1103 Montreal, Canada.
29. Brafman, R. I. (1997). A heuristic variable-grid solution method for POMDPs. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 727-733 Providence, RI.
30. Bryant, R. E. (1986). Graph-based algorithms for boolean function manipulation. IEEE Transactions on Computers, C-35(8), 677-691.
31. Bylander, T. (1994). The computational complexity of propositional STRIPS planning. Artificial Intelligence, 69, 161-204.
32. Caines, P. E. (1988). Linear stochastic systems. Wiley, New York.
33. Cassandra, A. R., Kaelbling, L. P., & Littman, M. L. (1994). Acting optimally in partially observable stochastic domains. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pp. 1023-1028 Seattle, WA.
34. Cassandra, A. R., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: A simple, fast, exact method for pomdps. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pp. 54-61 Providence, RI.
35. Chapman, D. (1987). Planning for conjunctive goals. Artificial Intelligence, 32(3), 333-377.
36. Chapman, D., & Kaelbling, L. P. (1991). Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. In Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, pp. 726-731 Sydney, Australia.
37. Dantzig, G., & Wolfe, P. (1960). Decomposition principle for dynamic programs. Operations Research, 8(1), 101-111.
38. Dean, T., Allen, J., & Aloimonos, Y. (1995). Artificial Intelligence: Theory and Practice. Benjamin Cummings.
39. Dean, T., & Givan, R. (1997). Model minimization in Markov decision processes. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 106-111 Providence, RI. AAAI.
40. Dean, T., Givan, R., & Kim, K.-E. (1998). Solving planning problems with large state and action spaces. In Proceedings of the Fourth International Conference on AI Planning Systems, pp. 102-110 Pittsburgh, PA.
41. Dean, T., Givan, R., & Leach, S. (1997). Model reduction techniques for computing approximately optimal solutions for Markov decision processes. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pp. 124-131 Providence, RI.
42. Dean, T., Kaelbling, L., Kirman, J., & Nicholson, A. (1993). Planning with deadlines in stochastic domains. In Proceedings of the Eleventh National Conference on Artificial Intelligence, pp. 574-579.
43. Dean, T., Kaelbling, L., Kirman, J., & Nicholson, A. (1995). Planning under time constraints in stochastic domains. Artificial Intelligence, 70(1-2), 3-74.
44. Dean, T., & Kanazawa, K. (1989). A model for reasoning about persistence and causation. Computational Intelligence, 5(3), 142-150.
45. Dean, T., & Lin, S.-H. (1995). Decomposition techniques for planning in stochastic domains. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1121-1127.
46. Dean, T., & Wellman, M. (1991). Planning and Control. Morgan Kaufmann, San Mateo, California.
47. Dearden, R., & Boutilier, C. (1994). Integrating planning and execution in stochastic domains. In Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pp. 162-169 Washington, DC.
48. Dearden, R., & Boutilier, C. (1997). Abstraction and approximate decision theoretic planning. Artificial Intelligence, 89, 219-283.
49. Dechter, R. (1996). Bucket elimination: A unifying framework for probabilistic inference. In Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence, pp. 211-219 Portland, OR.
50. Dechter, R. (1997). Mini-buckets: A general scheme for generating approximations in automated reasoning in probabilistic inference. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 1297-1302 Nagoya, Japan.
51. D'Epenoux, F. (1963). Sur un problème de production et de stockage dans l'aléatoire. Management Science, 10, 98-108.
52. Dietterich, T. G., & Flann, N. S. (1995). Explanation-based learning and reinforcement learning: A unified approach. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 176-184 Lake Tahoe, NV.
53. Draper, D., Hanks, S., & Weld, D. (1994a). A probabilistic model of action for least-commitment planning with information gathering. In Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pp. 178-186 Washington, DC.
54. Draper, D., Hanks, S., & Weld, D. (1994b). Probabilistic planning with information gathering and contingent execution. In Proceedings of the Second International Conference on AI Planning Systems, pp. 31-36.
55. Etzioni, O., Hanks, S., Weld, D., Draper, D., Lesh, N., & Williamson, M. (1992). An approach to planning with incomplete information. In Proceedings of the Third International Conference on Principles of Knowledge Representation and Reasoning, pp. 115-125 Boston, MA.
56. Fikes, R., Hart, P., & Nilsson, N. (1972). Learning and executing generalized robot plans. Artificial Intelligence, 3, 251-288.
57. Fikes, R., & Nilsson, N. J. (1971). STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2, 189-208.
58. Finger, J. (1986). Exploiting Constraints in Design Synthesis. Ph.D. thesis, Stanford University, Stanford.
59. Floyd, R. W. (1962). Algorithm 97 (shortest path). Communications of the ACM, 5(6), 345.
60. Fox, B. L., & Landi, D. M. (1968). An algorithm for identifying the ergodic subchains and transient states of a stochastic matrix. Communications of the ACM, 2, 619-621.
61. French, S. (1986). Decision Theory. Halsted Press, New York.
62. Geiger, D., & Heckerman, D. (1991). Advances in probabilistic reasoning. In Proceedings of the Seventh Conference on Uncertainty in Artificial Intelligence, pp. 118-126 Los Angeles, CA.
63. Givan, R., & Dean, T. (1997). Model minimization, regression, and prepositional STRIPS planning. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 1163-1168 Nagoya, Japan.
64. Givan, R., Leach, S., & Dean, T. (1997). Bounded-parameter Markov decision processes. In Proceedings of the Fourth European Conference on Planning (ECP'97), pp. 234-246 Toulouse, France.
65. Goldman, R. P., & Boddy, M. S. (1994). Representing uncertainty in simple planners. In Proceedings of the Fourth International Conference on Principles of Knowledge Representation and Reasoning, pp. 238-245 Bonn, Germany.
66. Haddawy, P., & Doan, A. (1994). Abstracting probabilistic actions. In Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pp. 270-277 Washington, DC.
67. Haddawy, P., & Hanks, S. (1998). Utility Models for Goal-Directed Decision-Theoretic Planners. Computational Intelligence, 14(3).
68. Haddawy, P., & Suwandi, M. (1994). Decision-theoretic refinement planning using inheritence abstraction. In Proceedings of the Second International Conference on AI Planning Systems, pp. 266-271 Chicago, IL.
69. Hanks, S. (1990). Projecting plans for uncertain worlds. Ph.D. thesis 756, Yale University, Department of Computer Science, New Haven, CT.
70. Hanks, S., & McDermott, D. V. (1994). Modeling a dynamic and uncertain world I: Symbolic and probabilistic reasoning about change. Artificial Intelligence, 66(1), 1-55.
71. Hanks, S., Russell, S., & Wellman, M. (Eds.). (1994). Decision Theoretic Planning: Proceedings of the AAAI Spring Symposium. AAAI Press, Menlo Park.
72. Hansen, E. A., & Zilberstein, S. (1998). Heuristic search in cyclic AND/OR graphs. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp. 412-418 Madison, WI.
73. Hauskrecht, M. (1997). A heuristic variable-grid solution method for POMDPs. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 734-739 Providence, RI.
74. Hauskrecht, M. (1998). Planning and Control in Stochastic Domains with Imperfect Information. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge.
75. Hauskrecht, M., Meuleau, N., Kaelbling, L. P., Dean, T., & Boutilier, C. (1998). Hierarchical solution of Markov decision processes using macro-actions. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 220-229 Madison, WI.
76. Hoey, J., St-Aubin, R., Hu, A., & Boutilier, C. (1999). SPUDD: Stochastic planning using decision diagrams. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence Stockholm. To appear.
77. Howard, R. A. (1960). Dynamic Programming and Markov Processes. MIT Press, Cambridge, Massachusetts.
78. Howard, R. A., & Matheson, J. E. (1984). Influence diagrams. In Howard, R. A., & Matheson, J. E. (Eds.), The Principles and Applications of Decision Analysis. Strategic Decisions Group, Menlo Park, CA.
79. Kambhampati, S. (1997). Refinement planning as a unifying framework for plan synthesis. AI Magazine, Summer 1997, 67-97.
80. Kearns, M., Mansour, Y., & Ng, A. Y. (1999). A sparse sampling algorithm for near-optimal planning in large markov decision processes. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence Stockholm. To appear.
81. Keeney, R. L., & Raiffa, H. (1976). Decisions with Multiple Objectives: Preferences and Value Tradeoffs. John Wiley and Sons, New York.
82. Kjaerulff, U. (1992). A computational scheme for reasoning in dynamic probabilistic networks. In Proceedings of the Eighth Conference on Uncertainty in AI, pp. 121-129 Stanford.
83. Knoblock, C. A. (1993). Generating Abstraction Hierarchies: An Automated Approach to Reducing Search in Planning. Kluwer, Boston.
84. Knoblock, C. A., Tenenberg, J. D., & Yang, Q. (1991). Characterizing abstraction hierarchies for planning. In Proceedings of the Ninth National Conference on Artificial Intelligence, pp. 692-697 Anaheim, CA.
85. Koenig, S. (1991). Optimal probabilistic and decision-theoretic planning using Markovian decision theory. M.sc. thesis UCB/CSD-92-685, University of California at Berkeley, Computer Science Department.
86. Koenig, S., & Simmons, R. (1995). Real-time search in nondeterministic domains. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1660-1667 Montreal, Canada.
87. Korf, R. (1985). Macro-operators: A weak method for learning. Artificial Intelligence, 26, 35-77.
88. Korf, R. E. (1990). Real-time heuristic search. Artificial Intelligence, 42, 189-211.
89. Kushmerick, N., Hanks, S., & Weld, D. (1995). An Algorithm for Probabilistic Planning. Artificial Intelligence, 76, 239-286.
90. Kushner, H. J., & Chen, C.-H. (1974). Decomposition of systems governed by Markov chains. IEEE Transactions on Automatic Control, 19(5), 501-507.
91. Lee, D., & Yannakakis, M. (1992). Online minimization of transition systems. In Proceedings of 24th Annual ACM Symposium on the Theory of Computing, pp. 264-274 Victoria, BC.
92. Lin, F., & Reiter, R. (1994). State constraints revisited. Journal of Logic and Computation, 4(5), 655-678.
93. Lin, S.-H. (1997). Exploiting Structure for Planning and Control. Ph.D. thesis, Department of Computer Science, Brown University.
94. Lin, S.-H., & Dean, T. (1995). Generating optimal policies for high-level plans with conditional branches and loops. In Proceedings of the Third European Workshop on Planning (EWSP'95), pp. 187-200.
95. Littman, M. L. (1997). Probabilistic propositional planning: Representations and complexity. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 748-754 Providence, RI.
96. Littman, M. L., Dean, T. L., & Kaelbling, L. P. (1995). On the complexity of solving Markov decision problems. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp. 394-402 Montreal, Canada.
97. Littman, M. L. (1996). Algorithms for sequential decision making. Ph.D. thesis CS-96-09, Brown University, Department of Computer Science, Providence, RI.
98. Lovejoy, W. S. (1991a). Computationally feasible bounds for partially observed Markov decision processes. Operations Research, 39(1), 162-175.
99. Lovejoy, W. S. (1991b). A survey of algorithmic methods for partially observed Markov decision processes. Annals of Operations Research, 28, 47-66.
100. Luenberger, D. G. (1973). Introduction to Linear and Nonlinear Programming. Addison-Wesley, Reading, Massachusetts.
101. Luenberger, D. G. (1979). Introduction to Dynamic Systems: Theory, Models and Applications. Wiley, New York.
102. Madani, O., Condon, A., & Hanks, S. (1999). On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems. In Proceedings of the Sixteenth National Conference on Artificial Intelligence Orlando, FL. To appear.
103. Mahadevan, S. (1994). To discount or not to discount in reinforcement learning: A case study in comparing R-learning and Q-learning. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 164-172 New Brunswick, NJ.
104. McAllester, D., & Rosenblitt, D. (1991). Systematic nonlinear planning. In Proceedings of the Ninth National Conference on Artificial Intelligence, pp. 634-639 Anaheim, CA.
105. McCallum, R. A. (1995). Instance-based utile distinctions for reinforcement learning with hidden state. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 387-395 Lake Tahoe, Nevada.
106. McCarthy, J., & Hayes, P. J. (1969). Some philosophical problems from the standpoint of artificial intelligence. Machine Intelligence, 4, 463-502.
107. Meuleau, N., Hauskrecht, M., Kim, K., Peshkin, L., Kaelbling, L., Dean, T., & Boutilier, C. (1998). Solving very large weakly coupled Markov decision processes. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp. 165-172 Madison, WI.
108. Moore, A. W., & Atkeson, C. G. (1995). The parti-game algorithm for variable resolution reinforcement learning in multidimensional state spaces. Machine Learning, 21, 199-234.
109. Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov chain decision processes. Mathematics of Operations Research, 12(3), 441-450.
110. Parr, R. (1998). Flexible decomposition algorithms for weakly coupled Markov decision processes. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 422-430 Madison, WI.
111. Parr, R., & Russell, S. (1995). Approximating optimal policies for partially observable stochastic domains. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1088-1094 Montreal.
112. Parr, R., & Russell, S. (1998). Reinforcement learning with hierarchies of machines. In Jordan, M., Kearns, M., & Solla, S. (Eds.), Advances in Neural Information Processing Systems 10, pp. 1043-1049. MIT Press, Cambridge.
113. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo.
114. Pednault, E. (1989). ADL: Exploring the middle ground between STRIPS and the situation calculus. In Proceedings of the First International Conference on Principles of Knowledge Representation and Reasoning, pp. 324-332 Toronto, Canada.
115. Penberthy, J. S., & Weld, D. S. (1992). UCPOP: A sound, complete, partial order planner for ADL. In Proceedings of the Third International Conference on Principles of Knowledge Representation and Reasoning, pp. 103-114 Boston, MA.
116. Peot, M., & Smith, D. (1992). Conditional Nonlinear Planning. In Proceedings of the First International Conference on AI Planning Systems, pp. 189-197 College Park, MD.
117. Perez, M. A., & Carbonell, J. G. (1994). Control knowledge to improve plan quality. In Proceedings of the Second International Conference on AI Planning Systems, pp. 323-328 Chicago, IL.
118. Poole, D. (1995). Exploiting the rule structure for decision making within the independent choice logic. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp. 454-463 Montreal, Canada.
119. Poole, D. (1997a). The independent choice logic for modelling multiple agents under uncertainty. Artificial Intelligence, 94(1-2), 7-56.
120. Poole, D. (1997b). Probabilistic partial evaluation: Exploiting rule structure in probabilistic inference. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 1284-1291 Nagoya, Japan.
121. Poole, D. (1998). Context-specific approximation in probabilistic inference. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 447-454 Madison, WI.
122. Precup, D., Sutton, R. S., & Singh, S. (1998). Theoretical results on reinforcement learning with temporally abstract behaviors. In Proceedings of the Tenth European Conference on Machine Learning, pp. 382-393 Chemnitz, Germany.
123. Pryor, L., & Collins, G. (1993). CASSANDRA: Planning for contingencies. Technical report 41, Northwestern University, The Institute for the Learning Sciences.
124. Puterman, M. L. (1994). Markov Decision Processes. John Wiley & Sons, New York.
125. Puterman, M. L., & Shin, M. (1978). Modified policy iteration algorithms for discounted Markov decision problems. Management Science, 24, 1127-1137.
126. Ross, K. W., & Varadarajan, R. (1991). Multichain Markov decision processes with a sample-path constraint: A decomposition approach. Mathematics of Operations Research, 16(1), 195-207.
127. Russell, S., & Norvig, P. (1995). Artificial Intelligence: A Modern Approach. Prentice Hall, Englewood Cliffs, NJ.
128. Sacerdoti, E. D. (1974). Planning in a hierarchy of abstraction spaces. Artificial Intelligence, 5, 115-135.
129. Sacerdoti, E. D. (1975). The nonlinear nature of plans. In Proceedings of the Fourth International Joint Conference on Artificial Intelligence, pp. 206-214.
130. Schoppers, M. J. (1987). Universal plans for reactive robots in unpredictable environments. In Proceedings of the Tenth International Joint Conference on Artificial Intelligence, pp. 1039-1046 Milan, Italy.
131. Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards. In Proceedings of the Tenth International Conference on Machine Learning, pp. 298-305 Amherst, MA.
132. Schweitzer, P. L., Puterman, M. L., & Kindle, K. W. (1985). Iterative aggregation-disaggregation procedures for discounted semi-Markov reward processes. Operations Research, 33, 589-605.
133. Shachter, R. D. (1986). Evaluating influence diagrams. Operations Research, 55(6), 871-882.
134. Shimony, S. E. (1993). The role of relevance in explanation I: Irrelevance as statistical independence. International Journal of Approximate Reasoning, 8(4), 281-324.
135. Simmons, R., & Koenig, S. (1995). Probabilistic robot navigation in partially observable environments. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1080-1087 Montreal, Canada.
136. Singh, S. P., & Cohn, D. (1998). How to dynamically merge Markov decision processes. In Advances in Neural Information Processing Systems 10, pp. 1057-1063. MIT Press, Cambridge.
137. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Reinforcement learning with soft state aggregation. In Hanson, S. J., Cowan, J. D., & Giles, C. L. (Eds.), Advances in Neural Information Processing Systems 7. Morgan-Kaufmann, San Mateo.
138. Smallwood, R. D., & Sondik, E. J. (1973). The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21, 1071-1088.
139. Smith, D., & Peot, M. (1993). Postponing threats in partial-order planning. In Proceedings of the Eleventh National Conference on Artificial Intelligence, pp. 500-506 Washington, DC.
140. Sondik, E. J. (1978). The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Operations Research, 26, 282-304.
141. Stone, P., & Veloso, M. (1999). Team-partitioned, opaque-transition reinforcement learning. In Asada, M. (Ed.), RoboCup-98: Robot Soccer World Cup II. Springer Verlag, Berlin.
142. Sutton, R. S. (1995). TD models: Modeling the world at a mixture of time scales. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 531-539 Lake Tahoe, NV.
143. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
144. Tash, J., & Russell, S. (1994). Control strategies for a stochastic planner. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pp. 1079-1085 Seattle, WA.
145. Tatman, J. A., & Shachter, R. D. (1990). Dynamic programming and influence diagrams. IEEE Transactions on Systems, Man, and Cybernetics, 20(2), 365-379.
146. Tesauro, G. J. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 6, 215-219.
147. Thrun, S., Fox, D., & Burgard, W. (1998). A probabilistic approach to concurrent mapping and localization for mobile robots. Machine Learning, 31, 29-53.
148. Thrun, S., & Schwartz, A. (1995). Finding structure in reinforcement learning. In Tesauro, G., Touretzky, D., & Leen, T. (Eds.), Advances in Neural Information Processing Systems 7 Cambridge, MA. MIT Press.
149. Warren, D. (1976). Generating conditional plans and programs. In Proceedings of AISB Summer Conference, pp. 344-354 University of Edinburgh.
150. Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.
151. Weld, D. S. (1994). An introduction to least commitment planning. AI Magazine, Winter 1994, 27-61.
152. White III, C. C., & Scherer, W. T. (1989). Solutions procedures for partially observed Markov decision processes. Operations Research, 57(5), 791-797.
153. Williamson, M. (1996). A value-directed approach to planning. Ph.D. thesis 96-06-03, University of Washington, Department of Computer Science and Engineering.
154. Williamson, M., & Hanks, S. (1994). Optimal planning with a goal-directed utility model. In Proceedings of the Second International Conference on AI Planning Systems, pp. 176-180 Chicago, IL.
155. Winston, P. H. (1992). Artificial Intelligence, Third Edition. Addison-Wesley, Reading, Massachusetts.
156. Yang, Q. (1998). Intelligent Planning : A Decomposition and Abstraction Based Approach. Springer Verlag.
157. Zhang, N. L., & Liu, W. (1997). A model approximation scheme for planning in partially observable stochastic domains. Journal of Artificial Intelligence Research, 7, 199-230.
158. Zhang, N. L., & Poole, D. (1996). Exploiting causal independence in Bayesian network inference. Journal of Artificial Intelligence Research, 5, 301-328.