Planning and acting in partially observable stochastic domains

Artificial Intelligence

Volumn 101, Issue 1-2, 1998, Pages 99-134

  Kaelbling, Leslie Pack ^a   Littman, Michael L ^b   Cassandra, Anthony R ^c  

^a Brown Univerity   (United States)

^b Duke University   (United States)

^c Microlectron and Comp Technology   (United States)

Author keywords

Partially observable Markov decision processes; Planning; Uncertainty

Indexed keywords

ALGORITHMS; MARKOV PROCESSES; OBSERVABILITY; PROBLEM SOLVING;

FINITE MEMORY CONTROLLERS; PARTIALLY OBSERVABLE STOCHASTIC DOMAINS;

DECISION THEORY;

EID: 0032073263     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/s0004-3702(98)00023-x     Document Type: Article

References (70)

1. K.J. Aström, Optimal control of Markov decision processes with incomplete state estimation, J. Math. Anal. Appl. 10 (1995) 174-205.
2. F. Bacchus, C. Boutilier and A. Grove, Rewarding behaviors, in: Proceedings Thirteenth National Conference on Artificial Intelligence (AAAI-96), Portland, OR, AAAI Press/MIT Press, Menlo Park, CA, 1996, pp. 1160-1167.
3. D.P. Bertsekas, Dynamic Programming and Optimal Control, Vols. 1 and 2, Athena Scientific, Belmont, MA, 1995.
4. A.L. Blum and M.L. Furst, Fast planning through planning graph analysis, Artificial Intelligence 90 (1-2) (1997) 279-298.
5. J. Blythe, Planning with external events, in: Proceedings Tenth Conference on Uncertainty in Artificial Intelligence (UAI-94), Seattle, WA, 1994, pp. 94-101.
6. C. Boutilier and D. Poole, Computing optimal policies for partially observable decision processes using compact representations, in: Proceedings Thirteenth National Conference on Artificial Intelligence (AAAI-96), Portland, OR, AAAI Press/MIT Press, Menlo Park, CA, 1996, pp. 1168-1175.
7. A. Cassandra, M.L. Littman and N.L. Zhang, Incremental Pruning: a simple, fast, exact method for partially observable Markov decision processes, in: Proceedings Thirteenth Annual Conference on Uncertainty in Artificial Intelligence (UAI-97), Morgan Kaufmann, San Francisco, CA, 1997, pp. 54-61.
8. A.R. Cassandra, L.P. Kaelbling and M.L. Littman, Acting optimally in partially observable stochastic domains, in: Proceedings Twelfth National Conference on Artificial Intelligence (AAAI-94), Seattle, WA, 1994, pp. 1023-1028.
9. A.R. Cassandra, Exact and approximate algorithms for partially observable Markov decision problems, Ph.D. Thesis, Department of Computer Science, Brown University, Providence, RI, 1998.
10. H.-T. Cheng, Algorithms for partially observable Markov decision processes, Ph.D. Thesis, University of British Columbia, Vancouver, BC, 1988.
11. L. Chrisman, Reinforcement learning with perceptual aliasing: The perceptual distinctions approach, in: Proceedings Tenth National Conference on Artificial Intelligence (AAAI-92), San Jose, CA, AAAI Press, San Jose, CA, 1992, pp. 183-188.
12. A. Condon, The complexity of stochastic games, Inform. and Comput. 96 (2) (1992) 203-224.
13. T. Dean, L.P. Kaelbling, J. Kirman and A. Nicholson, Planning under time constraints in stochastic domains, Artificial Intelligence 76 (1-2) (1995) 35-74.
14. D. Draper, S. Hanks and D. Weld, Probabilistic planning with information gathering and contingent execution, Technical Report 93-12-04, University of Washington, Seattle, WA, 1993.
15. M. Drummond and J. Bresina, Anytime synthetic projection: maximizing the probability of goal satisfaction, in: Proceedings Eighth National Conference on Artificial Intelligence (AAAI-90), Boston, MA, Morgan Kaufmann, San Francisco, CA, 1990, pp. 138-144.
16. J.N. Eagle, The optimal search for a moving target when the search path is constrained, Oper. Res. 32 (5) (1984) 1107-1115.
17. E. Fernández-Gaucherand, A. Arapostathis and S.I. Marcus, On the average cost optimality equation and the structure of optimal policies for partially observable Markov processes, Ann. Oper. Res. 29 (1991) 471-512.
18. R.P. Goldman and M.S. Boddy, Conditional linear planning, in: K. Hammond (Ed.), The Second International Conference on Artificial Intelligence Planning Systems, AAAI Press/MIT Press, Menlo Park, CA, 1994, pp. 80-85.
19. R.P. Goldman and M.S. Boddy, Epsilon-safe planning, in: Proceedings 10th Conference on Uncertainty in Artificial Intelligence (UAI-94), Seattle, WA, 1994, pp. 253-261.
20. R.P. Goldman and M.S. Boddy, Representing uncertainty in simple planners, in: Proceedings 4th International Conference on Principles of Knowledge Representation and Reasoning (KR-94), Bonn, Germany, 1994, pp. 238-245.
21. P. Haddawy and S. Hanks, Utility models for goal-directed decision-theoretic planners, Technical Report 93-06-04, Department of Computer Science and Engineering, University of Washington, 1993.
22. E.A. Hansen, Cost-effective sensing during plan execution, in: Proceedings Twelfth National Conference on Artificial Intelligence (AAAI-94), Seattle, WA, AAAI Press/MIT Press, Menlo Park, CA, 1994, pp. 1029-1035.
23. E.A. Hansen, An improved policy iteration algorithm for partially observable MDPs, in: Advances in Neural Information Processing Systems 10 (1998).
24. R.A. Howard, Dynamic Programming and Markov Processes, MIT Press, Cambridge, MA, 1960.
25. R.A. Howard, Information value theory, IEEE Trans. Systems Science and Cybernetics SSC-2 (1) (1966) 22-26.
26. R.E. Kalman, A new approach to linear filtering and prediction problems, Trans. American Society of Mechanical Engineers, Journal of Basic Engineering 82 (1960) 35-45.
27. S. Koenig, Optimal probabilistic and decision-theoretic planning using Markovian decision theory, Technical Report UCB/CSD 92/685, Berkeley, CA, 1992.
28. S. Koenig and R.G. Simmons, Risk-sensitive planning with probabilistic decision graphs, in: Proceedings 4th International Conference on Principles of Knowledge Representation and Reasoning (KR-94), Bonn, Germany, 1994, pp. 363-373.
29. J.R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, MIT Press, Cambridge, MA, 1992.
30. N. Kushmerick, S. Hanks and D.S. Weld, An algorithm for probabilistic planning, Artificial Intelligence 76 (1-2) (1995) 239-286.
31. S.-H. Lin and T. Dean, Generating optimal policies for high-level plans with conditional branches and loops, in: Proceedings Third European Workshop on Planning (1995) 205-218.
32. M.L. Littman, Memoryless policies: theoretical limitations and practical results, in: D. Cliff, P. Husbands, J.-A. Meyer and S.W. Wilson (Eds.), From Animals to Animats 3: Proceedings Third International Conference on Simulation of Adaptive Behavior, MIT Press, Cambridge, MA, 1994.
33. M.L. Littman, A.R. Cassandra and L.P. Kaelbling, Learning policies for partially observable environments: scaling up, in: A. Prieditis and S. Russell (Eds.), Proceedings Twelfth International Conference on Machine Learning, Morgan Kaufmann, San Francisco, CA, 1995, pp. 362-370. Reprinted in: M.H. Huhns and M.P. Singh (Eds.), Readings in Agents, Morgan Kaufmann, San Francisco, CA, 1998.
34. M.L. Littman, A.R. Cassandra and L.P. Kaelbling, Learning policies for partially observable environments: scaling up, in: A. Prieditis and S. Russell (Eds.), Proceedings Twelfth International Conference on Machine Learning, Morgan Kaufmann, San Francisco, CA, 1995, pp. 362-370. Reprinted in: M.H. Huhns and M.P. Singh (Eds.), Readings in Agents, Morgan Kaufmann, San Francisco, CA, 1998.
35. M.L. Littman, A.R. Cassandra and L.P. Kaelbling, Efficient dynamic-programming updates in partially observable Markov decision processes, Technical Report CS-95-19, Brown University, Providence, RI, 1996.
36. M.L. Littman, Algorithms for sequential decision making, Ph.D. Thesis, Department of Computer Science, Brown University, 1996; also Technical Report CS-96-09.
37. W.S. Lovejoy, A survey of algorithmic methods for partially observable Markov decision processes, Ann. Oper. Res. 28 (1) (1991) 47-65.
38. S.M. Majercik and M.L. Littman, MAXPLAN: a new approach to probabilistic planning, Technical Report CS-1998-01, Department of Computer Science, Duke University, Durham, NC, 1998; submitted for review.
39. T.M. Mansell, A method for planning given uncertain and incomplete information, in: Proceedings 9th Conference on Uncertainty in Artificial Intelligence (UAI-93), Morgan Kaufmann, San Mateo, CA, 1993, pp. 350-358.
40. D. McAllester and D. Rosenblitt, Systematic nonlinear planning, in: Proceedings 9th National Conference on Artificial Intelligence (AAAI-91), Anaheim, CA, 1991, pp. 634-639.
41. R.A. McCallum, Overcoming incomplete perception with utile distinction memory, in: Proceedings Tenth International Conference on Machine Learning, Morgan Kaufmann, Amherst, MA, 1993, pp. 190-196.
42. R.A. McCallum, Instance-based utile distinctions for reinforcement learning with hidden state, in: Proceedings Twelfth International Conference on Machine Learning, Morgan Kaufmann, San Francisco, CA, 1995, pp. 387-395.
43. G.E. Monahan, A survey of partially observable Markov decision processes: theory, models, and algorithms, Management Science 28 (1) (1982) 1-16.
44. R.C. Moore, A formal theory of knowledge and action, in: J.R. Hobbs and R.C. Moore (Eds.), Formal Theories of the Commonsense World, Ablex Publishing, Norwood, NJ, 1985, pp. 319-358.
45. L. Morgenstern, Knowledge preconditions for actions and plans, in: Proceedings 10th International Joint Conference on Artificial Intelligence (IJCAI-87), Milan, Italy, 1987, pp. 867-874.
46. J.S. Penberthy and D. Weld, UCPOP: a sound, complete, partial order planner for ADL, in: Proceedings Third International Conference on Principles of Knowledge Representation and Reasoning (KR-92), Cambridge, MA, 1992, pp. 103-114.
47. M.A. Peot and D.E. Smith, Conditional nonlinear planning, in: Proceedings First International Conference on Artificial Intelligence Planning Systems, 1992, pp. 189-197.
48. L.K. Platzman, A feasible computational approach to infinite-horizon partially-observed Markov decision problems, Technical Report, Georgia Institute of Technology, Atlanta, GA, 1981.
49. L. Pryor and G. Collins, Planning for contingencies: a decision-based approach, J. Artif. Intell. Res. 4 (1996) 287-339.
50. M.L. Puterman, Markov Decision Processes - Discrete Stochastic Dynamic Programming, Wiley, New York, NY, 1994.
51. L.R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Proc. IEEE 77 (2) (1989) 257-286.
52. K. Sawaki and A. Ichikawa, Optimal control for partially observable Markov decision processes over an infinite horizon, J. Oper. Res. Soc. Japan 21 (1) (1978) 1-14.
53. R.B. Scherl and H.J. Levesque, The frame problem and knowledge-producing actions, in: Proceedings 11th National Conference on Artificial Intelligence (AAAI-93), Washington, DC, 1993, pp. 689-697.
54. M.J. Schoppers, Universal plans for reactive robots in unpredictable environments, in: Proceedings Tenth International Joint Conference on Artificial Intelligence (IJCAI-87), Milan, Italy, 1987, pp. 1039-1046.
55. A. Schrijver, Theory of Linear and Integer Programming, Wiley-Interscience, New York, NY, 1986.
56. S.P. Singh, T. Jaakkola and M.I. Jordan, Model-free reinforcement learning for non-Markovian decision problems, in: Proceedings Eleventh International Conference on Machine Learning, Morgan Kaufmann, San Francisco, CA, 1994, pp. 284-292.
57. R.D. Smallwood and E.J. Sondik, The optimal control of partially observable Markov processes over a finite horizon, Oper. Res. 21 (1973) 1071-1088.
58. D.E. Smith and M. Williamson, Representation and evaluation of plans with loops, Working Notes for the 1995 Stanford Spring Symposium on Extended Theories of Action, 1995.
59. E. Sondik, The optimal control of partially observable Markov processes, Ph.D. Thesis, Stanford University, 1971.
60. E.J. Sondik, The optimal control of partially observable Markov processes over the infinite horizon: discounted costs, Oper. Res. 26 (2) (1978) 282-304.
61. A. Stolcke and S. Omohundro, Hidden Markov model induction by Bayesian model merging, in: S.J. Hanson, J.D. Cowan and C.L. Giles (Eds.), Advances in Neural Information Processing Systems 5, Morgan Kaufmann, San Mateo, CA, 1993, pp. 11-18.
62. J. Tash and S. Russell, Control strategies for a stochastic planner, in: Proceedings 12th National Conference on Artificial Intelligence (AAAI-94), Seattle, WA, 1994, pp. 1079-1085.
63. P. Tseng, Solving H-horizon, stationary Markov decision problems in time proportional to log(H), Oper. Res. Lett. 9 (5) (1990) 287-297.
64. C.C. White and D. Harrington, Application of Jensen's inequality for adaptive suboptimal design, J. Optim. Theory Appl. 32 (1) (1980) 89-99.
65. C.C. White III, Partially observed Markov decision processes: a survey, Ann. Oper. Res. 32 (1991).
66. C.C. White III and W.T. Scherer, Solution procedures for partially observed Markov decision processes, Oper. Res. 37 (5) (1989) 791-797.
67. R.J. Williams and L.C. Baird III, Tight performance bounds on greedy policies based on imperfect value functions, Technical Report NU-CCS-93-14, Northeastern University, College of Computer Science, Boston, MA, 1993.
68. N.L. Zhang and W. Liu, Planning in stochastic domains: problem characteristics and approximation, Technical Report HKUST-CS96-31, Department of Computer Science, Hong Kong University of Science and Technology, 1996.
69. J. Zhao and J.H. Schmidhuber, Incremental self-improvement for life-time multi-agent reinforcement learning, in: P. Maes, M.J. Mataric, J.-A. Meyer, J. Pollack and S.W. Wilson (Eds.), From Animals to Animats: Proceedings Fourth International Conference on Simulation of Adaptive Behavior, MIT Press, Cambridge, MA, 1996, pp. 516-525.
70. U. Zwick and M. Paterson, The complexity of mean payoff games on graphs, Theoret. Comput. Sci. 158 (1-2) (1996) 343-359.