Value-Function Approximations for Partially Observable Markov Decision Processes

Journal of Artificial Intelligence Research

Volumn 13, Issue , 2000, Pages 33-94

  Hauskrecht, Milos ^a  

^a Brown Univerity   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0001770240     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.678     Document Type: Article

References (77)

1. Astrom, K. J. (1965). Optimal control of Markov decision processes with incomplete state estimation. Journal of Mathematical Analysis and Applications, 10, 174-205.
2. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 30-37.
3. Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence, 72, 81-138.
4. Bellman, R. E. (1957). Dynamic programming. Princeton University Press, Princeton, NJ.
5. Bertsekas, D. P. (1994). A counter-example to temporal differences learning. Neural Computation, 7, 270-279.
6. Bertsekas, D. P. (1995). Dynamic programming and optimal control. Athena Scientific.
7. Bonet, B., & Geffner, H. (1998). Learning sorting and classification with POMDPs. In Proceedings of the Fifteenth International Conference on Machine Learning.
8. Boutilier, C., Dean, T., & Hanks, S. (1999). Decision-theoretic planning: Structural assumptions and computational leverage. Artificial Intelligence, 11, 1-94.
9. Boutilier, C., & Poole, D. (1996). Exploiting structure in policy construction. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pp. 1168-1175.
10. Boyan, J. A., & Moore, A. A. (1995). Generalization in reinforcement learning: safely approximating the value function. In Advances in Neural Information Processing Systems 7. MIT Press.
11. Boyen, X., & Koller, D. (1998). Tractable inference for complex stochastic processes. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 33-42.
12. Boyen, X., & Koller, D. (1999). Exploiting the architecture of dynamic systems. In Proceedings of the Sixteenth National Conference on Artificial Intelligence, pp. 313-320.
13. Brafman, R. I. (1997). A heuristic variable grid solution method for POMDPs. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pp. 727-233.
14. Burago, D., Rougemont, M. D., & Slissenko, A. (1996). On the complexity of partially observed Markov decision processes. Theoretical Computer Science, 157, 161-183.
15. Cassandra, A. R. (1998). Exact and approximate algorithms for partially observable Markov decision processes. Ph.D. thesis, Brown University.
16. Cassandra, A. R., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: a simple, fast, exact algorithm for partially observable Markov decision processes. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pp. 54-61.
17. Castañon, D. (1997). Approximate dynamic programming for sensor management. In Proceedings of Conference on Decision and Control.
18. Cheng, H.-T. (1988). Algorithms for partially observable Markov decision processes. Ph.D. thesis, University of British Columbia.
19. Condon, A. (1992). The complexity of stochastic games. Information and Computation, 96, 203-224.
20. Dean, T., & Kanazawa, K. (1989). A model for reasoning about persistence and causation. Computational Intelligence, 5, 142-150.
21. Dearden, R., & Boutilier, C. (1997). Abstraction and approximate decision theoretic planning. Artificial Intelligence, 89, 219-283.
22. Doucet, A. (1998). On sequential simulation-based methods for Bayesian filtering. Tech. rep. CUED/F-INFENG/TR 310, Department of Engineering, Cambridge University.
23. Drake, A. (1962). Observation of a Markov process through a noisy channel. Ph.D. thesis, Massachusetts Institute of Technology.
24. Draper, D., Hanks, S., & Weld, D. (1994). Probabilistic planning with information gathering and contingent execution. In Proceedings of the Second International Conference on AI Planning Systems, pp. 31-36.
25. Eagle, J. N. (1984). The optimal search for a moving target when search path is constrained. Operations Research, 32, 1107-1115.
26. Eaves, B. (1984). A course in triangulations for saving differential equations with deformations. Springer-Verlag, Berlin.
27. Gordon, G. J. (1995). Stable function approximation in dynamic programming. In Proceedings of the Twelfth International Conference on Machine Learning.
28. Hansen, E. (1998a). An improved policy iteration algorithm for partially observable MDPs. In Advances in Neural Information Processing Systems 10. MIT Press.
29. Hansen, E. (1998b). Solving POMDPs by searching in policy space. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 211-219.
30. Hauskrecht, M. (1997). Planning and control in stochastic domains with imperfect information. Ph.D. thesis, Massachusetts Institute of Technology.
31. Hauskrecht, M., & Fraser, H. (1998). Planning medical therapy using partially observable Markov decision processes. In Proceedings of the Ninth International Workshop on Principles of Diagnosis (DX-98), pp. 182-189.
32. Hauskrecht, M., & Fraser, H. (2000). Planning treatment of ischemic heart disease with partially observable Markov decision processes. Artificial Intelligence in Medicine, 18, 221-244.
33. Heyman, D., & Sobel, M. (1984). Stochastic methods in operations research: stochastic optimization. McGraw-Hill.
34. Howard, R. A. (1960). Dynamic Programming and Markov Processes. MIT Press, Cambridge.
35. Howard, R. A., & Matheson, J. (1984). Influence diagrams. Principles and Applications of Decision Analysis, 2.
36. Isard, M., & Blake, A. (1996). Contour tracking by stochastic propagation of conditional density. In Proceedings of Europian Conference on Computer Vision, pp. 343-356.
37. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1999). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99-134.
38. Kanazawa, K., Koller, D., & Russell, S. J. (1995). Stochastic simulation algorithms for dynamic probabilistic networks. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp. 346-351.
39. Kearns, M., Mansour, Y., & Ng, A. Y. (1999). A sparse sampling algorithm for near optimal planning in large Markov decision processes. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, pp. 1324-1331.
40. Kjaerulff, U. (1992). A computational scheme for reasoning in dynamic probabilistic networks. In Proceedings of the Eighth Conference on Uncertainty in Artificial Intelligence, pp. 121-129.
41. Korf, R. (1985). Depth-first iterative deepening: an optimal admissible tree search. Artificial Intelligence, 27, 97-109.
42. Kushmerick, N., Hanks, S., & Weld, D. (1995). An algorithm for probabilistic planning. Artificial Intelligence, 76, 239-286.
43. Lauritzen, S. L. (1996). Graphical models. Clarendon Press.
44. Littman, M. L. (1994). Memoryless policies: Theoretical limitations and practical results. In Cliff, D., Husbands, P., Meyer, J., & Wilson, S. (Eds.), From Animals to Animals 3: Proceedings of the Third International Conference on Simulation of Adaptive Behavior. MIT Press, Cambridge.
45. Littman, M. L. (1996). Algorithms for sequential decision making. Ph.D. thesis, Brown University.
46. Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995). Learning policies for partially observable environments: scaling up. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 362-370.
47. Lovejoy, W. S. (1991a). Computationally feasible bounds for partially observed Markov decision processes. Operations Research, 39, 192-175.
48. Lovejoy, W. S. (1991b). A survey of algorithmic methods for partially observed Markov decision processes. Annals of Operations Research, 28, 47-66.
49. Lovejoy, W. S. (1993). Suboptimal policies with bounds for parameter adaptive decision processes. Operations Research, 41, 583-599.
50. Lusena, C., Goldsmith, J., & Mundhenk, M. (1998). Nonapproximability results for Markov decision processes. Tech. rep., University of Kentucky.
51. Madani, O., Hanks, S., & Condon, A. (1999). On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision processes. In Proceedings of the Sixteenth National Conference on Artificial Intelligence.
52. McAllester, D., & Singh, S. P. (1999). Approximate planning for factored POMDPs using belief state simplification. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, pp. 409-416.
53. McCallum, R. (1995). Instance-based utile distinctions for reinforcement learning with hidden state. In Proceedings of the Twelfth International Conference on Machine Learning.
54. Monahan, G. E. (1982). A survey of partially observable Markov decision processes: theory, models, and algorithms. Management Science, 28, 1-16.
55. Mundhenk, M., Goldsmith, J., Lusena, C., & Allender, E. (1997). Encyclopaedia of complexity results for finite-horizon Markov decision process problems. Tech. rep., CS Dept TR 273-97, University of Kentucky.
56. Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12, 441-450.
57. Parr, R., & Russell, S. (1995). Approximating optimal policies for partially observable stochastic domains. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1088-1094.
58. Pearl, J. (1988). Probabilistic reasoning in intelligent systems. Morgan Kaufman.
59. Platzman, L. K. (1977). Finite memory estimation and control of finite probabilistic systems. Ph.D. thesis, Massachusetts Institute of Technology.
60. Platzman, L. K. (1980). A feasible computational approach to infinite-horizon partially-observed Markov decision problems. Tech. rep., Georgia Institute of Technology.
61. Puterman, M. L. (1994). Markov decision processes: discrete stochastic dynamic programming. John Wiley, New York.
62. Raiffa, H. (1970). Decision analysis. Introductory lectures on choices under uncertainty. Addison-Wesley.
63. Rumelhart, D., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In Parallel Distributed Processing, pp. 318-362.
64. Satia, J., & Lave, R. (1973). Markovian decision processes with probabilistic observation of states. Management Science, 20, 1-13.
65. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Learning without state-estimation in partially observable Markovian decision processes. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 284-292.
66. Smallwood, R. D., & Sondik, E. J. (1973). The optimal control of partially observable processes over a finite horizon. Operations Research, 21, 1071-1088.
67. Sondik, E. J. (1971). The optimal control of partially observable Markov decision processes. Ph.D. thesis, Stanford University.
68. Sondik, E. J. (1978). The optimal control of partially observable processes over the infinite horizon: Discounted costs. Operations Research, 26, 282-304.
69. Tatman, J., & Schachter, R. D. (1990). Dynamic programming and influence diagrams. IEEE Transactions on Systems, Man and Cybernetics, 20, 365-379.
70. Tsitsiklis, J. N., & Roy, B. V. (1996). Feature-based methods for large-scale dynamic programming. Machine Learning, 22, 59-94.
71. Washington, R. (1996). Incremental Markov model planning. In Proceedings of the Eight IEEE International Conference on Tools with Artificial Intelligence, pp. 41-47.
72. White, C. C., & Scherer, W. T. (1994). Finite memory suboptimal design for partially observed Markov decision processes. Operations Research, 42, 439-455.
73. Williams, R. J., & Baird, L. C. (1994). Tight performance bounds on greedy policies based on imperfect value functions. In Proceedings of the Tenth Yale Workshop on Adaptive and Learning Systems Yale University.
74. Yost, K. A. (1998). Solution of large-scale allocation problems with partially observable outcomes. Ph.D. thesis, Naval Postgraduate School, Monterey, CA.
75. Zhang, N. L., & Lee, S. S. (1998). Planning with partially observable Markov decision processes: Advances in exact solution method. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 523-530.
76. Zhang, N. L., & Liu, W. (1997a). A model approximation scheme for planning in partially observable stochastic domains. Journal of Artificial Intelligence Research, 7, 199-230.
77. Zhang, N. L., & Liu, W. (1997b). Region-based approximations for planning in stochastic domains. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pp. 472-480.