Restricted value iteration: Theory and algorithms

Journal of Artificial Intelligence Research

Volumn 23, Issue , 2005, Pages 123-165

  Zhang, Weihong ^a   Zhang, Nevin L ^b  

^a WASHINGTON UNIVERSITY   (United States)

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ITERATIVE METHODS; OPTIMAL CONTROL SYSTEMS; SYSTEMS ANALYSIS;

NEAR-OPTIMAL POLICIES; SUBSETS; VALUE ITERATION;

PROBLEM SOLVING;

EID: 27344454651     PISSN: 10769757     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.1379     Document Type: Article

References (46)

1. Aström, K. J. (1965). Optimal control of Markov decision processes with incomplete state estimation. Journal of Mathematical Analysis and Applications, 10, 403-406.
2. Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence, 72, 81-138.
3. Bellman, R. (1957). Dynamic Programming. Princeton University Press.
4. Bertsekas, D. P., & Gallagher, R. G. (1995). Data Networks. Prentice Hall.
5. Bonet, B., & Geffner, H. (2000). Planning with incomplete information as heuristic search in belief space. In Proceedings of the 6th International Conference on Artificial Intelligence in Planning Systems (AIPS), pp. 52-61. AAAI Press.
6. Boutilier, C., Brafman, R. I., & Geib, C. (1998). Structured reachability analysis for Markov decision processes. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 24-32.
7. Boutilier, C., & Poole, D. (1996). Computing optimal policies for partially observable decision processes using compact representations. In Thirteenth National Conference on Artificial Intelligence (AAAI), pp. 1168-1175. Portland, Oregon.
8. Burago, D., de Rougemont, M., & Slissekno, A. (1996). On the complexity of partially observed Markov decision processes. Theoretical Computer Science, 157(2), 161-183.
9. Cassandra, A. R. (1998a). Exact and approximate algorithms for partially observable Markov decision processes. Ph.D. thesis, Department of Computer science, Brown university.
10. Cassandra, A. R. (1998b). A survey of POMDP applications. In Working Notes of AAAI 1998 Fall Symposium on Planning with Partially Observable Markov Decision Processes, pp. 17-24.
11. Cassandra, A. R., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: a simple, fast, exact method for partially observable Markov decision processes. In Proceedings of the 13th Conference on Uncertainty in Artificial Intelligence, pp. 54-61.
12. Choi, S. P. M., Yeung, D. Y., & Zhang, N. L. (1999). An environment model for nonstationary reinforcement learning. In Advances in Neural Information Processing Systems 12, pp. 987-993.
13. Dean, T., Givan, R., & Kim, K. (1998). Solving planning problems with large state and action spaces. In Proceedings of the 4th International Conference on Artificial Intelligence in Planning Systems (AIPS), pp. 102-110. Pittsburgh, Pennsylvania.
14. Dean, T. L., Kaelbling, L. P., Kirman, J., & Nicholson, A. (1993). Planning with deadlines in stochastic domains. In Proceedings of the 9th National Conference on Artificial Intelligence (AAAI), pp. 574-579.
15. Dean, T. L., & Lin, S. H. (1995). Decomposition techniques for planning in stochastic domains. In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), pp. 1121-1127.
16. Drummond, M., & Bresina, J. (1990). Anytime synthetic projection: maximizing the probability of goal satisfaction. In Proceedings of National Conference on Artificial Intelligence (AAAI), pp. 138-144.
17. Geffner, H., & Bonet, B. (1998). Solving large POMDPs using real time dynamic programming. In Working Notes Fall AAAI Symposium on POMDPs, pp. 61-68.
18. Hansen, E. A. (1998). Finite memory control of partially observable systems. Ph.D. thesis, Dept of Computer Science, University of Massachusetts at Amherst.
19. Hansen, E. A., & Ziberstein, S. (1998). Heuristic search in cyclic AND/OR graphs. In Proceedings of National Conference on Artificial Intelligence (AAAI), pp. 412-417.
20. Hauskrecht, M. (1997). Incremental methods for computing bounds in partially observable Markov decision processes. In Proceedings of National Conference on Artificial Intelligence (AAAI), pp. 734-739.
21. Hauskrecht, M. (2000). Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 13, 33-94.
22. Hauskrecht, M., & Fraser, H. (1998). Modeling treatment of ischemic heart disease with partially observable Markov decision processes. In American Medical Informatics Association annual symposium on Computer Applications in Health Care, pp. 538-542. Orlando, Florida.
23. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2).
24. Koller, D., & Parr, R. (2000). Policy iteration for factored MDPs. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI), pp. 326-334.
25. Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995). Efficient dynamic programming updates in partially observable Markov decision processes. Tech. rep. CS-95-19, Department of Computer Science, Brown University.
26. Littman, M. L., Goldsmith, J., & Mundhenk, M. (1998). The computational complexity of probabilistic planning. Journal of Artificial Intelligence Research, 9, 1-36.
27. Lovejoy, W. S. (1991). Computationally feasible bounds for partially observed Markov decision processes. Operations Research, 39(1), 162-175.
28. Madani., O., Hanks, S., & Condon, A. (1999). On the undecidability of probabilistic planning and infinite horizon partially observable Markov decision problems.
29. Monahan, G. E. (1982). A survey of partially observable Markov decision processes: theory, models, and algorithms. Management Science, 28(1), 1-16.
30. Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441-450.
31. Parr, R. (1998). Flexible decomposition algorithms for weakly coupled Markov decision problems. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 422-430.
32. Parr, R., & Russell, S. (1995). Approximating optimal policies for partially observable stochastic domains. In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), pp. 1088-1094.
33. Pineau, J., Gordon, G., & Thrun, S. (2003). Point-based value iteration: an anytime algorithm for POMDPs. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 1025-1032.
34. Poupart, P., & Boutilier, C. (2002). Value-directed compresseion of POMDPs. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pp. 1547-1554.
35. Puterman, M. L. (1994). Markov decision processes: discrete stochastic dynamic programming. Wiley, New York, NY.
36. Roy, N., & Gordon, G. (2002). Exponential family PCA for belief compression in POMDPs. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pp. 1635-1642.
37. Smallwood, R. D., & Sondik, E. J. (1973). The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21, 1071-1088.
38. Sondik, E. J. (1971). The optimal control of partially observable decision processes. Ph.D. thesis, Stanford University, Stanford, California, USA.
39. Washington, R. (1997). BI-POMDP: Bounded, incremental partially-observable Markovmodel planning. In Proceedings of the 4th European Conference on Planning (ECP), pp. 440-451.
40. Zhang, N. L., & Liu, W. (1997). A model approximation scheme for planning in partially observable stochastic domains. Journal of Artificial Intelligence Research, 7, 199-230.
41. Zhang, N. L., & Zhang, W. (2001a). Space-progressive value iteration: an anytime algorithm for a class of POMDPs. In Sixth European Conference on Symbolic and Quantitative Approaches to Reasoning with Uncertainty (ECSQARU), pp. 72-83.
42. Zhang, N. L., & Zhang, W. (2001b). Speeding up the convergence of value iteration in partially observable Markov decision processes. Journal of Artificial Intelligence Research, 14, 29-51.
43. Zhang, W. (2001). Algorithms for partially observable Markov decision processes. Ph.D. thesis, Department of Computer Science, the Hong Kong University of Science and Technology.
44. Zhang, W., & Zhang, N. L. (2001). Solving informative partially observable Markov decision processes. In Proceedings of the 6th European Conference on Planning (ECP).
45. Zhou, R., & Hansen, E. (2001). An improved grid-based approximation algorithm for POMDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, pp. 707-716.
46. Zubek, V. B., & Dietterich, T. G. (2000). A POMDP approximation algorithm that anticipates the need to observe. In Proceedings of PRICAI-2000, pp. 521-532. Lecture Notes in Computer Science, New York: Springer-Verlag.