Point-based value iteration for continuous POMDPs

Journal of Machine Learning Research

Volumn 7, Issue , 2006, Pages 2329-2367

  Porta, Josep M ^a   Vlassis, Nikos ^b   Spaan, Matthijs T J ^b   Poupart, Pascal ^c  

^a CSIC   (Spain)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

^c UNIVERSITY OF WATERLOO   (Canada)

Author keywords

Continuous action space; Continuous observation space; Continuous state space; Partially observable Markov decision processes; Planning under uncertainty; Point based value iteration

Indexed keywords

COMPUTATIONAL COMPLEXITY; ITERATIVE METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; OPTIMIZATION;

CONTINUOUS ACTION SPACE; CONTINUOUS OBSERVATION SPACE; PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES; PLANNING UNDER UNCERTAINTY; POINT-BASED VALUE ITERATION;

LEARNING SYSTEMS;

EID: 33750724397     PISSN: 15337928     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (59)

1. K. J. Åström. Optimal control of Markov decision processes with incomplete state estimation. Journal of Mathematical Analysis and Applications, 10:174-205, 1965.
2. D. Aberdeen and J. Baxter. Scaling internal-state policy-gradient methods for POMDPs. In Proceedings of the International Conference on Machine Learning, pages 3-10, Sydney, Australia, 2002.
3. B. Bakker. Reinforcement learning with long short-term memory. In Advances in Neural Information Processing Systems 15 (NIPS-2002), pages 1475-1482. MIT Press, 2003.
4. J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319-350, 2001.
5. R. E. Bellman. Dynamic Programming. Princenton University Press, 1957.
6. D. P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, Belmont, MA, 2001. 2nd Edition.
7. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.
8. J. Boger, P. Poupart, J. Hoey, C. Boutilier, G. Fernie, and A. Mihailidis. A decision-theoretic approach to task assistance for persons with dementia. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1293-1299, Edinburgh, Scotland, 2005.
9. C. Boutilier. A POMDP formulation of preference elicitation problems. In Proceedings of the National Conference on Artificial Intelligence, pages 239-246, Edmonton, AB, 2002.
10. C. Boutilier and D. Poole. Computing optimal policies for partially observable decision processes using compact representations. In Proceedings of the National Conference on Artificial Intelligence, pages 1168-1175, Portland, OR, 1996.
11. A. Brooks, A. Makarenko, S. Williams, and H. Durrant-Whyte. Planning in continuous state spaces with parametric POMDPs. In Reasoning with Uncertainty in Robotics, Workshop of the International Joint Conference on Artificial Intelligence, pages 40-47, 2006.
12. A. R. Cassandra, L. P. Kaelbling, and J. A. Kurien. Acting under uncertainty: discrete Bayesian models for mobile-robot navigation. In Proceedings of the International Conference on Intelligent Robots and Systems, pages 963-972, 1996.
13. A. R. Cassandra, M. L. Littman, and N. L. Zhang. Incremental pruning: A simple, fast, exact algorithm for partially observable Markov decision processes. In Proceedings of Uncertainty in Artificial Intelligence, pages 54-61, 1997.
14. H. T. Cheng. Algorithms for Partially Observable Markov Decision Processes. PhD thesis, University of British Columbia, 1988.
15. T. Darrell and A. P. Pentland. Active gesture recognition using partially observable Markov decision processes. In IEEE International Conference on Pattern Recognition, pages 984-988, Vienna, Austria, 1996.
16. F. Dellaert, D. Fox, W. Burgard, and S. Thrun. Monte-Carlo localization for mobile robots. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 13221328, 1999.
17. A. Doucet, N. de Freitas, and N. Gordon. Sequential Monte Carlo in Practice. Springer-Verlag, New York, 2001.
18. M. Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massassachusetts Amherst, 2002.
19. E. B. Dynkin. Controlled random sequences. Theory of probability and its applications, 10(1): 1-14, 1965.
20. D. Fox. Kld-sampling: Adaptive particle niters. In Advances in Neural Information Processing Systems 14 (NIPS-2001), pages 713-720. MIT Press, 2002.
21. D. Fox. Adapting the sample size in particle niters through kld-sampling. International Journal of Robotics Research, 22(10-11):985-1004, 2003.
22. J. Goldberger and S. Roweis. Hierarchical clustering of a mixture model. In Advances in Neural Information Processing Systems 17 (NIPS-2004), pages 505-512. MIT Press, 2005.
23. J. Hoey and P. Poupart. Solving pomdps with continuous or large discrete observation spaces. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1332-1338, 2005.
24. M. Isard and A. Blake. Condensation - conditional density propagation for visual tracking. International Journal of Computer Vision, 29(1):5-28, 1998.
25. L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-134, 1998.
26. C. Lusena, J. Goldsmith, and M. Mundhenk. Nonapproximability results for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 14:83-103, 2001.
27. O. Madani, S. Hanks, and A. Condon. On the undecidability of probabilistic planning and infinitehorizon partially observable Markov decision problems. In Proceedings of the National Conference on Artificial Intelligence, pages 541-548, Orlando, FL, 1999.
28. N. Meuleau, L. Peshkin, K.-E. Kim, and L. P. Kaelbling. Learning finite-state controllers for partially observable environments. In Proceedings of Uncertainty in Artificial Intelligence, pages 427-436, Stockholm, 1999.
29. G. E. Monahan. A survey of partially observable Markov decision processes: Theory, models, and algorithms. Management Science, 28(1): 1-16, 1982.
30. M. Montemerlo, J. Pineau, N. Roy, S. Thrun, and V. Verma. Experiences with a mobile robotic guide for the elderly. In Proceedings of the National Conference on Artificial Intelligence, pages 587-592, Edmonton, AB, 2002.
31. A. Y. Ng and M. Jordan. PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of Uncertainty in Artificial Intelligence, pages 406-415, Stanford, CA, 2000.
32. C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):441-450, 1987.
33. J. Pineau, G. Gordon, and S. Thrun. Point-based value iteration: An anytime algorithm for pomdps. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1025-1032, 2003a.
34. J. Pineau, M. Montemerlo, M. Pollack, N. Roy, and S. Thrun. Towards robotic assistants in nursing homes: challenges and results. Robotics and Autonomous Systems, 42(3-4):271-281, 2003b.
35. M. K. Pitt and N. Shephard. Filtering via simulation: Auxiliary particle niters. Journal of the American Statististical Association, 94(446):590-599, 1999.
36. J. M. Porta and B. J. A. Kröse. Appearance-based concurrent map building and localization using a multi-hypotheses tracker. In Proceedings of the International Conference on Intelligent Robots and Systems, pages 3424-3429, Sendai, Japan, 2004.
37. J. M. Porta, M. T. J. Spaan, and N. Vlassis. Robot planning in partially observable continuous domains. In Robotics: Science and Systems I, pages 217-224, MIT, Cambridge, MA, 2005.
38. P. Poupart and C. Boutilier. Value-directed compressions of POMDPs. In Advances in Neural Information Processing Systems 15 (NIPS-2002), pages 1547-1554. MIT Press, 2003.
39. P. Poupart and C. Boutilier. Bounded finite state controllers. In Advances in Neural Information Processing Systems 16 (NIPS-2003), pages 823-830. MIT Press, 2004.
40. P. Poupart and C. Boutilier. VDCBPI: an approximate scalable algorithm for large POMDPs. In Advances in Neural Information Processing Systems 17 (NIPS-2004), pages 1081-1088. MIT Press, 2005.
41. P. Poupart, N. Vlassis, J. Hoey, and K. Regan. An analytic solution to discrete bayesian reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 697-704, 2006.
42. M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Mathematical Statistics. John Wiley and Sons, Inc., 1994.
43. N. Roy. Finding Approximate POMDP Solutions Through Belief Compression. PhD thesis, Carnegie Mellon University, 2003.
44. N. Roy and G. Gordon. Exponential family PCA for belief compression in POMDPs. In Advances in Neural Information Processing Systems 15 (NIPS-2002), pages 1635-1642. MIT Press, 2003.
45. N. Roy, G. Gordon, and S. Thrun. Finding approximate POMDP solutions through belief compression. Journal of Artificial Intelligence Research, 23:1-40, 2005.
46. N. Roy, J. Pineau, and S. Thrun. Spoken dialog management using probabilistic reasoning. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 93-100, Hong Kong, 2000.
47. Matt Rudary, Satinder Singh, and David Wingate. Predictive linear-gaussian models of stochastic dynamical systems. In Proceedings of Uncertainty in Artificial Intelligence, pages 501-508, 2005.
48. R. Simmons and S. Koenig. Probabilistic robot navigation in partially observable environments. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1080-1087, 1995.
49. T. Smith and R. Simmons. Heuristic search value iteration for POMDPs. In. Proceedings of Uncertainty in Artificial Intelligence, pages 520-527, Banff, Alberta, 2004.
50. E. J. Sondik. The Optimal Control of Partially Observable Markov Processes. PhD thesis, Stanford University, 1971.
51. M. T. J. Spaan and N. Vlassis. Perseus: Randomized point-based value iteration for pomdps. Journal of Artificial Intelligence Research, 24:195-220, 2005.
52. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
53. G. Theocharous and S. Mahadevan. Approximate planning with hierarchical partially observable Markov decision processes for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 1347-1352, 2002.
54. S. Thrun. Monte Carlo POMDPs. In Advances in Neural Information Processing Systems 12 (NIPS-1999), pages 1064-1070. MIT Press, 2000.
55. N. Vlassis, B. Terwijn, and B.J.A. Kröse. Auxiliary particle filter robot localization from high-dimensional sensor observations. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 7-12, 2002.
56. Steven D. Whitehead and Long-Ji Lin. Reinforcement learning of non-Markov decision processes. Artificial Intelligence, 73(1-2):271-306, 1995.
57. J. Williams, P. Poupart, and S. Young. Using factored Markov decision processes with continuous observations for dialogue management. Technical Report CUED/F-INFEG/TR.520, Cambridge University, Engineering Department, 2005.
58. B. Zhang, Q. Cai, J. Mao, and B. Guo. Planning and acting under uncertainty: a new model for spoken dialogue systems. In Proceedings of Uncertainty in Artificial Intelligence, pages 572-579, Seattle, WA, 2001.
59. N. L. Zhang and W. Zhang. Speeding up the convergence of value iteration in partially observable Markov decision processes. Journal of Artificial Intelligence Research, 14:29-51, 2001.