Bayesian approach for learning and planning in partially observable markov decision processes

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 1729-1770

  Ross, Stéphane ^a   Pineau, Joelle ^b   Chaib Draa, Brahim ^c   Kreitmann, Pierre ^d  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b MCGILL UNIVERSITY   (Canada)

^c UNIVERSITÉ LAVAL   (Canada)

^d Stanford University   (United States)

Author keywords

Bayesian inference; Partially observable Markov decision processes; Reinforcement learning

Indexed keywords

APPROXIMATE ALGORITHMS; BAYESIAN APPROACHES; BAYESIAN INFERENCE; EMPIRICAL RESULTS; LEARNING PERFORMANCE; MARKOV DECISION PROCESSES; MODEL ESTIMATES; OPTIMAL SEQUENCE; PARTIAL OBSERVABILITY; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; THEORETICAL RESULT;

BAYESIAN NETWORKS; INFERENCE ENGINES; REINFORCEMENT LEARNING;

MARKOV PROCESSES;

EID: 79960110381     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (56)

1. J. Asmuth, L. Li, M. Littman, A. Nouri, and D. Wingate. A bayesian sampling approach to exploration in reinforcement learning. In Conference on Uncertainty in Artificial Intelligence (UAI), 2009.
2. P. Auer and R. Ortner. Logarithmic online regret bounds for undiscounted reinforcement learning. In Neural Information Processing Systems (NIPS), volume 19, pages 49-56, 2006.
3. P. Auer, T. Jaksch, and R. Ortner. Near-optimal regret bounds for reinforcement learning. In Neural Information Processing Systems (NIPS), volume 21, 2009.
4. J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research (JAIR), 15:319-350, 2001.
5. R. Bellman. Adaptive Control Processes: A Guided Tour. Princeton University Press, 1961.
6. R. I. Brafman and M. Tennenholtz. R-max - a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research (JMLR), 3:213-231, 2003.
7. G. Casella and R. Berger. Statistical Inference. Duxbury Resource Center, 2001.
8. P. S. Castro and D. Precup. Using linear programming for bayesian exploration in markov decision processes. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2437-2442, 2007.
9. R. Dearden, N. Friedman, and S. J. Russell. Bayesian Q-learning. In AAAI Conference on Artificial Intelligence, pages 761-768, 1998.
10. R. Dearden, N. Friedman, and D. Andre. Model based bayesian exploration. In Conference on Uncertainty in Artificial Intelligence (UAI), pages 150-159, 1999.
11. E. Delage and S. Mannor. Percentile optimization in uncertain mdps with application to efficient exploration. In International Conference on Machine Learning (ICML), 2007.
12. F. Doshi, J. Pineau, and N. Roy. Reinforcement learning with limited reinforcement: Using Bayes risk for active learning in POMDPs. In International Conference on Machine Learning, pages 256-263. ACM, 2008.
13. F. Doshi-Velez. The infinite partially observable markov decision process. In Neural Information Processing Systems (NIPS), volume 22, 2010.
14. A. Doucet, N. de Freitas, and N. Gordon. Sequential Monte Carlo Methods In Practice. Springer, 2001.
15. M. Duff. Monte-Carlo algorithms for the improvement of finite-state stochastic controllers: Application to bayes-adaptive Markov decision processes. In International Workshop on Artificial Intelligence and Statistics (AISTATS), 2001.
16. M. Duff. Optimal Learning: Computational Procedures for Bayes-Adaptive Markov Decision Processes. PhD thesis, University of Massachusetts Amherst, Amherst, MA, 2002.
17. Y. Engel, S. Mannor, and R. Meir. Bayes meets Bellman: The gaussian process approach to temporal difference learning. In International Conference on Machine Learning (ICML), pages 154-161, 2003.
18. Y. Engel, S. Mannor, and R. Meir. Reinforcement learning with gaussian processes. In International Conference on Machine learning (ICML), pages 201-208, 2005. (Pubitemid 43183334)
19. A. A. Feldbaum. Dual control theory, parts i and ii. Automation and Remote Control, 21:874-880 and 1033-1039, 1961.
20. N.M. Filatov and H. Unbehauen. Survey of adaptive dual control methods. In IEEE Control Theory and Applications, volume 147, pages 118-128, 2000. (Pubitemid 30563857)
21. M. Ghavamzadeh and Y. Engel. Bayesian policy gradient algorithms. In Neural Information Processing Systems (NIPS), volume 19, pages 457-464, 2007a.
22. M. Ghavamzadeh and Y. Engel. Bayesian actor-critic algorithms. In International Conference on Machine Learning (ICML), pages 297-304, 2007b.
23. A. Greenfield and A. Brockwell. Adaptive control of nonlinear stochastic systems by particle filtering. In International Conference on Control and Automation (ICCA), pages 887-890, 2003. (Pubitemid 41244024)
24. D. Heckerman, D. Geiger, and D. M. Chickering. Learning bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20(3):197-243, 1995.
25. M. Hutter. Universal Artificial Intelligence. Springer, 2005.
26. R. Jaulmes, J. Pineau, and D. Precup. Active learning in partially observable markov decision processes. European Conference on Machine Learning, pages 601-608, 2005.
27. E. T. Jaynes. Prior probabilities. IEEE Transactions on Systems Science and Cybernetics, 4:227-241, 1968.
28. H. Jeffreys. Theory of Probability. Oxford University Press, 1961.
29. L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99-134, 1998. (Pubitemid 128387390)
30. M. Kearns and S. Singh. Near-optimal reinforcement learning in polynomial time. In International Conference on Machine Learning (ICML), pages 260-268, 1998.
31. M. J. Kearns, Y. Mansour, and A. Y. Ng. A sparse sampling algorithm for near-optimal planning in large markov decision processes. In International Joint Conference on Artificial Intelligence(IJCAI), pages 1324-1331, 1999.
32. J. Zico Kolter and Andrew Y. Ng. Near-bayesian exploration in polynomial time. In International Conference on Machine Learning (ICML), 2009.
33. M. L. Littman, R. S. Sutton, and S. Singh. Predictive representations of state. In Neural Information Processing Systems (NIPS), volume 14, pages 1555-1561, 2002.
34. A. K. McCallum. Reinforcement Learning with Selective Perception and Hidden State. PhD thesis, University of Rochester, 1996.
35. S. Paquet, L. Tobin, and B. Chaib-draa. An online POMDP algorithm for complex multiagent environments. In International Joint Conference on Autonomous Agents and Multi Agent Systems (AAMAS), pages 970-977, 2005.
36. J. Pineau, G. Gordon, and S. Thrun. Point-based value iteration: an anytime algorithm for POMDPs. In International Joint Conference on Artificial Intelligence (IJCAI), pages 1025-1032, 2003.
37. P. Poupart and N. Vlassis. Model-based bayesian reinforcement learning in partially observable domains. In International Symposium on Artificial Intelligence and Mathematics (ISAIM), 2008.
38. P. Poupart, N. Vlassis, J. Hoey, and K. Regan. An analytic solution to discrete bayesian reinforcement learning. In International Conference on Machine learning (ICML), pages 697-704, 2006.
39. R. Ravikanth, S.P. Meyn, and L.J. Brown. Bayesian adaptive control of time varying systems. In IEEE Conference on Decision and Control, pages 705-709, 1992.
40. S. Ross, B. Chaib-draa, and J. Pineau. Bayes-adaptive POMDPs. In Neural Information Processing Systems (NIPS), volume 20, pages 1225-1232, 2008a.
41. S. Ross, B. Chaib-draa, and J. Pineau. Bayesian reinforcement learning in continuous POMDPs. In International Conference on Robotics and Automation (ICRA), 2008b.
42. S. Ross, J. Pineau, S. Paquet, and B. Chaib-draa. Online POMDPs. Journal of Artificial Intelligence Research (JAIR), 32:663-704, 2008c.
43. I. Rusnak. Optimal adaptive control of uncertain stochastic discrete linear systems. In IEEE International Conference on Systems, Man and Cybernetics, pages 4521-4526, 1995.
44. D. Silver and J. Veness. Monte-Carlo planning in large POMDPs. In Neural Information Processing Systems (NIPS), 2010.
45. R. D. Smallwood and E. J. Sondik. The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21(5):1071-1088, Sep/Oct 1973.
46. T. Smith and R. Simmons. Heuristic search value iteration for POMDPs. In Conference on Uncertainty in Artificial Intelligence (UAI), pages 520-527, 2004.
47. E. J. Sondik. The Optimal Control of Partially Observable Markov Processes. PhD thesis, Stanford University, 1971.
48. M. T. J. Spaan and N. Vlassis. Perseus: randomized point-based value iteration for POMDPs. Journal of Artificial Intelligence Research (JAIR), 24:195-220, 2005. (Pubitemid 43130936)
49. A. L. Strehl and M. L. Littman. A theoretical analysis of model-based interval estimation. In International Conference on Machine learning (ICML), pages 856-863, 2005.
50. M. Strens. A bayesian framework for reinforcement learning. In International Conference on Machine Learning (ICML), 2000.
51. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998.
52. C. Szepesvari. Algorithms for Reinforcement Learning. Morgan & Claypool, 2010.
53. A. Tewari and P. Bartlett. Optimistic linear programming gives logarithmic regret for irreducible MDPs. In Neural Information Processing Systems (NIPS), volume 20, pages 1505-1512, 2008.
54. J. Veness, K. S. Ng, M. Hutter, W. Uther, and D. Silver. A monte-carlo aixi approximation. Journal of Artificial Intelligence Research (JAIR), 2011.
55. T. Wang, D. Lizotte, M. Bowling, and D. Schuurmans. Bayesian sparse sampling for on-line reward optimization. In International Conference on Machine learning (ICML), pages 956-963, 2005.
56. O. Zane. Discrete-time bayesian adaptive control problems with complete information. In IEEE Conference on Decision and Control, pages 2748-2749, 1992.