Scalable and efficient bayes-adaptive reinforcement learning based on Monte-Carlo tree search

Journal of Artificial Intelligence Research

Volumn 48, Issue , 2013, Pages 841-883

  Guez, Arthur ^a   Silver, David ^b   Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

BAYESIAN NETWORKS; LEARNING ALGORITHMS; MACHINE LEARNING; MONTE CARLO METHODS; UNCERTAINTY ANALYSIS;

ADAPTIVE REINFORCEMENT; BAYESIAN MODEL; BENCH-MARK PROBLEMS; EXPLORATION AND EXPLOITATION; INFINITE STATE SPACE; MODEL UNCERTAINTIES; MONTE-CARLO TREE SEARCHES; SAMPLING MODEL;

REINFORCEMENT LEARNING;

EID: 84893049023     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.4117     Document Type: Article

References (56)

1. Agrawal, S., & Goyal, N. (2011). Analysis of Thompson sampling for the multi-armed bandit problem. Arxiv preprint.
2. Araya-López, M., Buffet, O., & Thomas, V. (2012). Near-optimal BRL using optimistic local transitions. In Proceedings of the 29th International Conference on Machine Learning.
3. Asmuth, J., Li, L., Littman, M., Nouri, A., & Wingate, D. (2009). A Bayesian sampling approach to exploration in reinforcement learning. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 19-26.
4. Asmuth, J., & Littman, M. (2011). Approaching Bayes-optimality using Monte-Carlo tree search. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pp. 19-26.
5. Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2), 235-256. (Pubitemid 34126111)
6. Bellman, R. (1954). The theory of dynamic programming. Bull. Amer. Math. Soc, 60(6), 503-515.
7. Brafman, R., & Tennenholtz, M. (2003). R-max-a general polynomial time algorithm for near-optimal reinforcement learning. The Journal of Machine Learning Research, 3, 213-231.
8. Bubeck, S., Munos, R., & Stoltz, G. (2009). Pure exploration in multi-armed bandits problems. In Proceedings of the 20th international conference on Algorithmic learning theory, pp. 23-37. Springer-Verlag.
9. Castro, P., & Precup, D. (2010). Smarter sampling in model-based Bayesian reinforcement learning. In Machine Learning and Knowledge Discovery in Databases, pp. 200-214. Springer.
10. Castro, P. (2007). Bayesian exploration in Markov decision processes. Ph. D. thesis, McGill University.
11. Castro, P., & Precup, D. (2007). Using linear programming for Bayesian exploration in Markov decision processes. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 2437-2442.
12. Coquelin, P., & Munos, R. (2007). Bandit algorithms for tree search. In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, pp. 67-74.
13. Cozzolino, J., Gonzalez-Zubieta, R., & Miller, R. (1965). Markov decision processes with uncertain transition probabilities. Tech. rep., 11, Operations Research Center, MIT.
14. Davies, S., Ng, A., & Moore, A. (1998). Applying online search techniques to reinforcement learning. In Proceedings of the National Conference on Artificial Intelligence, pp. 753-760.
15. Dayan, P., & Sejnowski, T. (1996). Exploration bonuses and dual control. Machine Learning, 25(1), 5-22. (Pubitemid 126724387)
16. Dearden, R., Friedman, N., & Russell, S. (1998). Bayesian Q-learning. In Proceedings of the National Conference on Artificial Intelligence, pp. 761-768.
17. Doshi-Velez, F., Wingate, D., Roy, N., & Tenenbaum, J. (2010). Nonparametric bayesian policy priors for reinforcement learning. In Advances in Neural Information Processing Systems (NIPS).
18. Duff, M. (2003). Design for an optimal probe. In Proceedings of the 20th International Conference on Machine Learning, pp. 131-138.
19. Duff, M. (2002). Optimal Learning: Computational Procedures For Bayes-Adaptive Markov Decision Processes. Ph. D. thesis, University of Massachusetts Amherst.
20. Fonteneau, R., Busoniu, L., & Munos, R. (2013). Optimistic planning for belief-augmented Markov decision processes. In IEEE International Symposium on Adaptive Dynamic Programming and reinforcement Learning (ADPRL 2013).
21. Friedman, N., & Singer, Y. (1999). Efficient Bayesian parameter estimation in large discrete domains. Advances in Neural Information Processing Systems (NIPS), 1(1), 417-423.
22. Gelly, S., Kocsis, L., Schoenauer, M., Sebag, M., Silver, D., Szepesvári, C., & Teytaud, O. (2012). The grand challenge of computer Go: Monte Carlo tree search and extensions. Communications of the ACM, 55(3), 106-113.
23. Gelly, S., & Silver, D. (2007). Combining online and offline knowledge in UCT. In Proceedings of the 24th International Conference on Machine learning, pp. 273-280.
24. Gittins, J., Weber, R., & Glazebrook, K. (1989). Multi-armed bandit allocation indices. Wiley Online Library.
25. Guez, A., Silver, D., & Dayan, P. (2012). Efficient Bayes-adaptive reinforcement learning using sample-based search. In Advances in Neural Information Processing Systems (NIPS), pp. 1034-1042.
26. Hart, P., Nilsson, N., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. Systems Science and Cybernetics, IEEE Transactions on, 4(2), 100-107.
27. Jaksch, T., Ortner, R., & Auer, P. (2010). Near-optimal regret bounds for reinforcement learning. The Journal of Machine Learning Research, 99, 1563-1600.
28. Kearns, M., Mansour, Y., & Ng, A. (1999). A sparse sampling algorithm for near-optimal planning in large Markov decision processes. In Proceedings of the 16th international joint conference on Artificial intelligence-Volume 2, pp. 1324-1331.
29. Kocsis, L., & Szepesvári, C. (2006). Bandit based Monte-Carlo planning. In Machine Learning: ECML, pp. 282-293. Springer. (Pubitemid 44618839)
30. Kolter, J., & Ng, A. (2009). Near-Bayesian exploration in polynomial time. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 513-520.
31. Madani, O., Hanks, S., & Condon, A. (2003). On the undecidability of probabilistic planning and related stochastic optimization problems. Artificial Intelligence, 147(1), 5-34.
32. Martin, J. (1967). Bayesian decision problems and Markov chains. Wiley.
33. Meuleau, N., & Bourgine, P. (1999). Exploration of multi-state environments: Local measures and back-propagation of uncertainty. Machine Learning, 35(2), 117-154.
34. Mundhenk, M., Goldsmith, J., Lusena, C., & Allender, E. (2000). Complexity of finitehorizon markov decision process problems. Journal of the ACM (JACM), 47(4), 681-720.
35. Neal, R. M. (1993). Probabilistic inference using markov chain monte carlo methods. Tech. rep., University of Toronto.
36. Poupart, P., Vlassis, N., Hoey, J., & Regan, K. (2006). An analytic solution to discrete Bayesian reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 697-704. ACM.
37. Ross, S., Pineau, J., Chaib-draa, B., & Kreitmann, P. (2011). A Bayesian approach for learning and planning in Partially Observable Markov Decision Processes. Journal of Machine Learning Research, 12, 1729-1770.
38. Ross, S., Pineau, J., Paquet, S., & Chaib-Draa, B. (2008). Online planning algorithms for POMDPs. Journal of Artificial Intelligence Research, 32(1), 663-704.
39. Ross, S. (1983). Introduction to stochastic dynamic programming: Probability and mathematical. Academic Press, Inc.
40. Schmidhuber, J. (1991). Curious model-building control systems. In IEEE International Joint Conference on Neural Networks, pp. 1458-1463.
41. Silver, D., & Veness, J. (2010). Monte-Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems (NIPS), pp. 2164-2172.
42. Silver, D., Sutton, R. S., & Müller, M. (2008). Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th international conference on Machine learning, pp. 968-975. ACM.
43. Silver, E. (1963). Markovian decision processes with uncertain transition probabilities or rewards. Tech. rep., DTIC Document.
44. Strehl, A., Li, L., & Littman, M. (2009). Reinforcement learning in finite MDPs: PAC analysis. The Journal of Machine Learning Research, 10, 2413-2444.
45. Strens, M. (2000). A Bayesian framework for reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, pp. 943-950.
46. Sutton, R. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the Seventh International Conference on Machine Learning, Vol. 216, p. 224. Citeseer.
47. Sutton, R., & Barto, A. (1998). Reinforcement learning. MIT Press.
48. Szepesvári, C. (2010). Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers.
49. Thompson, W. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4), 285-294.
50. Tonk, S., & Kappen, H. (2010). Optimal exploration as a symmetry breaking phenomenon. Tech. rep., Radboud University Nijmegen.
51. Vien, N. A., & Ertel, W. (2012). Monte carlo tree search for bayesian reinforcement learning. In Machine Learning and Applications (ICMLA), 2012 11th International Conference on, Vol. 1, pp. 138-143. IEEE.
52. Walsh, T., Goschin, S., & Littman, M. (2010). Integrating sample-based planning and model-based reinforcement learning. In Proceedings of the 24th Conference on Artificial Intelligence (AAAI).
53. Wang, T., Lizotte, D., Bowling, M., & Schuurmans, D. (2005). Bayesian sparse sampling for on-line reward optimization. In Proceedings of the 22nd International Conference on Machine learning, pp. 956-963.
54. Wang, Y., Won, K., Hsu, D., & Lee, W. (2012). Monte Carlo Bayesian reinforcement learning. In Proceedings of the 29th International Conference on Machine Learning.
55. Watkins, C. (1989). Learning from delayed rewards. Ph. D. thesis, Cambridge.
56. Wingate, D., Goodman, N., Roy, D., Kaelbling, L., & Tenenbaum, J. (2011). Bayesian policy search with policy priors. In Proceedings of the International Joint Conferences on Artificial Intelligence.