Why is posterior sampling better than optimism for reinforcement learning?

34th International Conference on Machine Learning, ICML 2017

Volumn 6, Issue , 2017, Pages 4133-4148

  Osband, Ian ^a,b   Van Roy, Benjamin ^a  

^a STANFORD UNIVERSITY   (United States)

^b DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; MARKOV PROCESSES; REINFORCEMENT LEARNING;

BAYESIAN; COMPUTATIONAL RESULTS; EMPIRICAL EVALUATIONS; FINITE HORIZONS; MARKOV DECISION PROCESSES; REGRET BOUNDS;

LEARNING ALGORITHMS;

EID: 85048551505     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Araya, Mauricio, Buffet, Olivier, and Thomas, Vincent. Near-optimal brl using optimistic local transitions. arXiv preprint arXiv:1206.4613, 2012.
2. Asmuth, John, Li, Lihong, Littman, Michael L, Nouri, Ali, and Wingate, David. A Bayesian sampling approach to exploration in reinforcement learning. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 19-26. AUAI Press, 2009.
3. Bartlett, Peter L. and Tewari, Ambuj. REGAL: A regu- larization based algorithm for reinforcement learning in weakly communicating MDPs. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI2009), pp. 35-42, June 2009.
4. Bellman, Richard and Kalaba, Robert. On adaptive control processes. IRE Transactions on Automatic Control, 4(2): 1-9, 1959.
5. Brafman, Ronen I. and Tennenholtz, Moshe. R-max - A general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3:213-231, 2002.
6. Buldygin, Valerii V and Kozachenko, Y V. Sub-gaussian random variables. Ukrainian Mathematical Journal, 32 (6):483-489, 1980.
7. Burnetas, Apostolos N and Katehakis, Michael N. Optimal adaptive policies for Markov decision processes. Mathematics of Operations Research, 22(1):222-255, 1997.
8. Dani, Varsha, Hayes, Thomas P., and Kakadc, Sham M. Stochastic linear optimization under bandit feedback. In COLT, pp. 355-366, 2008.
9. Dann, Christoph and Brunskill, Emma. Sample complexity of episodic fixed-horizon reinforcement learning. In Advances in Neural Information Processing Systems, pp. TBA, 2015.
10. Filippi, Sarah, Cappe, Olivier, and Garivier, Aurdlien. Optimism in reinforcement learning and kullback-leibler divergence. In Communication, Control, and Computing (Allerton), 2010 48th Annual Allerton Conference on, pp. 115-122. IEEE, 2010.
11. Fonteneau, Raphael, Korda, Nathan, and Munos, Remi. An optimistic posterior sampling strategy for Bayesian reinforcement learning. In NIPS 2013 Workshop on Bayesian Optimization (Bayes0pt2013), 2013.
12. Gopalan, Aditya and Mannor, Shie. Thompson sampling for learning parameterized Markov decision processes. arXiv preprint arXiv:1406.7498, 2014.
13. Hadar, Josef and Russell, William R. Rules for ordering uncertain prospects. The American Economic Review, pp. 25-34, 1969.
14. Jaksch, Thomas, Ortner, Ronald, and Auer, Peter. Near- optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11:1563-1600, 2010.
15. Kakade, Sham. On the Sample Complexity of Reinforcement Learning. PhD thesis, University College London, 2003.
16. Kearns, Michael J. and Singh, Satinder P. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49(2-3):209-232, 2002.
17. Kolter, J Zico and Ng, Andrew Y. Near-Bayesian exploration in polynomial time. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 513-520. ACM, 2009.
18. Lattimore, Tor and Hutter, Marcus. PAC bounds for discounted MDPs. In Algorithmic learning theory, pp. 320- 334. Springer, 2012.
19. Munos, Remi. From bandits to monte-carlo tree search: The optimistic principle applied to optimization and planning. 2014.
20. Osband, Ian. Deep Exploration via Randomized Value Functions. PhD thesis, Stanford, 2016.
21. Osband, Ian and Van Roy, Benjamin. Model-based reinforcement learning and the eluder dimension. In Advances in Neural Information Processing Systems, pp. 1466-1474, 2014a.
22. Osband, Ian and Van Roy, Benjamin. Near-optimal reinforcement learning in factored MDPs. In Advances in Neural Information Processing Systems, pp. 604-612, 2014b.
23. Osband, Ian and Van Roy, Benjamin. On lower bounds for regret in reinforcement learning. arXiv preprint arXiv:1608.02732, 2016.
24. Osband, Ian and Van Roy, Benjamin. Gaussian-dirichlet posterior dominance in sequential learning. arXiv preprint arXiv:1702.04126, 2017.
25. Osband, Ian, Russo, Daniel, and Van Roy, Benjamin. (More) Efficient reinforcement learning via posterior sampling. In NIPS, pp. 3003-3011. Curran Associates, Inc., 2013.
26. Osband, Ian, Van Roy, Benjamin, and Wen, Zheng. Generalization and exploration via randomized value functions. arXiv preprint arXiv:1402.0635, 2014.
27. Russo, Daniel and Van Roy, Benjamin. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4): 1221-1243, 2014.
28. Strehl, Alexander L and Littman, Michael L. A theoretical analysis of model-based interval estimation. In Proceedings of the 22nd international conference on Machine learning, pp. 856-863. ACM, 2005.
29. Strehl, Alexander L, Li, Lihong, Wiewiora, Eric, Lang-Ford, John, and Littman, Michael L. PAC model-free reinforcement learning. In ICML, pp. 881-888, 2006.
30. Strens, Malcolm J. A. A Bayesian framework for reinforcement learning. In ICML, pp. 943-950, 2000.
31. Sutton, Richard and Barto, Andrew. Reinforcement Learning: An Introduction. MIT Press, March 1998.
32. Szita, Istvdn and Szepesvsri, Csaba. Model-based reinforcement learning with nearly tight exploration complexity bounds. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1031- 1038, 2010.
33. Thompson, W.R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
34. Vlassis, Nikos, Ghavamzadeh, Mohammad, Mannor, Shie, and Poupart, Pascal. Bayesian reinforcement learning. In Reinforcement Learning, pp. 359-386. Springer, 2012.