Kullback–leibler upper confidence bounds for optimal sequential allocation

Annals of Statistics

Volumn 41, Issue 3, 2013, Pages 1516-1541

  Cappé, Olivier ^a,b   Garivier, Aurélien ^a,c   Maillard, Odalric Ambrym ^a,d   Munos, Rémi ^a,e   Stoltz, Gilles ^a,f,g  

^a UNIVERSITÉ PAUL SABATIER   (France)

^b CNRS   (France)

^c INSTITUT DE MATHÉMATIQUES DE TOULOUSE   (France)

^d UNIVERSITY OF LEOBEN   (Austria)

^e INRIA   (France)

^f ECOLE NORMALE SUPÉRIEURE   (France)

^g HEC PARIS   (France)

Author keywords

Key words; Kullback Leibler divergence; Phrases. Multi armed bandit problems; Sequential testing; Upper confidence bound

Indexed keywords

EID: 84898949562     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/13-AOS1119     Document Type: Article

References (36)

1. AGRAWAL, R. (1995). Sample mean based index policies with O(log n) regret for the multi-armed bandit problem. Adv. in Appl. Probab. 27 1054–1078. MR1358906
2. AUDIBERT, J.-Y. and BUBECK, S. (2010). Regret bounds and minimax policies under partial monitoring. J. Mach. Learn. Res. 11 2785–2836. MR2738783
3. AUDIBERT, J.-Y., MUNOS, R. and SZEPESVÁRI, C. (2009). Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoret. Comput. Sci. 410 1876–1902. MR2514714
4. AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning 47 235–256.
5. BUBECK, S. and CESA-BIANCHI, N. (2012). Regret analysis of stochastic and nonstochastic multiarmed bandit problems. Foundations and Trends in Machine Learning 5 1–122.
6. BURNETAS, A. N. and KATEHAKIS, M. N. (1996). Optimal adaptive policies for sequential allocation problems. Adv. in Appl. Math. 17 122–142. MR1390571
7. BURNETAS, A. N. and KATEHAKIS, M. N. (1997). Optimal adaptive policies for Markov decision processes. Math. Oper. Res. 22 222–255. MR1436581
8. BURNETAS, A. N. and KATEHAKIS, M. N. (2003). Asymptotic Bayes analysis for the finite-horizon one-armed-bandit problem. Probab. Engrg. Inform. Sci. 17 53–82. MR1959385
9. CAPPÉ, O., GARIVIER, A. and KAUFMANN, E. (2012). py/maBandits: Matlab and Python packages for multi-armed bandits. Available at http://mloss.org/software/view/415/.
10. CAPPÉ, O., GARIVIER, A., MAILLARD, O.-A., MUNOS, R. and STOLTZ, G. (2013). Supplement to “Kullback–Leibler upper confidence bounds for optimal sequential allocation.” DOI:10.1214/13-AOS1119SUPP.
11. CHANG, F. and LAI, T. L. (1987). Optimal stopping and dynamic allocation. Adv. in Appl. Probab. 19 829–853. MR0914595
12. CHOW, Y. S. and TEICHER, H. (1988). Probability Theory: Independence, Interchangeability, Martingales, 2nd ed. Springer, New York. MR0953964
13. DEMBO, A. and ZEITOUNI, O. (1998). Large Deviations Techniques and Applications, 2nd ed. Applications of Mathematics (New York) 38. Springer, New York. MR1619036
14. FILIPPI, S., CAPPÉ, O. and GARIVIER, A. (2010). Optimism in reinforcement learning and Kullback–Leibler divergence. In Proceedings of the 48th Annual Allerton Conference on Communication, Control, and Computing. IEEE Press, Piscataway, NJ.
15. GARIVIER, A. and CAPPÉ, O. (2011). The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th Annual Conference on Learning Theory. JMLR C&WP.
16. GITTINS, J. C. (1979). Bandit processes and dynamic allocation indices (with discussion). J. R. Stat. Soc. Ser. B Stat. Methodol. 41 148–177. MR0547241
17. GITTINS, J., GLAZEBROOK, K. and WEBER, R. (2011). Multi-Armed Bandit Allocation Indices. Wiley, New York.
18. HOEFFDING, W. (1963). Probability inequalities for sums of bounded random variables. J. Amer. Statist. Assoc. 58 13–30. MR0144363
19. HONDA, J. and TAKEMURA, A. (2010). An asymptotically optimal bandit algorithm for bounded support models. In Proceedings of the 23rd Annual Conference on Learning Theory. Omnipress, Madison, WI.
20. HONDA, J. and TAKEMURA, A. (2011). An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Machine Learning 85 361–391.
21. HONDA, J. and TAKEMURA, A. (2012). Finite-time regret bound of a bandit algorithm for the semi-bounded support model. Available at arXiv:1202.2277.
22. KAUFMANN, E., CAPPÉ, O. and GARIVIER, A. (2012). On Bayesian upper confidence bounds for bandit problems. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics 22 592–600. JMLR W&CP.
23. KAUFMANN, E., KORDA, N. and MUNOS, R. (2012). Thompson sampling: An asymptotically optimal finite time analysis. In Proceedings of the 23rd International Conference on Algorithmic Learning Theory 199–213. Springer, New York.
24. LAI, T. L. and ROBBINS, H. (1985). Asymptotically efficient adaptive allocation rules. Adv. in Appl. Math. 6 4–22. MR0776826
25. LEHMANN, E. L. and CASELLA, G. (1998). Theory of Point Estimation, 2nd ed. Springer, New York. MR1639875
26. MAILLARD, O.-A., MUNOS, R. and STOLTZ, G. (2011). A finite-time analysis of multi-armed bandits problems with Kullback–Leibler divergences. In Proceedings of the 24th Annual Conference on Learning Theory. JMLR C&WP.
27. MASSART, P. (2007). Concentration Inequalities and Model Selection. Lecture Notes in Math. 1896. Springer, Berlin. MR2319879
28. OWEN, A. B. (2001). Empirical Likelihood. Chapman & Hall/CRC, Boca Raton, FL.
29. ROBBINS, H. (1952). Some aspects of the sequential design of experiments. Bull. Amer. Math. Soc. (N.S.) 58 527–535. MR0050246
30. THOMPSON, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25 285–294.
31. THOMPSON, W. R. (1935). On the Theory of Apportionment. Amer. J. Math. 57 450–456. MR1507085
32. VAN DER VAART, A. W. (2000). Asymptotic Statistics. Cambridge Univ. Press, Cambridge.
33. WAINWRIGHT, M. J. and JORDAN, M. I. (2008). Graphical models, exponential families, and variational inference. Foundation and Trends in Machine Learning 1 1–305.
34. WALD, A. (1945). Sequential tests of statistical hypotheses. Ann. Math. Statist. 16 117–186. MR0013275
35. WEBER, R. (1992). On the Gittins index for multiarmed bandits. Ann. Appl. Probab. 2 1024–1033. MR1189430
36. WHITTLE, P. (1980). Multi-armed bandits and the Gittins index. J. R. Stat. Soc. Ser. B Stat. Methodol. 42 143–149. MR0583348