Learning to optimize via posterior sampling

Mathematics of Operations Research

Volumn 39, Issue 4, 2014, Pages 1221-1243

  Russo, Daniel ^a   Van Roy, Benjamin ^a  

^a Stanford University   (United States)

Author keywords

Multiarmed bandits; Online optimization; Thompson sampling

Indexed keywords

RESEARCH;

EXPLORATION AND EXPLOITATION; GENERALIZED LINEAR MODEL; MULTI ARMED BANDIT; MULTI-ARMED BANDIT PROBLEM; ONLINE OPTIMIZATION; PROBABILITY MATCHING; THOMPSON SAMPLINGS; UPPER CONFIDENCE BOUND;

COMPUTER APPLICATIONS;

EID: 84908172119     PISSN: 0364765X     EISSN: 15265471     Source Type: Journal    
DOI: 10.1287/moor.2014.0650     Document Type: Article

References (36)

1. Abbasi-Yadkori Y, Antos A, Szepesvári C (2009) Forced-exploration based algorithms for playing in stochastic linear bandits. COLT Workshop: On-line Learn. Limited Feedback.
2. Abbasi-Yadkori Y, Pál D, Szepesvári C (2011) Improved algorithms for linear stochastic bandits. Shawe-Taylor J, Zemel RS, Bartlett PL, Pereira FCN, Weinberger KQ, eds. Adv. Neural Inform. Processing Systems (NIPS), (Currant Associates, Red Hook, NY), 2312-2320.
3. Abbasi-Yadkori Y, Pal D, Szepesvári C (2012) Online-to-confidence-set conversions and application to sparse stochastic bandits. Lawrence ND, Girolami MA, eds. Proc. 15th Internat. Conf. Artificial Intelligence Statist. (AISTATS), JMLR Workshop and Conference Proceedings, Vol. 22, 1-9.
4. Agrawal S, Goyal N (2012) Analysis of Thompson sampling for the multi-armed bandit problem. Mannor S, Srebro N, Williamson RC, eds. Proc. 25th Ann. Conf. Learn. Theory, JMLR Workshop and Conference Proceedings, Vol. 23, 39.1-39.26
5. Agrawal S, Goyal N (2012) Further optimal regret bounds for Thompson sampling. arXiv preprint arXiv:1209.3353.
6. Agrawal S, Goyal N (2012) Thompson sampling for contextual bandits with linear payoffs. arXiv preprint arXiv:1209.3352.
7. Amin K, Kearns M, Syed U (2011) Bandits, query learning, and the haystack dimension. Kakade SM, von Luxburg U, eds. Proc. 24th Annual Conf. Learn. Theory (COLT), JMLR Workshop and Conference Proceedings, Vol. 19, 87-106.
8. Audibert J-Y, Bubeck S (2009) Minimax policies for adversarial and stochastic bandits. Proc. 22th Annual Conf. Learn. Theory (COLT), (Omnipress, Madison, WI), 773-818.
9. Auer P, Cesa-Bianchi N, Fischer P (2002) Finite-time analysis of the multiarmed bandit problem. Machine Learn. 47(2):235-256.
10. Beygelzimer A, Langford J, Li L, Reyzin L, Schapire RE (2011) Contextual bandit algorithms with supervised learning guarantees. Proc. 14th Internat. Conf. Artificial Intelligence Statist. (AISTATS), JMLR Workshop and Conference Proceedings, Vol. 15, 19-26.
11. Bubeck S, Cesa-Bianchi N (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint arXiv:1204.5721.
12. Bubeck S, Liu C-Y (2013) Prior-free and prior-dependent regret bounds for Thompson sampling. Adv. Neural Inform. Processing Systems (NIPS), 638-646.
13. Bubeck S, Munos R, Stoltz G, Szepesvári C (2011) X -armed bandits. J. Machine Learn. Res. 12:1655-1695.
14. Cappé O, Garivier A, Maillard O-A, Munos R, Stoltz G (2013) Kullback-Leibler upper confidence bounds for optimal sequential allocation. Ann. Statist. 41(3):1516-1541.
15. Chapelle O, Li L (2011) An empirical evaluation of Thompson sampling. Shawe-Taylor J, Zemel RS, Bartlett PL, Pereria F, Weinberger KQ, eds. Adv. Neural Inform. Processing Systems (NIPS) (Currant Associates, Red Hook, NY), 2249-2257.
16. Dani V, Hayes TP, Kakade SM (2008) Stochastic linear optimization under bandit feedback. Proc. 21st Annual Conf. Learn. Theory (COLT) (Omnipress, Madison, WI), 355-366.
17. Deshpande Y, Montanari A (2012) Linear bandits in high dimension and recommendation systems. 50th Annual Allerton Conf. Communication, Control, Comput. (IEEE, Piscataway, NJ), 1750-1754.
18. Filippi S, Cappé O, Garivier A, Szepesvári C (2010) Parametric bandits: The generalized linear case. Lafferty J, Williams C, Shawe-Taylor J, Zemel RS, Culotta A, eds. Adv. Neural Inform. Processing Systems (NIPS), (Currant Associates, Red Hook, NY), 586-594.
19. Gittins JC, Jones DM (1979) A dynamic allocation index for the discounted multiarmed bandit problem. Biometrika 66(3):561-565.
20. Gittins J, Glazebrook K, Weber R (2011) Multi-Armed Bandit Allocation Indices (John Wiley & Sons, Chichester, UK).
21. Gopalan A, Mannor S, Mansour Y (2013) Thompson sampling for complex bandit problems. arXiv preprint arXiv:1311.0466.
22. Kauffmann E, Korda N, Munos R (2012) Thompson sampling: An asymptotically optimal finite time analysis. Bshouty NH, Stoltz G, Vayatis N, Zeugmann T, eds., Algorithmic Learn. Theory, Lecture Notes in Computer Science, Vol. 7568 (Springer-Verlag, Berlin, Heidelberg), 586-594.
23. Kleinberg R, Slivkins A, Upfal E (2008) Multi-armed bandits in metric spaces. Proc. 40th ACM Sympos. Theory Comput. (ACM, New York), 681-690.
24. Korda N, Kaufmann E, Munos R (2013) Thompson sampling for one-dimensional exponential family bandits. Burges CJC, Bottou L, Ghahramani Z, Weinberger KQ, eds. Adv. Neural Inform. Processing Systems, 1448-1456.
25. Lai TL (1987) Adaptive treatment allocation and the multi-armed bandit problem. Ann. Statist. 15(3):1091-1114.
26. Lai TL, Robbins H (1985) Asymptotically efficient adaptive allocation rules. Adv. Appl. Math. 6(1):4-22.
27. Lehmann EL, Casella G (1998) Theory of Point Estimation, 2nd ed. (Springer-Verlag, New York).
28. Li L (2013) Generalized Thompson sampling for contextual bandits. arXiv preprint arXiv:1310.7163.
29. Li L, Chapelle O (2012) Open problem: Regret bounds for Thompson sampling. Mannor S, Srebro B, Williamson RC, eds. Proc. 25th Annual Conf. Learn. Theory (COLT), JMLR Workshop and Conference Proceedings, Vol. 23, 43.1-43.3.
30. May BC, Korda N, Lee A, Leslie DS (2012) Optimistic Bayesian sampling in contextual-bandit problems. J. Machine Learn. Res. 13(1):2069-2106.
31. Rusmevichientong P, Tsitsiklis JN (2010) Linearly parameterized bandits. Math. Oper. Res. 35(2):395-411.
32. Ryzhov IO, Powell WB, Frazier PI (2012) The knowledge gradient algorithm for a general class of online learning problems. Oper. Res. 60(1):180-195.
33. Sahni A (1974) Computationally related problems. SIAM J. Comput. 3(4):262-279.
34. Scott SL (2010) A modern Bayesian look at the multi-armed bandit. Appl. Stochastic Models Bus. Indust. 26(6):639-658.
35. Srinivas N, Krause A, Kakade SM, Seeger M (2012) Information-theoretic regret bounds for Gaussian process optimization in the bandit setting. Inform. Theory, IEEE Trans. 58(5):3250-3265.
36. Valko M, Carpentier A, Munos R (2013) Stochastic simultaneous optimistic optimization. Dasgupta S, Mcallester D, eds. Proc. 30th Internat. Conf. Machine Learn. (ICML-13), JMLR Workshop and Conference Proceedings, Vol. 28, 19-27.