Stochastic multi-armed-bandit problem with non-stationary rewards

Advances in Neural Information Processing Systems

Volumn 1, Issue January, 2014, Pages 199-207

  Besbes, Omar ^a   Gur, Yonatan ^b   Zeevi, Assaf ^a  

^a Columbia University ^*  (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ECONOMIC AND SOCIAL EFFECTS; STOCHASTIC SYSTEMS;

DIRECT LINKS; MATHEMATICAL TRACTABILITY; MULTI ARMED BANDIT; MULTI-ARMED BANDIT PROBLEM; NONSTATIONARY; TEMPORAL UNCERTAINTY; TRADE OFF;

INFORMATION SCIENCE;

EID: 84937906754     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25:285-294, 1933.
2. H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 55:527-535, 1952.
3. M. Zelen. Play the winner rule and the controlled clinical trials. Journal of the American Statistical Association, 64:131-146, 1969.
4. D. Bergemann and J. Valimaki. Learning and strategic pricing. Econometrica, 64:1125-1149, 1996.
5. D. Bergemann and U. Hege. The financing of innovation: Learning and stopping. RAND Journal of Economics, 36(4):719-752, 2005.
6. B. Awerbuch and R. D. Kleinberg. Addaptive routing with end-to-end feedback: distributed learning and geometric approaches. In Proceedings of the 36th ACM Symposiuim on Theory of Computing (STOC), pages 45-53, 2004.
7. R. D. Kleinberg and T. Leighton. The value of knowing a demand curve: Bounds on regret for online posted-price auctions. In Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 594-605, 2003.
8. F. Caro and G. Gallien. Dynamic assortment with demand learning for seasonal consumer goods. Management Science, 53:276-292, 2007.
9. S. Pandey, D. Agarwal, D. Charkrabarti, and V. Josifovski. Bandits for taxonomies: A model-based approach. In SIAM International Conference on Data Mining, 2007.
10. D. A. Berry and B. Fristedt. Bandit problems: sequential allocation of experiments. Chapman and Hall, 1985.
11. J. C. Gittins. Multi-Armed Bandit Allocation Indices. John Wiley and Sons, 1989.
12. N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, Cambridge, UK, 2006.
13. T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6:4-22, 1985.
14. J. C. Gittins and D. M. Jones. A dynamic allocation index for the sequential design of experiments. North-Holland, 1974.
15. J. C. Gittins. Bandit processes and dynamic allocation indices (with discussion). Journal of the Royal Statistical Society, Series B, 41:148-177, 1979.
16. P. Whittle. Arm acquiring bandits. The Annals of Probability, 9:284-292, 1981.
17. P. Whittle. Restless bandits: Activity allocation in a changing world. Journal of Applied Probability, 25A:287-298, 1988.
18. C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of optimal queueing network control. In Structure in Complexity Theory Conference, pages 318-322, 1994.
19. D. Bertsimas and J. Nino-Mora. Restless bandits, linear programming relaxations, and primal dual index heuristic. Operations Research, 48(1):80-90, 2000.
20. S. Guha and K. Munagala. Approximation algorithms for partial-information based stochastic control with markovian rewards. In 48th Annual IEEE Symposium on Fundations of Computer Science (FOCS), pages 483-493, 2007.
21. R. Ortner, D. Ryabko, P. Auer, and R. Munos. Regret bounds for restless markov bandits. In Algorithmic Learning Theory, pages 214-228. Springer Berlin Heidelberg, 2012.
22. M. G. Azar, A. Lazaric, and E. Brunskill. Stochastic optimization of a locally smooth function under correlated bandit feedback. arXiv preprint arXiv:1402.0562, 2014.
23. D. Blackwell. An analog of the minimax theorem for vector payoffs. Pacific Journal of Mathematics, 6:1-8, 1956.
24. J. Hannan. Approximation to bayes risk in repeated plays, Contributions to the Theory of Games, Volume 3. Princeton University Press, Cambridge, UK, 1957.
25. D. P. Foster and R. V. Vohra. Regret in the on-line decision problem. Games and Economic Behaviour, 29:7-35, 1999.
26. O. Besbes, Y. Gur, and A. Zeevi. Non-stationary stochastic optimization. Working paper, 2014.
27. A. Garivier and E. Moulines. On upper-confidence bound policies for switching bandit problems. In Algorithmic Learning Theory, pages 174-188. Springer Berlin Heidelberg, 2011.
28. P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The non-stochastic multi-armed bandit problem. SIAM journal of computing, 32:48-77, 2002.
29. P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-246, 2002.
30. Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci., 55:119-139, 1997.
31. C. Hartland, S. Gelly, N. Baskiotis, O. Teytaud, and M. Sebag. Multi-armed bandit, dynamic environments and meta-bandits. NIPS-2006 workshop, Online trading between exploration and exploitation, Whistler, Canada, 2006.
32. A. Slivkins and E. Upfal. Adapting to a changing environment: The brownian restless bandits. In Proceedings of the 21st Annual Conference on Learning Theory (COLT), pages 343-354, 2008.