Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms

Proceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011

Volumn , Issue , 2011, Pages 297-306

  Li, Lihong ^a   Chu, Wei ^a   Langford, John ^a   Wang, Xuanhui ^a  

^a YAHOO RESEARCH   (United States)

Author keywords

Benchmark dataset; Contextual bandit; Multi armed bandit; Offline evaluation; Recommendation

Indexed keywords

BENCHMARK DATASETS; CONTEXTUAL BANDIT; MULTI-ARMED BANDIT; OFFLINE EVALUATION; RECOMMENDATION;

ALGORITHMS; DATA MINING; INFORMATION RETRIEVAL; ONLINE SYSTEMS; SIMULATORS;

WORLD WIDE WEB;

EID: 79952384747     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1935826.1935878     Document Type: Conference Paper

References (22)

1. Naoki Abe, Alan W. Biermann, and Philip M. Long. Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica, 37(4): 263-293, 2003.
2. Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. Explore/exploit schemes for web content optimization. In Proceedings of the Ninth International Conference on Data Mining, 2009.
3. Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. Spatio-temporal models for estimating click-through rate. In Proceedings of the Eighteenth International Conference on World Wide Web, 2009.
4. Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3: 397-422, 2002.
5. Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3): 235-256, 2002.
6. Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1): 48-77, 2002.
7. Donald A. Berry and Bert Fristedt. Bandit Problems: Sequential Allocation of Experiments. Monographs on Statistics and Applied Probability. Chapman and Hall, 1985.
8. J.C. Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodological), 41:148-177, 1979.
9. Leslie Pack Kaelbling. Associative reinforcement learning: Functions in k-DNF. Machine Learning, 15(3):279-298, 1994.
10. Michael J. Kearns, Yishay Mansour, and Andrew Y. Ng. Approximate planning in large POMDPs via reusable trajectories. In Advances in Neural Information Processing Systems 12, 2000.
11. Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1): 4-22, 1985.
12. John Langford, Alexander L. Strehl, and Jennifer Wortman. Exploration scavenging. In Proceedings of the Twenty-Fifth International Conference on Machine Learning, pages 528-535, 2008.
13. John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. In Advances in Neural Information Processing Systems 20, 2008.
14. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the Nineteenth International Conference on World Wide Web, 2010.
15. Colin McDiarmid. On the method of bounded differences. In J. Siemons, editor, Surveys in Combinatorics, volume 141 of London Mathematical Society Lecture Notes, pages 148-188. Cambridge University Press, 1989.
16. Taesup Moon, Lihong Li, Wei Chu, Ciya Liao, Zhaohui Zheng, and Yi Chang. Online learning for recency search ranking using real-time user feedback. In Proceedings of the Nineteenth International Conference on Knowledge Management, 2010.
17. Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766, 2000.
18. Alexander L. Strehl, John Langford, Lihong Li, and Sham M. Kakadě. Learning from logged implicit exploration data. In Advances in Neural Information Processing Systems 28, 2011.
19. Alexander L. Strehl, Chris Mesterharm, Michael L. Littman, and Haym Hirsh. Experience-efficient learning in associative bandit problems. In Proceedings of the Twenty-Third International Conference on Machine Learning, pages 889-896, 2006.
20. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, March 1998.
21. Chih-Chun Wang, Sanjeev R. Kulkarni, and H. Vincent Poor. Bandit problems with side observations. IEEE Transactions on Automatic Control, 50(3):338-355, 2005.
22. Michael Woodroofe. A one-armed bandit problem with a concomitant variable. Journal of the American Statistics Association, 74(368):799-806, 1979.