Sample-efficient nonstationary policy evaluation for contextual bandits

Uncertainty in Artificial Intelligence - Proceedings of the 28th Conference, UAI 2012

Volumn , Issue , 2012, Pages 247-254

  Dud'Ik, Miroslav ^a   Erhan, Dumitru ^b   Langford, John ^a   Li, Lihong ^a  

^a MICROSOFT RESEARCH   (United States)

^b YAHOO RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BIAS-VARIANCE TRADEOFFS; CONTEXTUAL BANDITS; IMPORTANCE WEIGHTING; LEARNING PROBLEM; LEARNING SETTINGS; NONSTATIONARY; POLICY EVALUATION; REAL-WORLD;

ARTIFICIAL INTELLIGENCE;

EID: 84885967848     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Arthur Asuncion and David J. Newman. UCI machine learning repository, 2007. http://www.ics.uci.edu/ -mlearn/MLRepository.html.
2. Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.
3. Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Computing, 32(1):48-77, 2002.
4. Alina Beygelzimer and John Langford. The offset tree for learning with partial labels. In KDD, pages 129-138, 2009.
5. Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandit algorithms with supervised learning guarantees. In AISTATS, 2011.
6. Claes M. Cassel, Carl E. Sarndal, and Jan H. Wretman. Some results on generalized difference estimation and generalized regression estimation for finite populations. Biometrika, 63:615-620, 1976.
7. David Chan, Rong Ge, Ori Gershony, Tim Hesterberg, and Diane Lambert. Evaluating online ad campaigns in a pipeline: Causal models at scale. In KDD, 2010.
8. Miroslav Dud?k, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In ICML, 2011.
9. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-RuiWang, and Chih-Jen Lin. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871-1874, 2008.
10. David A. Freedman. On tail probabilities for martingales. Annals of Probability, 3(1):100-118, 1975.
11. D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. J. Amer. Statist. Assoc., 47:663-685, 1952.
12. Sham Kakade, Michael Kearns, and John Langford. Exploration in metric state spaces. In ICML, 2003.
13. Joseph D. Y. Kang and Joseph L. Schafer. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statist. Sci., 22(4):523-539, 2007. With discussions.
14. Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. In ICML, 1998.
15. John Langford, Alexander L. Strehl, and Jennifer Wortman. Exploration scavenging. In ICML, pages 528-535, 2008.
16. David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, 2004.
17. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In WWW, 2010.
18. Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextualbandit-based news article recommendation algorithms. In WSDM, 2011.
19. Jared K. Lunceford and Marie Davidian. Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Statistics in Medicine, 23(19):2937-2960, 2004.
20. Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In ICML, pages 759-766, 2000.
21. James M. Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. J. Amer. Statist. Assoc., 90:122-129, 1995.
22. James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. J. Amer. Statist. Assoc., 89(427):846-866, 1994.
23. Alex Strehl, John Langford, Lihong Li, and Sham Kakade. Learning from logged implicit exploration data. In NIPS, pages 2217-2225, 2011.
24. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, March 1998.