Counterfactual estimation and optimization of click metrics in search engines: A case study

WWW 2015 Companion - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 929-934

  Li, Lihong ^a   Chen, Shunbao ^a   Kleban, Jim ^b   Gupta, Ankur ^a  

^a MICROSOFT   (United States)

^b FACEBOOK   (United States)

Author keywords

Contextual bandits; Counterfactual analysis; Experimental design; Information retrieval; Query correction rewriting; Web search

Indexed keywords

DESIGN OF EXPERIMENTS; INFORMATION RETRIEVAL; SEARCH ENGINES; WEBSITES;

CONTEXTUAL BANDITS; CONTROLLED EXPERIMENT; COUNTERFACTUAL ANALYSIS; ESTIMATION AND OPTIMIZATION; INTERACTIVE SYSTEM; ON-LINE EXPERIMENTS; QUERY CORRECTION/REWRITING; WEB SEARCHES;

WORLD WIDE WEB;

EID: 84968593890     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2740908.2742562     Document Type: Conference Paper

References (33)

1. Ricardo Baeza-Yates and Berthier Ribeiro-Neto. Modern Information Retrieval: The Concepts and Technology behind Search. Addison-Wesley Professional, 2nd edition, 2011.
2. Nicholas J. Belkin. Some (what) grand challenges for information retrieval. SIGIR Forum, 42 (1): 47-54, 2008.
3. Alina Beygelzimer and John Langford. The offset tree for learning with partial labels. In KDD, 2009.
4. Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis Xavier Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. JMLR, 14: 3207-3260, 2013.
5. David Chan, Rong Ge, Ori Gershony, Tim Hesterberg, and Diane Lambert. Evaluating online ad campaigns in a pipeline: Causal models at scale. In KDD, 2010.
6. Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. Large scale validation and analysis of interleaved search evaluation. ACM TOIS, 30 (1): 6, 2012.
7. Olivier Chapelle and Ya Zhang. A dynamic Bayesian network click model for Web search ranking. In WWW, 2009.
8. Silviu Cucerzan and Eric Brill. Spelling correction as an iterative process that exploits the collective knowledge of Web users. In EMNLP, 2004.
9. Fernando Diaz. Integration of news content into Web results. In WSDM, 2009.
10. Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In ICML, 2011.
11. Jianfeng Gao, Xiaolong Li, Daniel Micol, Chris Quirk, and Xu Sun. A large scale ranker-based system for search query spelling correction. In COLING, 2010.
12. Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael J. Taylor, Yi-Min Wang, and Christos Faloutsos. Click chain model in Web search. In WWW, 2009.
13. Katja Hofmann, Anne Schuth, Shimon Whiteson, and Maarten de Rijke. Reusing historical interaction data for faster online learning to rank for IR. In WSDM, 2013.
14. Paul W. Holland. Statistics and causal inference. Journal of the American Statistical Association, 81 (6): 945-960, 1986.
15. Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20 (4): 422-446, 2002.
16. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled experiments on the web: Survey and practical guide. Data Minining and Knowledge Discovery, 18: 140-181, 2009.
17. Diane Lambert and Daryl Pregibon. More bang for their bucks: Assessing new features for online advertisers. SIGKDD Explorations, 9 (2): 100-107, 2007.
18. John Langford, Alexander L. Strehl, and Jennifer Wortman. Exploration scavenging. In ICML, 2008.
19. Lihong Li, Shunbao Chen, Ankur Gupta, and Jim Kleban. Counterfactual analysis of click metrics for search engine optimization. Technical Report MSR-TR-2014-32, Microsoft Research, 2014.
20. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In WWW, 2010.
21. Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In WSDM, 2011.
22. Lihong Li, Jin Kim, and Imed Zitouni. Toward predicting the outcome of an A/B experiment for search relevance. In WSDM, 2015.
23. Nikunj C. Oza and Stuart Russell. Online bagging and boosting. In AISTATS, 2001.
24. James Pitokow, Hinrich Schütze, Todd Cass, Rob Cooley, Don Turnbull, Andy Edmonds, Eytan Adar, and Thomas Breuel. Personalized search. CACM, 45 (9): 50-55, 2002.
25. Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Desmond Brand, and Tapas Kanungo. Model characterization curves for federated search using click-logs: Predicting user engagement metrics for the span of feasible operating points. In WWW, 2011.
26. Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, editors. Covariate Shift and Local Learning by Distribution Matching. MIT Press, 2008.
27. Stephen Robertson. On the history of evaluation in IR. Journal of Information Science, 34 (4): 439-456, 2008.
28. Daniel Sheldon, Milad Shokouhi, Martin Szummer, and Nick Craswell. Lambdamerge: merging the results of query reformulations. In WSDM, 2011.
29. Alexander L. Strehl, John Langford, Lihong Li, and Sham M. Kakade. Learning from logged implicit exploration data. In NIPS 23, 2010.
30. Liang Tang, Romer Rosales, Ajit Singh, and Deepak Agarwal. Automatic ad format selection via contextual bandits. In CIKM, 2013.
31. Jaime Teevan, Susan T. Dumais, and Eric Horvitz. Potential for personalization. ACM Trans. Comput.-Hum. Interact., 17 (1), 2010.
32. TREC. The Text REtrieval Conference. http: //trec. nist. gov.
33. Andrew H. Turpin and William Hersh. Why batch and user evaluations do not give the same results. In SIGIR, 2001.