Toward predicting the outcome of an A/B experiment for search relevance

WSDM 2015 - Proceedings of the 8th ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2015, Pages 37-46

  Li, Lihong ^a   Kimy, Jin Young ^a   Zitouni, Imed ^a  

^a MICROSOFT   (United States)

Author keywords

Contextual bandits; Evaluation; Experimentation; Information retrieval; Web search

Indexed keywords

DATA MINING; ECONOMIC AND SOCIAL EFFECTS; FUNCTION EVALUATION; INFORMATION RETRIEVAL; ONLINE SYSTEMS; RECOMMENDER SYSTEMS; WEBSITES;

CONTEXTUAL BANDITS; CONTROLLED EXPERIMENT; EVALUATION; EXPERIMENTATION; OFFLINE EVALUATION; ON-LINE EXPERIMENTS; RANKING FUNCTIONS; WEB SEARCHES;

SEARCH ENGINES;

EID: 84928739493     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2684822.2685311     Document Type: Conference Paper

References (29)

1. nd edition, 2011.
2. Nicholas J. Belkin. Some(what) grand challenges for information retrieval. ACM SIGIR Forum, 42(1):47-54, 2008.
3. Léon Bottou, Jonas Peters, Joaquin Quinonero Candela, Denis Xavier Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14:3207-3260, 2013.
4. Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, pages 89-96, 2005
5. nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 67-74, 2009.
6. Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. Large scale validation and analysis of interleaved search evaluation. ACM Transactions on Information Science, 30(1), 2012.
7. Olivier Chapelle, Eren Manavoglu, and Romer Rosales. Simple and scalable response prediction for display advertising. ACM Transactions on Intelligent Systems and Technology. To appear.
8. th International Conference on World Wide Web, pages 1-10, 2009.
9. th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 493-502, 2013.
10. st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 331-338, 2008.
11. th International Conference on World Wide Web, pages 11-20, 2009.
12. th ACM International Conference on Web Search and Data Mining, pages 183-192, 2013.
13. st ACM International Conference on Information and Knowledge Management, pages 1779-1783, 2012.
14. Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
15. Gabriella Kazai and Homer Sung. Dissimilarity based query selection for efficient preference based IR evaluation. In Proceedings of the European Conference on Information Retrieval, pages 172-183, 2014.
16. Ron Kohavi, Roger Longbotham, Dan Sommer field, and Randal M. Henne. Controlled experiments on the web: Survey and practical guide. Data Minining and Knowledge Discovery, 18:140-181, 2009.
17. Diane Lambert and Daryl Pregibon. More bang for their bucks: Assessing new features for online advertisers. SIGKDD Explorations, 9(2):100-107, 2007.
18. th International Conference on Machine Learning, pages 528-535, 2008.
19. Lihong Li, Shunbao Chen, Ankur Gupta, and Jim Kleban. Counterfactual analysis of click metrics for search engine optimization. Technical Report MSR-TR-2014-32, Microsoft Research, 2014.
20. th International Conference on World Wide Web, pages 661-670, 2010.
21. th International Conference on Web Search and Data Mining, pages 297-306, 2011.
22. Lihong Li, Remi Munos, and Csaba Szepesvári. On minimax optimal off-policy policy evaluation. Technical report, Microsoft Research, 2014.
23. Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample-variance penalization. In Proceedings of the Twenty-Second Conference on Learning Theory, pages 247-254, 2009.
24. th ACM International Conference on Web Search and Data Mining, pages 245-254, 2013.
25. Stephen Robertson. On the history of evaluation in IR. Journal of Information Science, 34(4):439-456, 2008.
26. Alexander L. Strehl, John Langford, Lihong Li, and Sham M. Kakade. Learning from logged implicit exploration data. In Advances in Neural Information Processing Systems 23, pages 2217-2225, 2011.
27. nd ACM International Conference on Information & Knowledge Management, pages 1587-1594, 2013.
28. th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 225-231, 2001.
29. Zhaohui Zheng, Hongyuan Zha, Tong Zhang, Olivier Chapelle, Keke Chen, and Gordon Sun. A general boosting method and its application to learning ranking functions for web search. In Advances in Neural Information Processing Systems 20, pages 1000-1007, 2008.