Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

WSDM 2013 - Proceedings of the 6th ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2013, Pages 123-132

  Deng, Alex ^a   Xu, Ya ^a   Kohavi, Ron ^a   Walker, Toby ^a  

^a MICROSOFT   (United States)

Author keywords

a b testing; controlled experiment; power; pre experiment; search quality evaluation; sensitivity; variance

Indexed keywords

A/B TESTING; CONTROLLED EXPERIMENT; POWER; PRE-EXPERIMENT; SEARCH QUALITY; SENSITIVITY; VARIANCE;

DATA MINING; DIGITAL STORAGE; INFORMATION RETRIEVAL; INTERNET; WEBSITES;

EXPERIMENTS;

EID: 84874264953     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2433396.2433413     Document Type: Conference Paper

References (22)

1. Soren Asmussen and Peter Glynn. Stochastic Simulation. Springer-Verlag, 2008.
2. George E. P. Box, J. Stuart Hunter, and William G. Hunter. Statistics for experimenters: design, innovation, and discovery. Wiley, 2005.
3. Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. Large-scale validation and analysis of interleaved search evaluation. ACM Trans. Inf. Syst., 30(1): 6:1-6:41, March 2012. ISSN 1046-8188.
4. comScore. comscore releases june 2012 u.s. search engine rankings, June 2012. http://www.comscore.com/Press-Events/Press-Releases/2012/6/comScore- Releases-June-2012-U.S.-Search-Engine-Rankings.
5. M. Davidian, A.A. Tsiatis, and S. Leon. Semiparametric estimation of treatment effect in a pretest-posttest study with missing data. Statistical Science, 20, 2005.
6. Shaojie Deng, Roger Longbotham, TobyWalker, and Ya Xu. Choice of the randomization unit in online controlled experiment. JSM proceedings, 2011.
7. Andrew Gelman and Jennifer Hill. Data analysis using regression and multilevel/hierarchical models. Cambridge University Press, 2006.
8. Geoffrey Keppel, William H. Saufley, and Howard Tokunaga. Introduction to design and analysis: A Student's Handbook. Worth Publishers, 1992.
9. Ron Kohavi, Thomas Crook, and Roger Longbotham. Online experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize., 2009a. http://www.exp-platform.com/Pages/expMicrosoft.aspx.
10. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled experiments on the web: survey and practical guide. Data Mining Knowledge Discovery, 18:140-181, 2009b. http://www.exp-platform.com/Pages/hippo- long.aspx.
11. Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining, 2012. http://www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx.
12. S. Leon, A.A. Tsiatis, and M. Davidian. Semiparametric estimation of treatment effect in a pretest-posttest study. Biometrika, 59, 2003.
13. Jim Manzi. Uncontrolled: The Surprising Payoff of Trial-and- Error for Business, Politics, and Society. Basic Books, 2012.
14. Robert L. Mason, Richard F. Gunst, and James L. Hess. Statistical design and analysis of experiments with applications to engineering and science. Wiley, 1989.
15. Student. The probable error of a mean. Biometrika, 6:1-25, 1908.
16. Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. Overlapping experiment infrastructure: More, better, faster experimentation (presentation). 2010a. URL http://research.google.com/archive/papers/Overlapping-Experiment- Infrastructure-More-Be.pdf.
17. Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. Proceedings of the 16th Conference on Knowledge Discovery and Data Mining, 2010b.
18. Anastasios Tsiatis. Semiparametric Theory and Missing Data. Springer-Verlag, 2006.
19. Anastasios A. Tsiatis, Marie Davidian, Min Zhang, and Xiaomin Lu. Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: A principled yet flexible approach. Statistics in Medicine, 27, 2008.
20. Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2003.
21. Wikipedia. A/b testing. http://en.wikipedia.org/wiki/A/B-testing.
22. Li Yang and Anastasios A. Tsiatis. Efficiency study of estimators for a treatment effect in a pretest-posttest trial. The American Statistician, 55, 2001.