Statistical inference in two-stage online controlled experiments with treatment selection and validation

WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web

Volumn , Issue , 2014, Pages 609-618

  Deng, Alex ^a   Li, Tianxi ^b   Guo, Yu ^a  

^a MICROSOFT   (United States)

^b UNIVERSITY OF MICHIGAN   (United States)

Author keywords

A B testing; Bias correction; Controlled experiment; Empirical Bayes; Meta analysis; Search quality evaluation

Indexed keywords

DECISION MAKING; ONLINE SYSTEMS;

A/B TESTING; BIAS CORRECTION; CONTROLLED EXPERIMENT; EMPIRICAL BAYES; META ANALYSIS; SEARCH QUALITY;

QUALITY CONTROL;

EID: 84909620005     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2566486.2568028     Document Type: Conference Paper

References (22)

1. Yoav Benjamini. Simultaneous and selective inference: current successes and future challenges. Biometrical Journal, 52(6):708-721, 2010.
2. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), pages 289-300, 1995.
3. Brian Christian. The a/b test: Inside the technology that's changing the rules of business, April 2012. URL http://www.wired.com/business/2012/04/-rfpart-abtesting/.
4. Anirban DasGupta. Asymptotic Theory of Statistics and Probability. Springer, 2008.
5. Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the sixth ACM international conference on Web search and data mining, pages 123-132. ACM, 2013.
6. Bradley Efron. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, volume 1. Cambridge University Press, 2010.
7. Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):1602-1614, 2011.
8. Bradley Efron and Carl Morris. Stein's estimation rule and its competitors-an empirical bayes approach. Journal of the American Statistical Association, 68(341):117-130, 1973. ISSN 01621459.
9. Sir Ronald Aylmer Fisher, Statistiker Genetiker, Ronald Aylmer Fisher, Statistician Genetician, Great Britain, Ronald Aylmer Fisher, and Statisticien Généti-cien. Statistical methods for research workers, volume 14. Oliver and Boyd Edinburgh, 1970.
10. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
11. W. James and James Stein. Estimation with quadratic loss. In Jerzy Neyman, editor, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, pages 361-379. University of California Press, 1961.
12. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled experiments on the web: survey and practical guide. Data Mining Knowledge Discovery, 18:140-181, 2009.
13. Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining, 2012.
14. Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker nd Ya Xu, and Nils Pohlmann. Online controlled experiments at large scale. Proceedings of he 19th Conference on Knowledge Discovery and Data Mining, 2013.
15. Jim Manzi. Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. Basic Books, 2012.
16. Filip Radlinski and Nick Craswell. Optimized interleaving for online retrieval evaluation. In Proceedings of the sixth ACM international conference on Web search and data mining, pages 245-254. ACM, 2013.
17. Joseph P Romano. Testing statistical hypotheses. Springer, 2005.
18. Charles Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, Calif., volume 1, pages 197-206. University of California Press, Berkeley, Calif., 1956.
19. Student. The probable error of a mean. Biometrika, 6:1-25, 1908.
20. Larry E Toothaker. Multiple comparison procedures. Number 89. Sage, 1993.
21. Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2003.
22. Bernard L Welch. The generalization of student's' problem when several different population variances are involved. Biometrika, 34(1/2):28-35, 1947.