Objective bayesian two sample hypothesis testing for online controlled experiments

WWW 2015 Companion - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 923-928

  Deng, Alex ^a  

^a MICROSOFT   (United States)

Author keywords

A B testing; Bayesian statistics; Controlled experiments; Empirical Bayes; Multiple testing; Objective Bayes; Optional stopping; Prior

Indexed keywords

BAYESIAN NETWORKS; WORLD WIDE WEB;

A/B TESTING; BAYESIAN STATISTICS; CONTROLLED EXPERIMENT; EMPIRICAL BAYES; MULTIPLE TESTING; OBJECTIVE BAYES; OPTIONAL STOPPING; PRIOR;

STATISTICAL TESTS;

EID: 84968616006     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2740908.2742563     Document Type: Conference Paper

References (33)

1. Benjamini, Y. and Hochberg, Y. [1995], Controlling the false discovery rate: A practical and powerful approach to multiple testing', J. R. Stat. Soc. Ser. B pp. 289-300.
2. Berger, J. [2006], The case for objective Bayesian analysis', Bay-esian Anal. (3), 385-402.
3. Berger, J. O. and Bayarri, M. J. [2004], The Interplay of Bay-esian and Frequentist Analysis', Stat. Sci. 19 (1), 58-80.
4. Berger, J. O. and Wolpert, R. L. [1988], The Likelihood Princi-ple.
5. Dempster, A. P., Laird, N. M. and Rubin, D. B. [1977], Maxi-mum likelihood from incomplete data via the EM algorithm', J. R. Stat. Soc. Ser. B 39 (1), 1-38.
6. Deng, A. and Hu, V. [2015], Diluted Treatment effiect Estimation for Trigger Analysis in Online Controlled Experiments, in Proc. 8th ACM Int. Conf. Web search data Min. '.
7. Deng, A., Li, T. and Guo, Y. [2014], Statistical Inference in Two-stage Online Controlled Experiments with Treatment Se-lection and Validation, in Proc. 23rd Int. Conf. World Wide Web', WWW '14, pp. 609-618.
8. Deng, A., Xu, Y., Kohavi, R. and Walker, T. [2013], Improving the sensitivity of online controlled experiments by utilizing pre-experiment data, in Proc. 6th ACM Int. Conf. Web search data Min. ', ACM, pp. 123-132.
9. Efron, B. [1986], Why isn't everyone a Bayesian?', Am. Stat. 40 (1), 1-5.
10. Efron, B. [2010], Large-scale inference: empirical Bayes meth-ods for estimation, testing, and prediction, Vol. 1, Cambridge University Press.
11. Efron, B. [2011], Tweedie's formula and selection bias', J. Am. Stat. Assoc. 106 (496), 1602-1614.
12. Efron, B. [2013a], A 250-year argument: belief, behavior, and the bootstrap', Bull. Am. Math. Soc. 50 (1), 129-146.
13. Efron, B. [2013b], Empirical Bayes modeling, computation, and accuracy, Technical report.
14. Efron, B. [2014], Frequentist accuracy of Bayesian estimates', J. R. Stat. Soc. Ser. B.
15. Gelman, A. and Hill, J. [2006], Data analysis using regres-sion and multilevel/hierarchical models, Cambridge University Press.
16. Johnson, V. E. [2013], Revised standards for statistical evi-dence', Proc. Natl. Acad. Sci. 110 (48), 19313-19317.
17. Kass, R. and Raftery, A. [1995], Bayes factors', J. Am. Stat. Assoc. 90 (430), 773-795.
18. Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T. and Xu, Y. [2012], Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained', Proc. 18th Conf. Knowl. Discov. Data Min.
19. Kohavi, R., Deng, A., Frasca, B., Xu, Y., Walker, T. and Pohlmann, N. [2013], Online Controlled Experiments at Large Scale', Proc. 19th Conf. Knowl. Discov. Data Min.
20. Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. [2014], Seven rules of thumb for web site experimenters, in Proc. 20th Conf. Knowl. Discov. Data Min. ', KDD '14, New York, USA, pp. 1857-1866.
21. Kohavi, R., Longbotham, R., Sommer field, D. and Henne, R. M.[2009], Controlled Experiments on the Web: survey and practi-cal guide', Data Min. Knowl. Discov. 18, 140-181.
22. Kruschke, J. K. [2013], Bayesian estimation supersedes the t test. ', J. Exp. Psychol. Gen. 142 (2), 573.
23. Masson, M. E. J. [2011], A tutorial on a practical Bayesian al-ternative to null-hypothesis significance testing. ', Behav. Res. Methods 43 (3), 679-90.
24. Muller, P., Parmigiani, G. and Rice, K. [2006], FDR and Bay-esian multiple comparisons rules, in 8th World Meet. Bayesian Stat. ', Vol. 0.
25. Murphy, K. P. [2012], Machine learning: A probabilistic per-spective, MIT press.
26. Park, T. and Casella, G. [2008], The bayesian lasso', J. Am. Stat. Assoc. 103 (482), 681-686.
27. Rouder, J. N. [2014], Optional stopping: no problem for Bayesians. ', Psychon. Bull. Rev. 21 (March), 301-8.
28. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D. and Iverson, G. [2009], Bayesian t tests for accepting and rejecting the null hypothesis. ', Psychon. Bull. Rev. 16 (2), 225-37.
29. Scott, J. and Berger, J. [2006], An exploration of aspects of Bayesian multiple testing', J. Stat. Plan. Inference.
30. Sellke, T., Bayarri, M. and Berger, J. [2001], Calibration of values for testing precise null hypotheses', Am. Stat. 55 (1), 62-71.
31. Senn, S. [2008], A note concerning a selection "paradox" of Dawid's', Am. Stat. Assoc. 62 (3), 206-210.
32. Student [1908], The probable error of a mean', Biometrika 6, 1-25.
33. Tang, D., Agarwal, A., O'Brien, D. and Meyer, M. [2010], Over-lapping Experiment Infrastructure: More, Better, Faster Exper-imentation', Proc. 16th Conf. Knowl. Discov. Data Min.