Uncertainty in online experiments with dependent data: An evaluation of bootstrap methods

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn Part F128815, Issue , 2013, Pages 1303-1311

  Bakshy, Eytan ^a   Eckles, Dean ^a  

^a FACEBOOK   (United States)

Author keywords

A A tests; A B testing; Bootstrapping; Causal inference; Field experiments; Random effects; User item data

Indexed keywords

DATA MINING; ERRORS; ONLINE SYSTEMS; STATISTICAL METHODS;

A/B TESTING; BOOTSTRAPPING; CAUSAL INFERENCES; FIELD EXPERIMENT; RANDOM EFFECTS; USER-ITEM DATA;

BEHAVIORAL RESEARCH;

EID: 84954118473     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2487575.2488218     Document Type: Conference Paper

References (22)

1. A. Agresti. Categorical Data Analysis. Wiley-Interscience, 2nd edition, July 2002.
2. R. H. Baayen, D. J. Davidson, and D. M. Bates. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390-412, 2008.
3. E. Bakshy, D. Eckles, R. Yan, and I. Rosenn. Social influence in social advertising: Evidence from field experiments. In Proceedings of the 13th ACM Conference on Electronic Commerce, pages 146-161. ACM, 2012.
4. G. E. Box, J. S. Hunter, and W. G. Hunter. Statistics for Experimenters: Design, Innovation, and Discovery, volume 13. Wiley Online Library, 2005.
5. R. L. Brennan, D. J. Harris, and B. A. Hanson. The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts. American College Testing Program, 1987.
6. A. Cameron, J. Gelbach, and D. Miller. Robust inference with multi-way clustering. Journal of Business & Economic Statistics, 29(2):238-249, 2011.
7. T. Crook, B. Frasca, R. Kohavi, and R. Longbotham. Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1105-1114. ACM, 2009.
8. B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1-26, 1979.
9. A. Gelman and F. Tuerlinckx. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15(3):373-390, 2000.
10. A. S. Gerber and D. P. Green. Field Experiments: Design, Analysis, and Interpretation. WW Norton, 2012.
11. R. Kohavi, A. Deng, B. Frasca, R. Longbotham, T. Walker, and Y. Xu. Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 786-794. ACM, 2012.
12. R. Kohavi, R. Longbotham, D. Sommerfield, and R. Henne. Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1):140-181, 2009.
13. W. Lin. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique. The Annals of Applied Statistics, 7(1):295-318, 2013.
14. P. McCullagh. Resampling and exchangeable arrays. Bernoulli, 6(2):285-301, 2000.
15. L. W. Miratrix, J. S. Sekhon, and B. Yu. Adjusting treatment effect estimates by post-stratification in randomized experiments. JR Stat. Soc. Ser. B. Stat. Methodol. To appear, 2012.
16. A. B. Owen. The pigeonhole bootstrap. The Annals of Applied Statistics, 1(2):386-411, 2007.
17. A. B. Owen and D. Eckles. Bootstrapping data arrays of arbitrary order. The Annals of Applied Statistics, 6(3):895-927, 2012.
18. N. C. Oza and S. Russell. Experimental comparisons of online and batch versions of bagging and boosting. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 359-364. ACM, 2001.
19. D. B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688-701, 1974.
20. D. B. Rubin. The Bayesian bootstrap. The Annals of Statistics, 9(1):130-134, 1981.
21. S. L. Scott. A modern Bayesian look at the multi-Armed bandit. Applied Stochastic Models in Business and Industry, 26(6):639-658, 2010.
22. S. R. Searle, G. Casella, C. E. McCulloch, et al. Variance Components. Wiley New York, 1992.