Seven rules of thumb for web site experimenters

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn , Issue , 2014, Pages 1857-1866

  Kohavi, Ron ^a   Deng, Alex ^a   Longbotham, Roger ^b   Xu, Ya ^c  

^a MICROSOFT   (United States)

^b SOUTHWEST JIAOTONG UNIVERSITY   (China)

^c LINKEDIN CORPORATION   (United States)

Author keywords

a b testing; controlled experiments; randomized experiments

Indexed keywords

EID: 84907031426     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2623330.2623341     Document Type: Conference Paper

References (44)

1. Kohavi, Ron and Round, Matt. Front Line Internet Analytics at Amazon.com. [ed.] Jim Sterne. Santa Barbara, CA : s.n., 2004. http://ai.stanford.edu/ ~ronnyk/emetricsAmazon.pdf.
2. McKinley, Dan. Design for Continuous Experimentation: Talk and Slides. [Online] Dec 22, 2012. http://mcfunley.com/design-for-continuous- experimentation.
3. Bakshy, Eytan and Eckles, Dean. Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods. KDD 2013: Proceedings of the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2013.
4. Tang, Diane, et al. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. Proceedings 16th Conference on Knowledge Discovery and Data Mining. 2010.
5. Moran, Mike. Multivariate Testing in Action: Quicken Loan's Regis Hadiaris on multivariate testing. Biznology Blog by Mike Moran. [Online] December 2008. www.biznology.com/2008/12/multivariate-testing-in-action/.
6. Posse, Christian. Key Lessons Learned Building LinkedIn Online Experimentation Platform. Slideshare. [Online] March 20, 2013. http://www.slideshare.net/HiveData/googlecontrolled-experimentationpanelthe- hive.
7. Kohavi, Ron, Crook, Thomas and Longbotham, Roger. Online Experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize. 2009. http://exp-platform.com/expMicrosoft.aspx.
8. Amatriain, Xavier and Basilico , Justin. Netflix Recommendations: Beyond the 5 stars. [Online] April 2012. http://techblog.netflix.com/2012/04/netflix- recommendations-beyond-5-stars.html.
9. McFarland, Colin. Experiment!: Website conversion rate optimization with A/B and multivariate testing. s.l. : New Riders, 2012. 978-0321834607.
10. Smietana, Brandon. Zynga: What is Zynga's core competency? Quora. [Online] Sept 2010. http://www.quora.com/Zynga/What-is-Zyngas-core-competency/ answer/Brandon-Smietana.
11. Kohavi, Ron, et al. Online Controlled Experiments at Large Scale. KDD 2013: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 2013. http://bit.ly/ExPScale.
12. Brutlag, Jake . Speed Matters . Google Research blog. [Online] June 23, 2009. http://googleresearch.blogspot.com/2009/06/speed-matters.html.
13. Sullivan, Nicole . Design Fast Websites. Slideshare. [Online] Oct 14, 2008. http://www.slideshare.net/stubbornella/designing-fast-websites- presentation.
14. Kohavi, Ron, Henne, Randal M and Sommerfield, Dan. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. The Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2007). August 2007, pp. 959-967. http://www.exp-platform.com/ Documents/GuideControlledExperiments.pdf.
15. Ries, Eric. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses . s.l. : Crown Business, 2011. 978-0307887894.
16. Kohavi, Ron, et al. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. February 2009, Vol. 18, 1, pp. 140-181. http://www.exp-platform.com/Pages/hippo-long.aspx.
17. Crook, Thomas, et al. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. [ed.] Peter Flach and Mohammed Zaki. KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 2009, pp. 1105-1114. http://www.exp-platform.com/ Pages/ExPpitfalls.aspx.
18. Kohavi, Ron, et al. Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining. 2012, www.exp-platform.com/Pages/ PuzzingOutcomesExplained.aspx.
19. Wikipedia contributors. Fisher's method. Wikipedia. [Online] Jan 2014. http://en.wikipedia.org/wiki/Fisher%27s-method.
20. Eisenberg, Bryan. How to Increase Conversion Rate 1,000 Percent. ClickZ. [Online] Feb 28, 2003. http://www.clickz.com/showPage.html?page=1756031.
21. Spool, Jared. The $300 Million Button. USer Interface Engineering. [Online] 2009. http://www.uie.com/articles/three-hund-million-button/.
22. Goldstein, Noah J, Martin, Steve J and Cialdini, Robert B. Yes!: 50 Scientifically Proven Ways to Be Persuasive. s.l. : Free Press, 2008. 1416570969.
23. Collins , Jim and Porras , Jerry I. Built to Last: Successful Habits of Visionary Companies. s.l. : HarperBusiness, 2004. 978-0060566104.
24. Badam, Kiran. Looking Beyond Page Load Times - How a relentless focus on Task Completion Times can benefit your users. Velocity: Web Performance and Operations. 2013. http://velocityconf.com/velocityny2013/public/schedule/detail/ 32820.
25. Why Most Published Research Findings Are False. Ioannidis, John P. 8, 2005, PLoS Medicine, Vol. 2, p. e124. http://www.plosmedicine.org/article/info: doi/10.1371/journal.pmed.0020124.
26. Wacholder, Sholom, et al. Assessing the Probability That a Positive Report is False: An Approach for Molecular Epidemiology Studies. Journal of the National Cancer Institute. 2004, Vol. 96, 6. http://jnci.oxfordjournals.org/ content/96/6/434.long.
27. Ehrenberg, A. S. C. The Teaching of Statistics: Corrections and Comments. Journal of the Royal Statistical Society. Series A, 1974, Vol. 138, 4.
28. Ron Kohavi, David Messner, Seth Eliot, Juan Lavista Ferres, Randy Henne, Vignesh Kannappan, Justin Wang. Tracking Users' Clicks and Submits: Tradeoffs between User Experience and Data Loss. Redmond : s.n., 2010.
29. Patel , Neil . 11 Obvious A/B Tests You Should Try. QuickSprout. [Online] Jan 14, 2013. http://www.quicksprout.com/2013/01/14/11-obvious-ab-tests-you- should-try/.
30. Porter, Joshua. The Button Color A/B Test: Red Beats Green. Hutspot. [Online] Aug 2, 2011. http://blog.hubspot.com/blog/tabid/6307/bid/20566/The- Button-Color-A-B-Test-Red-Beats-Green.aspx.
31. Linden, Greg. Marissa Mayer at Web 2.0 . Geeking with Greg . [Online] Nov 9, 2006. http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html.
32. Performance Related Changes and their User Impact. Schurman, Eric and Brutlag, Jake. s.l. : Velocity 09: Velocity Web Performance and Operations Conference, 2009.
33. Souders, Steve. High Performance Web Sites: Essential Knowledge for Front-End Engineers. s.l. : O'Reilly Media, 2007. 978-0596529307.
34. Linden, Greg. Make Data Useful. [Online] Dec 2006. http://sites.google. com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt.
35. Wikipedia contributors. Above the fold. Wikipedia, The Free Encyclopedia. [Online] Jan 2014. http://en.wikipedia.org/wiki/Above-the-fold.
36. Souders, Steve. Moving beyond window.onload(). High Performance Web Sites Blog. [Online] May 13, 2013. http://www.stevesouders.com/blog/2013/05/13/ moving-beyond-window-onload/.
37. Brutlag, Jake, Abrams, Zoe and Meenan, Pat . Above the Fold Time: Measuring Web Page Performance Visually. Velocity: Web Performance and Operations Conference. 2011. http://en.oreilly.com/velocitymar2011/public/ schedule/detail/18692.
38. Meenan, Patrick. Speed Index. WebPagetest. [Online] April 2012. https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/ speed-index.
39. Meenan, Patrick, Feng, Chao (Ray) and Petrovich, Mike . Going Beyond onload - How Fast Does It Feel? Velocity: Web Performance and Operations. 2013. http://velocityconf.com/velocityny2013/public/schedule/detail/31344.
40. Fisher, Ronald A. Presidential Address. Sankhya: The Indian Journal of Statistics. 1938, Vol. 4, 1. http://www.jstor.org/stable/40383882.
41. Kohavi, Ron and Longbotham, Roger. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations. 2010, Vol. 12, 2. http://www.exp-platform.com/Documents/2010-12%20ExPUnexpectedSIGKDD.pdf.
42. Montgomery, Douglas C. Applied Statistics and Probability for Engineers. 5th. s.l. : John Wiley & Sons, Inc, 2010. 978-0470053041.
43. Boos, Dennis D and Hughes-Oliver, Jacqueline M. How Large Does n Have to be for Z and t Intervals? The American Statistician. 2000, Vol. 54, 2, pp. 121-128.
44. Efron, Bradley and Robert J. Tibshirani. An Introduction to the Bootstrap. New York : Chapman & Hall, 1993. 0-412-04231-2.