Online controlled experiments at large scale

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn Part F128815, Issue , 2013, Pages 1168-1176

  Kohavi, Ron ^a   Deng, Alex ^a   Frasca, Brian ^a   Walker, Toby ^a   Xu, Ya ^a   Pohlmann, Nils ^a  

^a MICROSOFT   (United States)

Author keywords

A b testing; Controlled experiments; Online experiments; Search

Indexed keywords

DATA MINING;

A/B TESTING; CLASSICAL-TESTING; CONTROLLED EXPERIMENT; EXPERIMENTATION SYSTEMS; ON-LINE EXPERIMENTS; ONLINE CONTROLLED EXPERIMENTS; SEARCH; STATISTICAL INTERACTION;

SOCIAL NETWORKING (ONLINE);

EID: 84985031279     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2487575.2488217     Document Type: Conference Paper

References (45)

1. Kohavi, Ron and Round, Matt. Front Line Internet Analytics at Amazon.com. [ed.] Jim Sterne. Santa Barbara, CA : s.n., 2004. http://ai.stanford.edu/ronnyk/emetricsAmazon.pdf.
2. McKinley, Dan. Design for Continuous Experimentation: Talk and Slides. [Online] Dec 22, 2012. http://mcfunley.com/design-forcontinuous-experimentation.
3. Tang, Diane, et al. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. Proceedings 16th Conference on Knowledge Discovery and Data Mining. 2010.
4. Moran, Mike. Multivariate Testing in Action: Quicken Loan's Regis Hadiaris on multivariate testing. Biznology Blog by Mike Moran. [Online] December 2008. www.biznology.com/2008/12/multivariate-Testing-in-Action/.
5. Kohavi, Ron, Crook, Thomas and Longbotham, Roger. Online Experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize. 2009. http://expplatform.com/expMicrosoft.aspx.
6. Amatriain, Xavier and Basilico, Justin. Netflix Recommendations: Beyond the 5 stars. [Online] April 2012. http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html.
7. McFarland, Colin. Experiment: Website conversion rate optimization with A/B and multivariate testing. s.l. : New Riders, 2012. 978- 0321834607.
8. Kolar, Sumanth. Recommendations and Discovery at StumbleUpon. [Online] Sept 2012. www.slideshare.net/sumanthkolar/recsys-2012-sumanth-14260370.
9. Smietana, Brandon. Zynga: What is Zynga's core competency? Quora. [Online] Sept 2010. http://www.quora.com/Zynga/What-is-Zyngascore-competency/answer/Brandon-Smietana.
10. Blank, Steven Gary. The Four Steps to the Epiphany: Successful Strategies for Products that Win . s.l. : Cafepress.com, 2005. 978- 0976470700.
11. Ries, Eric. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses . s.l. : Crown Business, 2011. 978-0307887894.
12. Wikipedia. Lean Startup. [Online] 2013. http://en.wikipedia.org/wiki/Lean-Startup.
13. Sinofsky, Steven. One Strategy: Organization, Planning, and Decision Making . s.l. : Wiley, 2009. 978-0470560457 .
14. comScore. comScore Releases January 2013 U.S. Search Engine Rankings. [Online] Feb 13, 2013. http://www.comscore.com/Insights/Press-Releases/2013/2/comScore-Releases-January-2013-U.S.-Search-Engine-Rankings.
15. SCOPE: Parallel Databases Meet MapReduce. Zhou, Jingren, et al. s.l. : VLDB Journal, 2012. http://research.microsoft.com/enus/um/people/jrzhou/pub/Scope-VLDBJ.pdf.
16. Klein, Peter and Suh, Chris. Microsoft Second Quarter 2013 Earnings Calls Transcript. Microsoft Investor Relations. [Online] Jan 24, 2013. http://www.microsoft.com/global/Investor/RenderingAssets/Downloads/FY13/Q2/Microsoft-Q2-2013-PreparedRemarks.docx.
17. Manzi, Jim. Uncontrolled: The Surprising Payoff of Trial-And-Error for Business, Politics, and Society. s.l. : Basic Books, 2012. 978-0- 465-02931-0.
18. Kohavi, Ron, et al. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. February 2009, Vol. 18, 1, pp. 140-181. http://www.expplatform.com/Pages/hippo-long.aspx.
19. Eisenberg, Bryan. How to Improve A/B Testing. ClickZ Network. [Online] April 29, 2005. www.clickz.com/clickz/column/1717234/how-improve-A-b-Testing.
20. Box, George E.P., Hunter, J Stuart and Hunter, William G. Statistics for Experimenters: Design, Innovation, and Discovery. 2nd. s.l. : John Wiley & Sons, Inc, 2005. 0471718130.
21. Kohavi, Ron. Online Controlled Experiments: Introduction, Learnings, and Humbling Statistics. The ACM Conference on Recommender Systems. 2012. Industry Keynote. http://www.expplatform.com/Pages/2012RecSys.aspx.
22. Crook, Thomas, et al. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. [ed.] Peter Flach and Mohammed Zaki. KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 2009, pp. 1105-1114. http://www.expplatform.com/Pages/ExPpitfalls.aspx.
23. Kohavi, Ron and Longbotham, Roger. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations. 2010, Vol. 12, 2. http://www.exp-platform.com/Documents/2010-12%20ExPUnexpectedSIGKDD.pdf.
24. Segall, Ken. Insanely Simple: The Obsession That Drives Apple's Success. s.l. : Portfolio Hardcover, 2012. 978-1591844839.
25. Kohavi, Ron, et al. Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining. 2012, www.expplatform.com/Pages/PuzzingOutcomesExplained.aspx.
26. Kohavi, Ron, Longbotham, Roger and Walker, Toby. Online Experiments: Practical Lessons. [ed.] Simon S.Y. Shim. IEEE Computer. September 2010, Vol. 43, 9, pp. 82-85. http://www.expplatform.com/Documents/IEEE2010ExP.pdf.
27. Kaushik, Avinash. Experimentation and Testing: A Primer. Occam's Razor. [Online] May 22, 2006. http://www.kaushik.net/avinash/2006/05/experimentation-And-Testinga-primer.html.
28. Moran, Mike. Do It Wrong Quickly: How the Web Changes the Old Marketing Rules . s.l. : IBM Press, 2007. 0132255960.
29. McKinley, Dan. Testing to Cull the Living Flower. [Online] Jan 2013. http://mcfunley.com/testing-To-cull-The-living-flower.
30. Deming, data and observational studies: A process out of control and needing fixing. Young, S Stanley and Karr, Allan. 3, 2011, Significance, Vol. 8. http://www.niss.org/sites/default/files/Young%20Karr%20Obs%20Study%20Problem.pdf.
31. Why Most Published Research Findings Are False. Ioannidis, John P. 8, 2005, PLoS Medicine, Vol. 2, p. e124. http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124.
32. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Ioannidis, John P. 2, s.l. : The Journal of the American Medical Association, 2005, Vol. 294. http://jama.jamanetwork.com/article.aspx?articleid=201218.
33. Weiss, Rick. Study Debunks Theory On Teen Sex, Delinquency. Washington Post. 2007, Nov 11. http://www.washingtonpost.com/wpdyn/content/story/2007/11/11/ST2007111100542.html.
34. Myopia, How Science Works: The Story of Night-Light. Myopia: Prevention and Control. [Online] http://www.myopiaprevention.org/references-night-lights.html.
35. Kohavi, Ron. Online Controlled Experiments: Listening to the Customers, not to the HiPPO. Keynote at EC10: The 11th ACM Conference on Electronic Commerce. 2010. http://www.expplatform.com/Documents/2010-06%20EC10.pptx.
36. Kuhn, Thomas. The Structure of Scientific Revolutions. 3rd. 1996. 978-0226458083 .
37. Wikipedia. Semmelweis reflex. http://en.wikipedia.org/wiki/Semmelweis-reflex.
38. Hubbard, Douglas W. How to Measure Anything: Finding the Value of Intangibles in Business. 2nd. s.l. : Wiley, 2010.
39. Wooton, David. Bad Medicine: Doctors Doing Harm Since Hippocrates. s.l. : Oxford University Press, 2007.
40. Linden, Greg. Make Data Useful. [Online] Dec 2006. www.home.blarg.net/glinden/StanfordDataMining.2006-11-29.ppt.
41. Performance Related Changes and their User Impact. Schurman, Eric and Brutlag, Jake. s.l. : Velocity 09: Velocity Web Performance and Operations Conference, 2009.
42. Douglas Bowman. Goodbye, Google. StopDesign. [Online] http://stopdesign.com/archive/2009/03/20/goodbye-google.html.
43. A Multiple Testing Procedure for Clinical Trials. O'Brien, Peter C. and Fleming, Thomas R. 3, September 1979, Biometrics, Vol. 35, pp. 549-556.
44. A unified approach to false discovery rate estimation. Strimmer, Korbinian. 1, s.l. : Bmc Bioinformatics, 2008, Vol. 9.
45. Deng, Alex, et al. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. WSDM 2013: Sixth ACM International Conference on Web Search and Data Mining. 2013. www.exp-platform.com/Pages/CUPED.aspx.