Seven pitfalls to avoid when running controlled experiments on the web

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn , Issue , 2009, Pages 1105-1113

  Crook, Thomas ^a   Frasca, Brian ^a   Kohavi, Ron ^a   Longbotham, Roger ^a  

^a MICROSOFT   (United States)

Author keywords

A B testing; Controlled experiments; E commerce; Robot detection; Simpson's paradox

Indexed keywords

A/B TESTING; CONTROLLED EXPERIMENTS; E-COMMERCE; ROBOT DETECTION; SIMPSON'S PARADOX;

ELECTRONIC COMMERCE; ROBOTS; SURVEYS;

EXPERIMENTS;

EID: 70350625361     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1557019.1557139     Document Type: Conference Paper

References (25)

1. Bacher, Paul, et al. 2005. Know your Enemy: Tracking Botnets. The Honeynet Project. [Online] March 13, 2005. http://www.honeynet.org/ papers/bots/.
2. Bomhardt, Christian, Gaul, Wolfgang and Schmidt-Thieme, Lars. 2005. Web Robot Detection - Preprocessing Web Logfiles for Robot Detection. [book auth.] Maurizio Vichi, et al. New Developments in Classification and Data Analysis. s.l. : Springer, 2005.
3. Box, George E.P., Hunter, J Stuart and Hunter, William G. 2005. Statistics for Experimenters: Design, Innovation, and Discovery. 2nd. s.l. : John Wiley & Sons, Inc, 2005. 0471718130.
4. Claypool, Mark, et al. 2001. Inferring user interest. IEEE Internet Computing. 2001, Vol. 5, pp. 32-39. http://citeseerx.ist.psu.edu/ viewdoc/summary?doi=10.1.1.28.5967.
5. Efron, Bradley and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. New York : Chapman & Hall, 1993. 0-412-04231-2.
6. Fieller, E C. 1940. The Biological Standardization of Insulin. Supplement to the Journal of the Royal Statistical Society. 1940, Vol. 7, 1, pp. 1-64.
7. Fox, Steve, et al. 2005. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems (TOIS). 2005, Vol. 23, 2, pp. 147-168. http://portal.acm.org/citation.cfm?id=1059981.1059982.
8. Hill, Nigel, Roche, Greg and Allen, Rachel. 2007. Customer Satisfaction: The Customer Experience Through the Customer's Eyes. s.l. : Cogent Publishing, 2007.
9. Hopkins, Claude. 1923. Scientific Advertising. New York City : Crown Publishers Inc., 1923.
10. Keppel, Geoffrey, Saufley, William H and Tokunaga, Howard. 1992. Introduction to Design and Analysis. 2nd. s.l. : W.H. Freeman and Company, 1992.
11. Kohavi, Ron, et al. 2009. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. February 2009, Vol. 18, 1, pp. 140-181. http://expplatform.com/hippo-long.aspx.
12. Kohavi, Ron, et al. 2004. Lessons and Challenges from Mining Retail E-Commerce Data. 2004, Vol. 57, 1-2, pp. 83-113. http://ai.stanford.edu/ ~ronnyk/lessonsInDM.pdf.
13. Kohavi, Ron, Henne, Randal M and Sommerfield, Dan. 2007. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. The Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2007). August 2007, pp. 959-967. http://expplatform.com/hippo.aspx.
14. Koselka, Rita. 1996. The New Mantra: MVT. Forbes. March 11, 1996, pp. 114-118.
15. Malinas, Gary and Bigelow, John. 2004. Simpson's Paradox. Stanford Encyclopedia of Philosophy. [Online] 2004. [Cited: February 28, 2008.] http://plato.stanford.edu/entries/paradoxsimpson/.
16. Mason, Robert L, Gunst, Richard F and Hess, James L. 1989. Statistical Design and Analysis of Experiments With Applications to Engineering and Science. s.l. : John Wiley & Sons, 1989. 047185364X.
17. Montgomery, Douglas C. 2005. Design and Analysis of Experiments. 6th edition. s.l. : John Wiley & Sons, Inc, 2005. 0-471-66159-7.
18. Rao, C. Radhakrishna. 1973. Linear Statistical Inference and Its Applications. 2nd. s.l. : John Wiley & Sons, Inc., 1973.
19. Roy, Ranjit K. 2001. Design of Experiments using the Taguchi Approach : 16 Steps to Product and Process Improvement. s.l. : John Wiley & Sons, Inc, 2001. 0-471-36101-1.
20. Simpson, Edward H. 1951. The Interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society, Ser. B. 1951, Vol. 13, pp. 238-241.
21. Spears, Steven J. 2004. Learning to Lead at Toyota. Harvard Business Review. May 2004, pp. 78-86.
22. Tan, Pang-Ning and Kumar, Vipin. 2002. Discovery of Web Robot Sessions based on their Navigational Patterns. Data Mining and Knowledge. 2002, Vol. 6, 1, pp. 9-35. http://citeseer.ist.psu.edu/article/tan02discovery. html.
23. Wikipedia: Botnet. 2008. Botnet. Wikipedia. [Online] 2008. [Cited: February 28, 2008.] http://en.wikipedia.org/wiki/Botnet.
24. Wikipedia: Internet bot. 2008. Internet Bot. Wikipedia. [Online] 2008. [Cited: February 28, 2008.] http://en.wikipedia.org/wiki/ Internet-bot.
25. Wikipedia: Simpson's Paradox. 2008. Simpson's paradox. Wikipedia. [Online] 2008. [Cited: February 28, 2008.] http://en.wikipedia.org/wiki/Simpson%27s-paradox.