From infrastructure to culture: A/B testing challenges in large scale social networks

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn 2015-August, Issue , 2015, Pages 2227-2236

  Xu, Ya ^a   Chen, Nanyu ^a   Fernandez, Adrian ^a   Sinno, Omar ^a   Bhasin, Anmol ^a  

^a LINKEDIN CORPORATION   (United States)

Author keywords

A B testing; Controlled experiments; Measurement; Network A B testing; Online experiments; Social network

Indexed keywords

DATA MINING; MEASUREMENTS;

A/B TESTING; CONTROLLED EXPERIMENT; DATA DRIVEN DECISION; EXPERIMENTATION PLATFORMS; NETWORK EFFECTS; NETWORK SETTINGS; ON-LINE EXPERIMENTS; SOCIAL NETWORK SITES;

SOCIAL NETWORKING (ONLINE);

EID: 84954107438     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2783258.2788602     Document Type: Conference Paper

References (35)

1. Rubin, Donald B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
2. Ugander, Johan, Karrer, Brian, Backstrom, Lars and Kleinberg, Jon. Graph cluster randomization: network exposure to multiple universes. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329-337. ACM, 2013.
3. Katzir, Liran, Liberty, Edo and Somekh Oren. Framework and algorithms for network bucket testing. Proceedings of the 21st international conference on World Wide Web, pages 1029-1036. ACM, 2012.
4. Toulis, Panos and Kao, Edward. Estimation of causal peer influence effects. Proceedings of The 30th International Conference on Machine Learning, pages 1489-1497, 2013.
5. Eckles, Dean, Karrer, Brian and Ugander, Johan. Design and analysis of experiments in networks: Reducing bias from interference. arXiv preprint arXiv:1404.7530, 2014.
6. Aronow, Peter M, and Samii, Cyrus. Estimating average causal effects under general interference. arXiv preprint arXiv:1305.6156, 2013
7. Kohavi, Ron, et al. Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining. 2012, www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx.
8. th Conference on Knowledge Discovery and Data Mining. 2010.
9. Kohavi, Ron, et al. Online Controlled Experiments at Large Scale. KDD 2013: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 2013. http://bit.ly/ExPScale.
10. Kohavi, Ron, et al. Seven Rules of Thumb for Web Site Experimenters. KDD 2014: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 2014.
11. Yates, Frank, Sir Ronald Fisher and the Design of Experiments. Biometrics, 20(2):307-321, 1964.
12. Bakshy, Eytan, Echles, Dean and Bernstein, Michael S. Designing and Deploying Online Field Experiments. Proceedings of the 23rd international conference on World Wide Web, pages 283-292, ACM, 2014
13. Kohavi, Ron. et al. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. February 2009, Vol. 18, 1, pp. 140-181. http://www.exp-platform.com/Pages/hippo-long.aspx.
14. Crook, Thomas, et al. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. [ed.] Peter Flach and Mohammed Zaki. KDD'09: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining. 2009, pp. 1105-1114. http://www.exp-platform.com/Pages/ExPpitfalls.aspx.
15. Ioannidis, John PA. "Why most published research findings are false." PLoS medicine 2.8 (2005): e124.
16. Wacholder, Sholom, et al. "Assessing the probability that a positive report is false: an approach for molecular epidemiology studies." Journal of the National Cancer Institute 96.6 (2004): 434-442.
17. Benjamini, Yoav, and Yosef Hochberg. "Controlling the false discovery rate: a practical and powerful approach to multiple testing." Journal of the Royal Statistical Society. Series B (Methodological) (1995): 289-300.
18. Saaty, Thomas L. "How to make a decision: the analytic hierarchy process." European journal of operational research 48.1 (1990): 9-26.
19. Gui, Huan, Xu, Ya, Bhasin, Anmol, Han Jiawei. Network A/B Testing: From Sampling to Estimation. Proceedings of the 24rd international conference on World Wide Web, ACM, 2015
20. Box, George EP, J. Stuart Hunter, and William G. Hunter. "Statistics for experimenters: design, innovation, and discovery." AMC 10 (2005): 12.
21. Gerber, A. S., and Green, D. P. Field Experiments: Design, Analysis, and Interpretation. WW Norton, 2012
22. Sumbaly, Roshan, et al. "Serving large-scale batch computed data with project voldemort." Proceedings of the 10th USENIX conference on File and Storage Technologies. USENIX Association, 2012.
23. Tate, Ryan. The Software Revolution Behind LinkedIn's Gushing Profits. [Online] http://www.wired.com/2013/04/linkedin-software-revolution
24. Auradkar, Aditya, et al. "Data infrastructure at LinkedIn." Data Engineering (ICDE), 2012 IEEE 28th International Conference on. IEEE, 2012.
25. Kreps, Jay, Neha Narkhede, and Jun Rao. "Kafka: A distributed messaging system for log processing." Proceedings of 6th International Workshop on Networking Meets Databases (NetDB), Athens, Greece. 2011.
26. Naga, Praveen Neppalli, Real-time Analytics at Massive Scale with Pinot. [Online] September 29, 2014 http://engineering.linkedin.com/analytics/real-time-analytics-massive-scale-pinot
27. Fisher, Ronald A. Presidential Address. Sankhya: The Indian Journal of Statistics. 1938, Vol. 4, 1. http://www.jstor.org/stable/40383882.
28. Montgomery, Douglas C. Design and analysis of experiments. John Wiley & Sons, 2008.
29. Betz, Joe, Tagle, Moira. Rest.li:RESTful Service Architecture at Scale. [Online] February, 19, 2013 https://engineering.linkedin.com/architecture/restli-restful-service-architecture-scale
30. Romano, Joseph P. Azeem M. Shaikh and Michael Wolf. 2010b Multiple Testing. New Palgrave Dictionary of Economics. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.418.4975&rep=rep1&type=pdf
31. Wikipedia. Simpson's Paradox. [Online] http://en.wikipedia.org/wiki/Simpson%27s-paradox
32. McFarland, Colin. Experiment!: Website conversion rate optimization with A/B and multivariate testing. s.1.: New Riders, 2012. 978-0321834607
33. Eisenberg, Bryan. How to Improve A/B Testing. ClickZ Network. [Online] April 29, 2005. www.clickz.com/clickz/column/1717234/how-improvem-a-b-testing.
34. Vemuri, Srinivas, Varshney, Maneesh, Puttaswamy, Krishna and Liu, Rui. Execution Primitives for Scalable Joins and Aggregations in Map Reduce. Proceedings of the VLDB Endowment, Vol. 7, No. 13
35. Varshney, Maneesh, Vemuri, Srinivas. Open Sourcing Cubert: A High Performance Computation Engine for Complex Big Data Analytics [Online] November 11, 2014 https://engineering.linkedin.com/big-data/open-sourcing-cubert-high-performance-computation-engine-complex-big-data-analytics