Characterizing experimentation in continuous deployment: A case study on bing

Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017

Volumn , Issue , 2017, Pages 123-132

  Kevic, Katja ^a   Murphy, Brendan ^b   Williams, Laurie ^c   Beckmann, Jennifer ^d  

^a UNIVERSITY OF ZURICH   (Switzerland)

^b MICROSOFT RESEARCH   (United States)

^c NORTH CAROLINA STATE UNIVERSITY   (United States)

^d MICROSOFT   (United States)

Author keywords

continuous deployment; empirical analysis; experimentation; full deployment cycle

Indexed keywords

BINS; CODES (SYMBOLS);

CONTINUOUS DEPLOYMENT; CONTROLLED EXPERIMENT; DEPLOYMENT PROCESS; EMPIRICAL ANALYSIS; EXPERIMENTATION; FEATURE CHANGES; FULL DEPLOYMENT CYCLE; MULTIPLE ITERATIONS;

SOFTWARE ENGINEERING;

EID: 85026847860     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICSE-SEIP.2017.19     Document Type: Conference Paper

References (29)

1. Experiments-articles, & solutions. https://developers.google.com/analytics/solutions/experiments. Accessed: 2016-10-21
2. Planout a framework for online field experiments. https://facebook.github.io/planout/. Accessed: 2016-10-21
3. B. Adams, and S. McIntosh. Modern release engineering in a nutshell-why researchers should care. In IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, pages 78-90, 2016
4. E. Bakshy, D. Eckles, and M. S. Bernstein. Designing, and deploying online field experiments. In 23rd International Conference on World Wide Web, pages 283-292, 2014
5. J. Bosch. Building products as innovation experiment systems. In 3rd International Conference on Software Business, Lecture Notes in Business Information Processing, pages 27-39, 2012
6. T. Crook, B. Frasca, R. Kohavi, and R. Longbotham. Seven pitfalls to avoid when running controlled experiments on the web. In 15th ACM International Conference on Knowledge Discovery, and Data Mining, pages 1105-1114, 2009
7. T. H. Davenport. How to design smart business experiments. Harvard Business Review, pages 69-76, 2009
8. A. Deng. Objective bayesian two sample hypothesis testing for online controlled experiments. In 24th International Conference on World Wide Web (Companion), pages 923-928, 2015
9. A. Deng, Y. Xu, R. Kohavi, and T. Walker. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In 6th ACM International Conference on Web Search, and Data Mining, pages 123-132, 2013
10. P. Duvall, S. Matyas, and A. Glover. Continuous Integration: Improving Software Quality, and Reducing Risk. A Martin Fowler signature book. Addison-Wesley, 2007
11. F. Fagerholm, A. S. Guinea, and H. M? The {RIGHT} model for continuous experimentation. Journal of Systems, and Software, pages-, 2016
12. F. Fagerholm, A. S. Guinea, H. Mäenpää, and J. Münch. Building blocks for continuous experimentation. In 1st International Workshop on Rapid Continuous Software Engineering, pages 26-35, 2014
13. J. Humble, and D. Farley. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley Signature Series (Fowler). Pearson Education, 2010
14. M. Kim, T. Zimmermann, R. DeLine, and A. Begel. The emerging role of data scientists on software development teams. In 38th International Conference on Software Engineering, pages 96-107, 2016
15. R. Kohavi, T. Crook, R. Longbotham, B. Frasca, R. Henne, J. L. Ferres, and T. Melamed. Online experimentation at microsoft. In Workshop on Data Mining Case Studies, and Practice Prize, 2009
16. R. Kohavi, A. Deng, B. Frasca, R. Longbotham, T. Walker, and Y. Xu. Trustworthy online controlled experiments: Five puzzling outcomes explained. In 18th ACM International Conference on Knowledge Discovery, and Data Mining, pages 786-794, 2012
17. R. Kohavi, A. Deng, B. Frasca, T. Walker, Y. Xu, and N. Pohlmann. Online controlled experiments at large scale. In 19th ACM International Conference on Knowledge Discovery, and Data Mining, pages 1168-1176, 2013
18. R. Kohavi, A. Deng, R. Longbotham, and Y. Xu. Seven rules of thumb for web site experimenters. In 20th ACM International Conference on Knowledge Discovery, and Data Mining, pages 1857-1866, 2014
19. R. Kohavi, and R. Longbotham. Online controlled experiments, and a/b tests (to appear). In Encyclopedia of Machine Learning, and Data Mining, 2016
20. R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: survey, and practical guide. Data Mining, and Knowledge Discovery, 18(1): 140-181, 2009
21. E. Lindgren, and J. Münch. Software Development as an Experiment System: A Qualitative Survey on the State of the Practice. 2015
22. R. Mason, R. Gunst, and J. Hess. Statistical design, and analysis of experiments: with applications to engineering, and science. Wiley series in probability, and mathematical statistics: Applied probability, and statistics. 1989
23. D. Mindell. Digital Apollo: Human, and Machine in Spaceflight. Inside Technology Series. 2008
24. H. H. Olsson, H. Alahyari, and J. Bosch. Climbing the "stairway to heaven"-a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In 38th Euromicro Conference on Software Engineering, and Advanced Applications, pages 392-399. IEEE Computer Society, 2012
25. C. Parnin, E. Helms, C. Atlee, H. Boughton, M. Ghattas, A. Glover, J. Holman, J. Micco, B. Murphy, T. Savor, M. Stumm, S. Whitaker, and L. Williams. Top 10 adages in continuous deployment. In IEEE Software, to appear. IEEE Computer Society, 2016
26. E. Ries. The Lean Startup: How Today?s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Publishing Group, 2011
27. O. Rissanen, and J. Münch. Continuous experimentation in the b2b domain: A case study. In 2nd Rapid Continuous Software Engineering, pages 12-18. IEEE/ACM, 2015
28. D. Tang, A. Agarwal, D. O?Brien, and M. Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In 16th Conference on Knowledge Discovery, and Data Mining, pages 17-26, 2010
29. Y. Xu, N. Chen, A. Fernandez, O. Sinno, and A. Bhasin. From infrastructure to culture: A/b testing challenges in large scale social networks. In 21th ACM International Conference on Knowledge Discovery, and Data Mining, pages 2227-2236. ACM, 2015