On rapid releases and software testing: a case study and a semi-systematic literature review

Empirical Software Engineering

Volumn 20, Issue 5, 2015, Pages 1384-1425

  Mäntylä, Mika V ^a   Adams, Bram ^b   Khomh, Foutse ^b   Engström, Emelie ^c   Petersen, Kai ^d  

^a AALTO UNIVERSITY   (Finland)

^b ÉCOLE POLYTECHNIQUE DE MONTRÉAL   (Canada)

^c LUND UNIVERSITY   (Sweden)

^d BLEKINGE INSTITUTE OF TECHNOLOGY   (Sweden)

Author keywords

Agile releases; Bugs; Builds; Mozilla; Open source; Release model; Software testing

Indexed keywords

OPEN SOURCE SOFTWARE; OPEN SYSTEMS; PROGRAM DEBUGGING; SOFTWARE DESIGN; WEB BROWSERS;

AGILE RELEASES; BUGS; BUILDS; MOZILLA; OPEN-SOURCE;

SOFTWARE TESTING;

EID: 84940719377     PISSN: 13823256     EISSN: 15737616     Source Type: Journal    
DOI: 10.1007/s10664-014-9338-4     Document Type: Article

References (86)

1. Agarwal P (2011) Continuous scrum: Agile management of saas products, pp 51–60
2. Azham Z, Ghani I, Ithnin N (2011) Security backlog in scrum security practices, pp 414–417
3. Baskerville R, Ramesh B, Levine L, Pries-Heje J (2006) High-speed software development practices: What works, what doesn’t. IT Professional 8:29–36
4. Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th Working Conf. on Mining Software Repositories (MSR), pp 238–241
5. Baysal O, Kononenko O, Holmes R, Godfrey M (2012) The secret life of patches: a firefox case study, pp 447–455
6. Beck K, Andres C (2004) Extreme Programming explained: embrace change, 2nd edn. Addison-Wesley
7. Begel A, Nagappan N (2007) Usage and perceptions of agile software development in an industrial context: an exploratory study, pp 255–264
8. Bell R, Ostrand T, Weyuker E (2006) Looking for bugs in all the right places, vol. 2006, pp 61–71
9. Berner S, Weber R, Keller RK (2005) Observations and lessons learned from automated testing. In: Proceedings of the 27th international conference on Software engineering, ACM, pp 571–579
10. Boshuizen C, Marshall W, Bridges C, Kenyon S, Klupar P (2011) Learning to follow: embracing commercial technologies and open source for space missions 10:8097–8101
11. Brown AW (2011) A case study in agile-at-scale delivery. In: Proceeding of the 12th Intl. Conf. on Agile Processes in Software Engineering and Extreme Programming (XP), vol 77, pp 266–281
12. Byrt T, Bishop J, Carlin JB (1993) Bias, prevalence and kappa. J Clin Epidemiol 46:423–429
13. Carney PA, Sickles EA, Monsees BS, Bassett LW, Brenner RJ, Feig SA, Smith RA, Rosenberg RD, Bogart TA, Browning S et al (2010) Identifying minimally acceptable interpretive performance criteria for screening mammography 1. Radiology 255:354–361
14. Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: Ii. resolving the paradoxes. J Clin Epidemiol 43:551–558
15. Codabux Z, Williams B (2013) Managing technical debt: an industrial case study, pp 8–15
16. Cohan S (2007) Successful integration of agile development techniques within disa, pp 255–260
17. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Academic Press
18. Cusumano MA, Yoffie DB (1999) Competing on internet time: Lessons from netscape and its battle with microsoft. Simon and Schuster
19. Gamma E (2005) Agile, open source, distributed, and on-time – inside the eclipse development process. Keynote at the 27th Intl. Conf. on Software Engineering (ICSE)
20. Greiler M, van Deursen A, Zaidman A (2012) Measuring test case similarity to support test suite understanding. In: Objects, models components patterns. Springer, pp 91–107
21. Gundebahar M, Kus Khalilov M (2013) A hybrid deployment model for financial systems with service oriented architecture (soa): Running from client via branch server, pp 365–370
22. Heikkilä V, Rautiainen K, Jansen S (2010) A revelatory case study on scaling agile release planning, pp 289–296
23. Hemmati H, Arcuri A, Briand L (2013) Achieving scalable model-based testing through test case diversity. ACM Trans Softw Eng Methodol (TOSEM) 22:6
24. Hodgetts P, Phillips D (2002) 30. In: eXtreme Adoption eXperiences of a B2B Start Up. Addison-Wesley Longman Publishing Co., Inc. Extreme Programming Perspectives
25. Hollander M, Wolfe DA (1999) Nonparametric statistical methods, 2nd edn. John Wiley and Sons inc
26. HP (2012) Shorten release cycles aby bringing developers to application lifecycle management. HP applications handbook, Retrieved on February 08, 2012
27. Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional
28. InvestmentWatch (2011) Mozilla puts out firefox 5.0 web browser which carries over 1,000 improvements in just about 3 months of development. http://bit.ly/aecRrL
29. Iwai A, Aoyama M (2011) Automotive cloud service systems based on service-oriented architecture and its evaluation, pp 638–645
30. Jansen S, Brinkkemper S (2006) Ten misconceptions about product software release management explained using update cost/value functions. In: Proceedings of the International Workshop on Software Product Management, pp 44–50
31. Jenkins J (2011) Velocity culture (the unmet challenge in ops). Presentation at O’Reilly Velocity Conference
32. Jones G, Leung V (2005) Visual surveillance: a systems engineering approach for rapid development. Number 2005-11033, pp 161–166
33. Kaply M (2012) Why do companies stay on old technology? Retrieved on January 12, 2012
34. Kettunen V, Kasurinen J, Taipale O, Smolander K (2010) A study on agility and testing processes in software organizations. In: Proc. of the 19th Intl. Symp. on Software Testing and Analysis (ISSTA), pp 231–240
35. Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? An empirical case study of mozilla firefox. In: MSR, pp 179–188
36. Kim S (2012) ppcor: Partial and semi-partial (part) correlation. http://bit.ly/XkWuyn
37. Kitchenham B (2004) Procedures for performing systematic reviews, vol. 33:2004
38. Kong S, Kendall JE, Kendall KE (2009) The challenge of improving software quality: Developers’ beliefs about the contribution of agile practices. In: Proc. of the Americas Conf. on Information Systems (AMCIS), p 12
39. Kühner G, Bluhm T, Heimann P, Hennig C, Kroiss H, Krom J, Laqua H, Lewerentz M, Maier J, Schacht J, Spring A, Werner A, Zilker M (2012) Progress on standardization and automation in software development on w7x. Fusion Eng Des 87:2232–2237
40. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics159–174
41. Lavoie T, Merlo E (2013) How much really changes? a case study of firefox version evolution using a clone detector, pp 83–89
42. Li J, Moe NB, Dybå T (2010) Transition from a plan-driven process to scrum: a longitudinal case study on software quality. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 13:1–13:10
43. Li X, Li Y, Xie M, Ng S (2011) Reliability analysis and optimal version-updating for open source software. Inf Softw Technol 53:929–936
44. Ltd. RS (2013) Web browsers (global marketshare). http://bit.ly/81klgi
45. Maalej W (2009) Task-first or context-first? tool integration revisited, pp 344–355
46. Mäntylä MV, Itkonen J (2013) More testers âĂŞ the effect of crowd size and time restriction in software testing. Inf Softw Technol 55:986–1003
47. Mantyla M, Khomh F, Adams B, Engstrom E, Petersen K (2013) On rapid releases and software testing. In: Proceedings of the 29th IEEE International Conference on Software Maintenance (ICSM), pp 20–29
48. Marschall M (2007) Transforming a six month release cycle to continuous flow. In: Proc. of the conf. on AGILE, pp 395–400
49. Martin D, Rooksby J, Rouncefield M, Sommerville I (2007) ’good’organisational reasons for’bad’software testing: An ethnographic study of testing in a small software company. In: Software Engineering, 2007. ICSE 2007. 29th International Conference on, IEEE, pp 602–611
50. McDaniel LS (1990) The effects of time pressure and audit program structure on audit performance. J Account Res 28:267–285
51. Middleton P (2001) Lean software development two case studies. Software Qual J 9:241–252
52. Middleton P, Joyce D (2012) Lean software management: Bbc worldwide case study. IEEE Trans Eng Manag 59:20–32
53. Mozilla (2013) Litmus wiki. http://mzl.la/evJmTW
54. Mozilla (2013) Moztrap wiki. http://bit.ly/XBGMfu
55. Mozilla (2013) Mozilla source code mercurial repositories
56. Olsson H, Alahyari H, Bosch J (2012) Climbing the “stairway to heaven” - a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software, pp 392–399
57. Olsson H, Bosch J, Alahyari H (2013) Towards r&d as innovation experiment systems: a framework for moving beyond agile software development, pp 798–805
58. Otte T, Moreton R, Knoell HD (2008) Applied quality assurance methods under the open source development model. In: Proceeding of the 32nd Annual IEEE Intl. Computer Software and Applications Conf. (COMPSAC), pp 1247–1252
59. Oza N, Ebert C, Abrahamsson P (2012) Lean software development. IEEE Softw 29:0022–25
60. Paasivaara M, Lassenius C (2003) Collaboration practices in global inter-organizational software development projects. Softw Process Improv Pract 8:183–199
61. Patel C, Lycett M, Macredie R, De Cesare S (2006) Perceptions of agility and collaboration in software development practice, vol. 1, p 10c
62. Paul R (2011) Mozilla outlines 16-week firefox development cycle. http://bit.ly/fLHEfo
63. Paul R (2012) Firefox extended support will mitigate rapid release challenges. http://ars.to/M2TbFQ
64. Pérez-Castillo R, Sánchez-González L, Piattini M, García F, Garcia-Rodriguez de Guzman I (2011) Obtaining thresholds for the effectiveness of business process mining. In: Empirical software engineering and measurement (ESEM), 2011 International Symposium on IEEE, pp 453–462
65. Petersen K, Wohlin C (2009) A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. J Syst Softw 82:1479–1490
66. Petersen K, Wohlin C (2010) The effect of moving from a plan-driven to an incremental software development approach with agile practices. Empir Softw Eng 15:654–693
67. Poppendieck M (2007) Lean software development. In: Companion to the proceedings of the 29th International Conference on Software Engineering, IEEE Computer Society, pp 165–166
68. Porter A, Yilmaz C, Memon AM, Krishna AS, Schmidt DC, Gokhale A (2006) Techniques and processes for improving the quality and performance of open-source software. Softw Process Improv Pract 11:163–176
69. Rafi DM, Moses KRK, Petersen K, Mäntylä MV (2012) Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In: Automation of Software Test (AST), 2012 7th International Workshop on, IEEE, pp 36–42
70. Raymond ES (1999) The Cathedral and the Bazaar, 1st edn. O’Reilly & Associates, Inc., Sebastopol
71. Reinertsen DG (2009) The principles of product development flow: second generation lean product development. Celeritas Redondo Beach, Canada
72. Rogmann JJ (2013) Orddom: Ordinal dominance statistics. http://bit.ly/Y0K0eo
73. Ruhe G, Saliu MO (2005) The art and science of software release planning. IEEE Softw 22:47–53
74. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14:131–164
75. Schwaber K (1995) Scrum development process. In: Proceedings of the 10th Annual ACM Conference on Object Oriented Programming Systems, Languages, and Applications (OOPSLA), pp 117–134
76. Shankland S (2010) Google ethos speeds up chrome release cycle. http://cnet.co/wlS24U
77. Shankland S (2011) Rapid-release firefox meets corporate backlash. http://cnet.co/ktBsUU
78. Sicore D (2011) New channels for firefox rapid releases. http://bit.ly/hc1zmY
79. Sonnentag S, Brodbeck FC, Heinbokel T, Stolte W (1994) Stressor-burnout relationship in software development teams. J Occup Organ Psychol 67:327–341
80. Sundmark D, Petersen K, Larsson S (2011) An exploratory case study of testing in an automotive electrical system release process, pp 166–175
81. Szőke Á (2011) Conceptual scheduling model and optimized release scheduling for agile environments. Inf Softw Technol 53:574–591
82. Torkar R, Minoves P, Garrigós J (2011) Adopting free/libre/open source software practices, techniques and methods for industrial use. J Assoc Inf Syst 12:88–122
83. Wikipedia (2013) Firefox release history. http://bit.ly/Ngvfln
84. Wikipedia (2013) Extended support release. http://bit.ly/ZlgqoM#Extended_Support_Release
85. Wikipedia (2013) Locale. http://bit.ly/2iJLwB
86. VersionOne (2012) 7th annual state of agile survey. http://www.versionone.com/pdf/7th-Annual-State-of-Agile-Development-Survey.pdf