Considering rigor and relevance when evaluating test driven development: A systematic review

Information and Software Technology

Volumn 56, Issue 4, 2014, Pages 375-394

  Munir, Hussan ^a   Moayyed, Misagh ^b   Petersen, Kai ^c  

^a LUND UNIVERSITY   (Sweden)

^b Unicon Inc   (United States)

^c BLEKINGE INSTITUTE OF TECHNOLOGY   (Sweden)

Author keywords

External code quality; Internal code quality; Productivity; Test driven development (TDD); Test last development (TLD)

Indexed keywords

CODES (SYMBOLS); PRODUCTIVITY; TESTING;

CODE QUALITY; EXTERNAL CODE; LITERATURE REVIEWS; LONGITUDINAL CASE STUDY; RIGOR AND RELEVANCES; SYSTEMATIC LITERATURE REVIEW; TEST DRIVEN DEVELOPMENT; TRADITIONAL APPROACHES;

COMPUTER PROGRAMMING;

EID: 84894046936     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2014.01.002     Document Type: Review

References (89)

1. K. Beck, Extreme Programming Explained: Embrace Change, second ed., 2004
2. M. Fowler Refactoring: Improving the Design of Existing Code 1999 Addison-Wesley Professional
3. M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar Towards empirical evaluation of test-driven development in a university environment Proceedings of IEEE Region 8 EUROCON 2003, Computer as a Tool, 22-24 September 2003 vol. 2 2003 IEEE Piscataway, NJ, USA 83 86
4. H. Cibulski, and A. Yehudai Regression test selection techniques for test-driven development 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) 2011 IEEE 115 124
5. L. Huang, and M. Holcombe Empirical investigation towards the effectiveness of test first programming Inform. Softw. Technol. 51 1 2009 182 194
6. H. Erdogmus, M. Morisio, and M. Torchiano On the effectiveness of the test-first approach to programming IEEE Trans. Softw. Eng. 31 3 2005 226 237 (Pubitemid 40890004)
7. S. Kollanus Test-driven development - still a promising approach? 7th International Conference on the Quality of Information and Communications Technology, QUATIC 2010, September 29, 2010-October 2, 2010 2010 IEEE Computer Society 403 408
8. A. Causevic, D. Sundmark, S. Punnekkat, Factors limiting industrial adoption of test driven development: a systematic review, in: Proceedings 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation (ICST 2011), Los Alamitos, CA, USA, 2011, pp. 337-46.
9. F. Shull, G. Melnik, B. Turhan, L. Layman, M. Diep, and H. Erdogmus What do we know about test-driven development? IEEE Softw. 27 6 2010 16 19
10. P. Sfetsos, and I. Stamelos Empirical studies on quality in agile practices: a systematic literature review 7th International Conference on the Quality of Information and Communications Technology, QUATIC 2010, September 29, 2010-October 2, 2010 2010 IEEE Computer Society Porto, Portugal 44 53
11. S. Kollanus Critical issues on test-driven development 12th International Conference on Product-Focused Software Process Improvement, PROFES 2011, June 20, 2011-June 22, 2011 LNCS vol. 6759 2011 Springer Verlag Torre Canne, Italy 322 336
12. M. Ivarsson, and T. Gorschek A method for evaluating rigor and industrial relevance of technology evaluations Empir. Softw. Eng. 16 3 2011 365 395
13. S. Kollanus, and V. Isomottonen Test-driven development in education: experiences with critical viewpoints Proceedings of the Conference on Integrating Technology into Computer Science Education, ITiCSE, June 30, 2008-July 2, 2008 2008 Association for Computing Machinery Madrid, Spain 124 127
14. Y. Rafique, and V. Misic The effects of test-driven development on external quality and productivity: a meta-analysis IEEE Trans. Softw. Eng. 39 6 2013 835 856
15. R. Jeffries, and G. Melnik Guest editors' introduction: TDD-the art of fearless programming IEEE Softw. 24 3 2007 24 30 (Pubitemid 46739296)
16. K.S. Khan, G. Ter Riet, J. Glanville, A.J. Sowden, and J. Kleijnen et al. second ed. Undertaking Systematic Reviews of Research on Effectiveness: CRD's Guidance for Carrying Out or Commissioning Reviews vol. 4 2001 NHS Centre for Reviews and Dissemination
17. T. Greenhalgh How to Read a Paper: The Basics of Evidence-based Medicine third ed. 2006 Blackwell Malden, Mass.
18. B. Turhan, L. Layman, M. Diep, H. Erdogmus, F. Shull, How effective is test-driven development, Making Software: What Really Works, and Why We Believe It, 2010, pp. 207-217.
19. T. Greenhalgh, and R. Peacock Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources BMJ 331 7524 2005 1064 1065 (Pubitemid 41597859)
20. S. Jalali, C. Wohlin, Systematic literature studies: database searches vs. backward snowballing, in: 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2012), 2012, pp. 29-38.
21. B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman Systematic literature reviews in software engineering - a systematic literature review Inform. Softw. Technol. 51 1 2009 7 15
22. K. Petersen, N.B. Ali, Identifying strategies for study selection in systematic reviews and maps, in: Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement, ESEM 2011, Banff, AB, Canada, September 22-23, 2011, pp. 351-354.
23. B. Kitchenham, P. Brereton, D. Budgen, Mapping study completeness and reliability - a case study, in: Proceedings of the 16th International Conference on Evaluation & Assessment in Software Engineering (EASE 2012), 2012, pp. 126-135.
24. C. Wohlin, and R. Prikladnicki Systematic literature reviews in software engineering Inform. Softw. Technol. 55 6 2013 919 920
25. J.E. Hannay, T. Dybå, E. Arisholm, and D.I.K. Sjøberg The effectiveness of pair programming: a meta-analysis Inform. Softw. Technol. 51 7 2009 1110 1122
26. T. Dybå, T. Dingsøyr, G.K. Hanssen, Applying systematic reviews to diverse study types: an experience report, in: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, Madrid, Spain, September 20-21, 2007, pp. 225-234.
27. H. Kundel, and M. Polansky Measurement of observer agreement Radiology 228 2 2003 303
28. C. Wohlin, P. Runeson, P.A. da Mota Silveira Neto, E. Engström, I. do Carmo Machado, and E.S. de Almeida On the reliability of mapping studies in software engineering J. Syst. Softw. 86 10 2013 2594 2610
29. C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell, and A. Wesslén Experimentation in Software Engineering 2012 Springer Berlin
30. P. Runeson, and M. Höst Guidelines for conducting and reporting case study research in software engineering Empir. Softw. Eng. 14 2 2009 131 164
31. K. Petersen, C. Wohlin, Context in industrial software engineering research, in: Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, Lake Buena Vista, Florida, USA, October 15-16, 2009, pp. 401-404.
32. D.I.K. Sjøberg, T. Dybå, M. Jørgensen, The future of empirical methods in software engineering research, in: Workshop on the Future of Software Engineering (FOSE 2007), 2007, pp. 358-378.
33. M.V. Zelkowitz, D.R. Wallace, D. Binkley, Culture conflicts in software engineering technology transfer, in: NASA Goddard Software Engineering Workshop, 1998, p. 52.
34. D.I.K. Sjøberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.-K. Liborg, and A.C. Rekdal A survey of controlled experiments in software engineering IEEE Trans. Softw. Eng. 31 9 2005 733 753 (Pubitemid 41555415)
35. S. Barney, M. Khurum, K. Petersen, M. Unterkalmsteiner, and R. Jabangwe Improving students with rubric-based self-assessment and oral feedback IEEE Trans. Educat. 55 3 2012 319 325
36. A. Jonsson, and G. Svingby The use of scoring rubrics: reliability, validity and educational consequences Educat. Res. Rev. 2 2 2007 130 144
37. B. Moskal, K. Miller, and L. King Grading essays in computer ethics: rubrics considered helpful ACM SIGCSE Bull. 34 1 2002 101 105 (Pubitemid 35000249)
38. J.W. Wilkerson, J.F. Nunamaker Jr., and R. Mercer Comparing the defect reduction benefits of code inspection and test-driven development IEEE Trans. Softw. Eng. 38 3 2011
39. M. Muller, and O. Hagner Experiment about test-first programming IEE Proc.-Softw. 149 5 2002 131 136 (Pubitemid 35313947)
40. L. Madeyski Preliminary analysis of the effects of pair programming and test-driven development on the external code quality Proc. 2005 Conf. Softw. Eng.: Evol. Emerg. Technol. 130 2006 113
41. M. Pancur, and M. Ciglaric Impact of test-driven development on productivity, code and tests: a controlled experiment Inform. Softw. Technol. 53 6 2011 557 573
42. A. Gupta, and P. Jalote An experimental evaluation of the effectiveness and efficiency of the test driven development Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) 2007 IEEE Computer Society Madrid, Spain 285 294
43. G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C.A. Visaggio Evaluating advantages of test driven development: a controlled experiment with professionals 5th ACM-IEEE International Symposium on Empirical Software Engineering, September 21, 2006-September 22, 2006 2006 Association for Computing Machinery 364 371 (Pubitemid 46634471)
44. D. Janzen, and H. Saiedian Does test-driven development really improve software design quality? IEEE Softw. 25 2 2008 7784
45. J.H. Vu, N. Frojd, C. Shenkel-Therolf, and D.S. Janzen Evaluating test-driven development in an industry-sponsored capstone project 6th International Conference on Information Technology: New Generations, ITNG 2009, April 27, 2009-April 29, 2009 2009 IEEE Computer Society Las Vegas, NV, United states 229 234
46. B. George, and L.A. Williams A structured experiment of test-driven development Inform. Softw. Technol. 46 2004 337 342
47. D.S. Janzen, and H. Saiedian A leveled examination of test-driven development acceptance 29th International Conference on Software Engineering, ICSE 2007, May 20, 2007-May 26, 2007 2007 IEEE Computer Society Minneapolis, MN, United states 719 722 (Pubitemid 47430706)
48. C. Desai, D.S. Janzen, and J. Clements Implications of integrating test-driven development into CS1/CS2 curricula 40th ACM Technical Symposium on Computer Science Education, SIGCSE 2009, March 4, 2009-March 7, 2009 2009 Association for Computing Machinery Chattanooga, TN, United states 148 152
49. A. Geras, M. Smith, and J. Miller A prototype empirical evaluation of test driven development Proceedings - 10th International Symposium on Software Metrics, METRICS 2004, September 14, 2004-September 16, 2004 2004 IEEE Computer Society 405 416 (Pubitemid 40338179)
50. L. Madeyski The impact of test-first programming on branch coverage and mutation score indicator of unit tests: an experiment Inform. Softw. Technol. 52 2 2010 169 184
51. D.S. Janzen, and H. Saiedian Test-driven learning in early programming courses 39th ACM Technical Symposium on Computer Science Education, SIGCSE 2008, March 12, 2008-March 15, 2008 2008 Association for Computing Machinery Portland, OR, United states 532 536
52. T. Flohr, and T. Schneider An XP experiment with students - setup and problems Proceedings of the 6th International Conference on Product Focused Software Process Improvement, PROFES 2005, 13-15 June 2005 2005 Springer-Verlag Berlin, Germany 474 486 (Pubitemid 41423006)
53. L. Madeyski, L. Szala, The impact of test-driven development on software development productivity - an empirical study, 2007, pp. 200-211.
54. D.S. Janzen, and H. Saiedian On the influence of test-driven development on software design Proceedings of the 19th Conference on Software Engineering Education and Training, 2006 2006 IEEE 141 148 (Pubitemid 44592196)
55. M.A. Domino, R.W. Collins, A.R. Hevner, and C.F. Cohen Conflict in collaborative software development Proceedings of the 2003 ACM SIGMIS CPR Conference, April 10, 2003-April 12, 2003 2003 Association for Computing Machinery Philadelphia, PA, United states 44 51
56. T. Flohr, T. Schneider, Lessons learned from an xp experiment with students: test-first needs more teachings, in: PROFES, 2006, pp. 305-318. (Pubitemid 44090525)
57. N. Laranjeiro, and M. Vieira Extending test-driven development for robust web services 2009 2nd International Conference on Dependability, DEPEND 2009, June 18, 2009-June 23, 2009 2009 IEEE Computer Society Athens, Glyfada, Greece 122 127
58. L. Huang, and M. Holcombe Empirical assessment of test-first approach 1st Testing: Academic and Industrial Conference - Practice and Research Techniques, TAIC PART 2006, August 29, 2006-August 31, 2006 2006 IEEE Computer Society Windsor, United kingdom 197 200
59. T. Dogsa, and D. Batic The effectiveness of test-driven development: an industrial case study Softw. Qual. J. 19 4 2011 643 661
60. S. Bannerman, and A. Martin A multiple comparative study of test-with development product changes and their effects on team speed and product quality Empir. Softw. Eng. 16 2 2011 177 210
61. M. Siniaalto, and P. Abrahamsson A comparative case study on the impact of test-driven development on program design and test coverage 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, September 20, 2007-September 21, 2007 2007 Inst. of Elec. and Elec. Eng. Computer Society 275 284
62. L.-O. Damm, and L. Lundberg Results from introducing component-level test automation and test-driven development J. Syst. Softw. 79 7 2006 1001 1014 (Pubitemid 43765969)
63. O.P.N. Slyngstad, J. Li, R. Conradi, H. Rnneberg, E. Landre, and H. Wesenberg The impact of test driven development on the evolution of a reusable framework of components - an industrial case study 3rd International Conference on Software Engineering Advances, ICSEA 2008, October 26, 2008-October 31, 2008 2008 IEEE Computer Society Sliema, Malta 214 223
64. T. Bhat, and N. Nagappan Evaluating the efficacy of test-driven development: industrial case studies ISCE'06 - 5th ACM-IEEE International Symposium on Empirical Software Engineering, September 21, 2006-September 22, 2006 2006 Association for Computing Machinery 356 363 (Pubitemid 46634470)
65. L. Crispin Driving software quality: how test-driven development impacts software quality IEEE Softw. 23 6 2006 70 71 (Pubitemid 44824159)
66. N. Nagappan, E.M. Maximilien, T. Bhat, and L. Williams Realizing quality improvement through test driven development: results and experiences of four industrial teams Empir. Softw. Eng. 13 3 2008 289 302
67. L. Williams, E. Maximilien, and M. Vouk Test-driven development as a defect-reduction practice 14th International Symposium on Software Reliability Engineering, 17-20 November 2003 2003 IEEE Comput. Soc Los Alamitos, CA, USA 34 45
68. K. McDaid, A. Rust, B. Bishop, Test-driven development: can it work for spreadsheets? in: Proceedings of the 4th International Workshop on End-user Software Engineering, 2008, p. 2529.
69. E. Maximilien, and L. Williams Assessing test-driven development at IBM IEEE 25th International Conference on Software Engineering, 3-10 May 2003 2003 IEEE Comput. Soc. Los Alamitos, CA, USA 564 569
70. M. Ficco, R. Pietrantuono, and S. Russo Bug localization in test-driven development Adv. Softw. Eng. 2011 2011 2
71. M. Siniaalto, P. Abrahamsson, Does test-driven development improve the program code? alarming results from a comparative case study, 2007, pp. 143-156.
72. M.F. Aniche, and M.A. Gerosa Most common mistakes in test-driven development practice: results from an online survey with developers 3rd International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2010, April 6, 2010-April 10, 2010 2010 IEEE Computer Society Paris, France 469 478
73. C. Desai, D. Janzen, and K. Savage A survey of evidence for test-driven development in academia SIGCSE Bull. 40 2 2008 97 101
74. J. Sanchez, L. Williams, E. Maximilien, On the sustained use of a test-driven development practice at IBM, in: AGILE 2007, p. 514.
75. K. Petersen Measuring and predicting software productivity: a systematic map and review Inform. Softw. Technol. 53 4 2011 317 343
76. N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach, Revised, second ed., Course Technology, 1998.
77. M. Domino, R. Collins, and A. Hevner Controlled experimentation on adaptations of pair programming Inform. Technol. Manage. 8 4 2007 297 312
78. L. Madeyski Test-Driven Development: An Empirical Evaluation of Agile Practice 2010 Springer
79. L. Zhang, S. Akifuji, K. Kawai, and T. Morioka Comparison between test driven development and waterfall development in a small-scale project Extreme Programming and Agile Processes in Software Engineering 2006 Springer 211 212 (Pubitemid 44098526)
80. S. Rahman Applying the tbc method in introductory programming courses 37th Annual Frontiers In Education Conference-Global Engineering: Knowledge Without Borders, Opportunities Without Passports, 2007, FIE'07 2007 IEEE pp. T1E-20
81. S.H. Edwards, Using test-driven development in the classroom: providing students with automatic, concrete feedback on performance, in: Proceedings of the International Conference on Education and Information Systems: Technologies and Applications EISTA, vol. 3, 2003.
82. B. George, and L. Williams An initial investigation of test driven development in industry Proceedings of the 2003 ACM Symposium on Applied Computing 2003 ACM 1135 1139
83. S. Xu, and T. Li Evaluation of test-driven development: an academic case study Software Engineering Research, Management and Applications 2009 2009 Springer 229 238
84. G. Melnik, and F. Maurer A cross-program investigation of students' perceptions of agile methods Proceedings of the 27th International Conference on Software Engineering, 2005, ICSE 2005 2005 IEEE 481 488
85. P. Abrahamsson, A. Hanhineva, and J. Jäälinoja Improving business agility through technical solutions: a case study on test-driven development in mobile software development Business Agility and Information Technology Diffusion 2005 Springer 227 243
86. R.A. Ynchausti, Integrating unit testing into a software development teams process, in: Intl. Conf. eXtreme Programming and Flexible Processes in Software Engineering (XP 2001), 2001, pp. 79-83.
87. S. Yenduri, L.A. Perkins, Impact of using test-driven development: a case study, in: Software Engineering Research and Practice, 2006, pp. 126-129.
88. R. Kaufmann, and D. Janzen Implications of test-driven development: a pilot study Companion of the 18th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications 2003 ACM 298 299
89. C. Gencel, K. Petersen, Worldviews, research methods, and their relationsihpi to validity in empirical software engineering research, in: Proceedings of the International Workshop on Software Measurement (Mensura 2013), 2013.