The effects of test-driven development on external quality and productivity: A meta-analysis

IEEE Transactions on Software Engineering

Volumn 39, Issue 6, 2013, Pages 835-856

  Rafique, Yahya ^a   Mǐsí, Vojislav B ^a  

^a RYERSON UNIVERSITY   (Canada)

Author keywords

agile software development; code quality; meta analysis; programmer productivity; Test driven development

Indexed keywords

AGILE SOFTWARE DEVELOPMENT; CODE QUALITY; META-ANALYSIS; PROGRAMMER PRODUCTIVITY; TEST DRIVEN DEVELOPMENT;

DROPS; PRODUCTIVITY; SOFTWARE ENGINEERING;

QUALITY CONTROL;

EID: 84878407475     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2012.28     Document Type: Review

References (71)

1. S. Ambler, "Test-Driven Development Is the Combination of Test First Development and Refactoring," Dr. Dobbs' Agile Newsletter, June 2006.
2. D. Astels, Test Driven Development: A Practical Guide. Prentice-Hall, 2003.
3. K. Beck, Test-Driven Development: By Example. Prentice-Hall, 2003.
4. J. Andrea, "Test-Driven Development: Driving New Standards of Beauty," Beautiful Testing: Leading Professionals Reveal How They Improve Software, pp. 181-194, O'Reilly Media, 2009.
5. H. Erdogmus, M. Morisio, and M. Torchiano, "On the Effectiveness of the Test-First Approach to Programming," IEEE Trans. Software Eng., vol. 31, no. 3, pp. 226-237, Mar. 2005. (Pubitemid 40890004)
6. M. Lipsey and D. Wilson, Practical Meta-Analysis. Sage Publications, Inc., 2001.
7. B. Kitchenham and S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software Engineering," Keele Univ., Technical Report EBSE 2007-001, 2007.
8. J. Hannay, T. Dyba°, E. Arisholm, and D. Sjøberg, "The Effectiveness of Pair Programming: A Meta-Analysis," Information and Software Technology, vol. 51, no. 7, pp. 1110-1122, 2009.
9. T. Huedo-Medina, J. Sa'nchez-Meca, F. Mari'n-Marti'nez, and J. Botella, "Assessing Heterogeneity in Meta-Analysis: Q statistic or I2 Index?" Psychological Methods, vol. 11, no. 2, pp. 193-206, 2006. (Pubitemid 44001277)
10. G. Melnik and F. Maurer, "A Cross-Program Investigation of Students' Perceptions of Agile Methods," Proc. 27th Int'l Conf. Software Eng., pp. 481-488, 2005.
11. S. Kollanus and V. Isomottonen, "Understanding TDD in Academic Environment: Experiences from Two Experiments," Proc. Eighth Int'l Conf. Computing Education Research, pp. 25-31, 2008.
12. A. Rendell, "Effective and Pragmatic Test Driven Development," Proc. AGILE, pp. 298-303, 2008.
13. J. Langr, "Evolution of Test and Code via Test-First Design," Proc. ACM Conf. Object-Oriented Programming, Systems, Languages, and Applications, http://www.objectmentor.com/resources/articles/ tfd.pdf, Oct. 2001.
14. D.H. Steinberg, "The Effect of Unit Tests on Entry Points, Coupling and Cohesion in an Introductory Java Programming Course," Proc. XP Universe, Oct. 2001.
15. P. Abrahamsson, A. Hanhineva, and J. Jaalinoja, "Improving Business Agility through Technical Solutions: A Case Study on Test-Driven Development in Mobile Software Development," Proc. Int'l Working Conf. Int'l Federation for Information Processing, pp. 227-243, 2005.
16. J. Sanchez, L. Williams, and E. Maximilien, "On the Sustained Use of a Test-Driven Development Practice at IBM," Proc. AGILE, pp. 5-14, 2007.
17. L. Madeyski, "The Impact of Pair Programming and Test-Driven Development on Package Dependencies in Object-Oriented Design - An Experiment," Proc. Seventh Int'l Conf. Product-Focused Software Process Improvement, pp. 278-289, 2006. (Pubitemid 44090523)
18. M. Siniaalto, "The Impact of Test-Driven Development on Design Quality," Information Technology for European Advancement, technical report, D.5.2.10 v1.0, 2006.
19. D. Janzen, C.S. Turner, and H. Saiedian, "Empirical Software Eng. in Industry Short Courses," Proc. 20th Conf. Software Eng. Education and Training, pp. 89-96, 2007. (Pubitemid 47516520)
20. M. Siniaalto and P. Abrahamsson, "A Comparative Case Study on the Impact of Test-Driven Development on Program Design and Test Coverage," Proc. First Int'l Symp. Empirical Software Eng. and Measurement, pp. 275-284, 2007.
21. M. Siniaalto and P. Abrahamsson, "Does Test-Driven Development Improve the Program Code? Alarming Results from a Comparative Case Study," Proc. Second IFIP TC 2 Central and East European Conf. Software Eng. Techniques, pp. 143-156, 2007.
22. D. Janzen and H. Saiedian, "Does Test-Driven Development Really Improve Software Design Quality," IEEE Software, vol. 25, no. 2, pp. 77-84, Mar./Apr. 2008. (Pubitemid 351404579)
23. L. Madeyski, "The Impact of Test-First Programming on Branch Coverage and Mutation Score Indicator of Unit Tests: An Experiment," Information and Software Technology, vol. 52, no. 2, pp. 169-184, 2010.
24. R. Ynchausti, "Integrating Unit Testing into a Software Development Teams Process," Proc. Second Int'l Conf. Extreme Programming and Flexible Processes in Software Eng., pp. 79-83, May 2001.
25. L. Damm, L. Lundberg, and D. Olsson, "Introducing Test Automation and Test-Driven Development: An Experience Report," Electronic Notes in Theoretical Computer Science, vol. 116, pp. 3-15, 2005. (Pubitemid 40415088)
26. L. Madeyski and L- . Szała, "The Impact of Test-Driven Development on Software Development Productivity - An Empirical Study," Proc. 14th European Conf. Software Process Improvement, pp. 200-211, 2007.
27. B. George and L. Williams, "An Initial Investigation of Test Driven Development in Industry," Proc. ACM Symp. Applied Computing, pp. 1135-1139, 2003.
28. B. George, "Analysis and Quantification of Test-Driven Development Approach," master's thesis, North Carolina State Univ., 2002.
29. E. Maximilien and L. Williams, "Assessing Test-Driven Development at IBM," Proc. 25th Int'l Conf. Software Eng., pp. 564-569, 2003.
30. L. Williams, E. Maximilien, and M. Vouk, "Test-Driven Development as a Defect-Reduction Practice," Proc. IEEE Int'l Symp. Software Reliability Eng., pp. 34-45, 2003.
31. S. Edwards, "Using Software Testing to Move Students from Trialand- Error to Reflection-in-Action," ACM SIGCSE Bulletin, vol. 36, no. 1, pp. 26-30, 2004. (Pubitemid 43773190)
32. S. Edwards, "Using Test-Driven Development in the Classroom: Providing Students with Automatic, Concrete Feedback on Performance," Proc. Int'l Conf. Education and Information Systems Technologies and Applications, vol. 3, pp. 421-426, 2003.
33. B. George and L. Williams, "A Structured Experiment of Test- Driven Development," Information and Software Technology, vol. 46, no. 5, pp. 337-342, 2004.
34. A. Geras, "The Effectiveness of Test-Driven Development," master's thesis, Univ. of Calgary, Canada, 2004.
35. A. Geras, M. Smith, and J. Miller, "A Prototype Empirical Evaluation of Test Driven Development," Proc. 10th Int'l Symp. Software Metrics, pp. 405-416, 2004. (Pubitemid 40338179)
36. A. Hanhineva, "Test-Driven Development in Mobile Java Environment," master's thesis, Univ. of Oulu, Finland, 2004.
37. T. Bhat and N. Nagappan, "Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng., pp. 356-363, 2006. (Pubitemid 46634470)
38. N. Nagappan, E. Maximilien, T. Bhat, and L. Williams, "Realizing Quality Improvement through Test Driven Development: Results and Experiences of Four Industrial Teams," Empirical Software Eng., vol. 13, no. 3, pp. 289-302, 2008.
39. G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C. Visaggio, "Productivity of Test Driven Development: A Controlled Experiment with Professionals," Proc. Seventh Int'l Conf. Product-Focused Software Process Improvement, pp. 383-388, 2006. (Pubitemid 44090531)
40. G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C. Visaggio, "Evaluating Advantages of Test Driven Development: A Controlled Experiment with Professionals," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng., pp. 364-371, 2006. (Pubitemid 46634471)
41. D. Janzen and H. Saiedian, "On the Influence of Test-Driven Development on Software Design," Proc. 19th Conf. Software Eng. Education and Training, pp. 141-148, Apr. 2006. (Pubitemid 44592196)
42. D. Janzen, "An Empirical Evaluation of the Impact of Test-Driven Development on Software Quality," PhD dissertation, Univ. of Kansas, 2006.
43. F. Shull, G. Melnik, B. Turhan, L. Layman, M. Diep, and H. Erdogmus, "What Do We Know about Test-Driven Development?" IEEE Software, vol. 27, no. 6, pp. 16-19, Nov./Dec. 2010.
44. http://www.meta-analysis.com/, 2013.
45. V. Kampenes, T. Dyba°, J. Hannay, and D. Sjøberg, "A Systematic Review of Effect Size in Software Engineering Experiments," Information and Software Technology, vol. 49, nos. 11/12, pp. 1073- 1086, 2007. (Pubitemid 47464910)
46. M. Borenstein, L.V. Hedges, J.P. Higgins, and H.R. Rothstein, "A Basic Introduction to Fixed-Effect and Random-Effects Models for Meta-Analysis," Research Synthesis Methods, vol. 1, no. 2, pp. 97-111, 2010.
47. A. Gupta and P. Jalote, "An Experimental Evaluation of the Effectiveness and Efficiency of the Test Driven Development," Proc. First Int'l Symp. Empirical Software Eng. and Measurement, pp. 285-294, 2007.
48. L. Madeyski, "Preliminary Analysis of the Effects of Pair Programming and Test-Driven Development on the External Code Quality," Proc. Conf. Software Eng.: Evolution and Emerging Technologies, pp. 113-123, 2005.
49. M. Pan-cur and M. Ciglari-c, "Impact of Test-Driven Development on Productivity, Code, and Tests: A Controlled Experiment," Information and Software Technology, vol. 53, pp. 557-573, 2011.
50. C. Desai, D. Janzen, and J. Clements, "Implications of Integrating Test-Driven Development into CS1/CS2 Curricula," ACM SIGCSE Bulletin, vol. 41, no. 1, pp. 148-152, 2009.
51. T. Flohr and T. Schneider, "Lessons Learned from an XP Experiment with Students: Test-First Needs More Teachings," Product-Focused Software Process Improvement, pp. 305-318, Springer, 2006. (Pubitemid 44090525)
52. L. Huang and M. Holcombe, "Empirical Investigation Towards the Effectiveness of Test First Programming," Information and Software Technology, vol. 51, no. 1, pp. 182-194, 2009.
53. R. Kaufmann and D. Janzen, "Implications of Test-Driven Development: A Pilot Study," Proc. 18th Ann. ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 298-299, 2003.
54. M.M. Müller and O. Hagner, "Experiment about Test-First Programming," Proc. IEE Software, vol. 149, no. 5, pp. 131-136, Oct. 2002. (Pubitemid 35313947)
55. M. Pan-cur, M. Ciglari-c, M. Trampu-s, and T. Vidmar, "Towards Empirical Evaluation of Test-Driven Development in a University Environment," Proc. Int'l Conf. Computer as a Tool, pp. 83-86, 2003.
56. S. Rahman, "Applying the TBC Method in Introductory Programming Courses," Proc. 37th Ann. Frontiers in Education Conf.-Global Eng.: Knowledge without Borders, Opportunities without Passports, pp. T1E-20-T1E-21, 2007.
57. J. Vu, N. Frojd, C. Shenkel-Therolf, and D. Janzen, "Evaluating Test-Driven Development in an Industry-Sponsored Capstone Project," Proc. Sixth Int'l Conf. Information Technology: New Generations, pp. 229-234, 2009.
58. S. Xu and T. Li, "Evaluation of Test-Driven Development: An Academic Case Study," Software Eng. Research, Management and Applications, pp. 229-238, Springer, 2009.
59. S. Yenduri and L. Perkins, "Impact of Using Test-Driven Development: A Case Study," Software Eng. Research and Practice, pp. 126-129, 2006.
60. L. Zhang, S. Akifuji, K. Kawai, and T. Morioka, "Comparison between Test Driven Development and Waterfall Development in a Small-Scale Project," Extreme Programming and Agile Processes in Software Engineering, pp. 211-212, Springer, 2006. (Pubitemid 44098526)
61. T. Dogsa and D. Bati c, "The Effectiveness of Test-Driven Development: An Industrial Case Study," Software Quality J., vol. 19, pp. 643-661, 2011.
62. K. Lui and K. Chan, "Test Driven Development and Software Process Improvement in China," Extreme Programming and Agile Processes in Software Eng., pp. 219-222, Springer, 2004. (Pubitemid 38846789)
63. O. Slyngstad, J. Li, R. Conradi, H. Ronneberg, E. Landre, and H. Wesenberg, "The Impact of Test Driven Development on the Evolution of a Reusable Framework of Components: An Industrial Case Study," Proc. Third Int'l Conf. Software Eng. Advances, pp. 214- 223, 2008.
64. P. Hodgetts, "Refactoring the Development Process: Experiences with the Incremental Adoption of Agile Practices," Proc. Agile Development Conf., pp. 106-113, 2004. (Pubitemid 40442846)
65. B. Turhan, L. Layman, M. Diep, H. Erdogmus, and F. Shull, "How Effective Is Test Driven Development?" Making Software What Really Works, and Why We Believe It, A. Oram and G. Wilson, eds. pp. 207-219, O'Reilly Media, 2010.
66. M. Müller and A. Höfer, "The Effect of Experience on the Test- Driven Development Process," Empirical Software Eng., vol. 12, pp. 593-615, 2007. (Pubitemid 350045889)
67. A. Höfer and M. Philipp, "An Empirical Study on the TDD Conformance of Novice and Expert Pair Programmers," Agile Processes in Software Eng. and Extreme Programming, pp. 33-42, Springer, 2009.
68. L. Madeyski, "Is External Code Quality Correlated with Programming Experience or Feelgood Factor?" Extreme Programming and Agile Processes in Software Eng., pp. 65-74, Springer, 2006. (Pubitemid 44098505)
69. M. Siniaalto, "Test Driven Development: Empirical Body of Evidence," Information Technology for European Advancement, Eindhoven, The Netherlands, Agile Deliverable, vol. D.2.7, 2006.
70. S. Kollanus, "Test-Driven Development - Still a Promising Approach?" Proc. Seventh Int'l Conf. Quality of Information and Comm. Technology, pp. 403-408, Oct. 2010.
71. L. Damm and L. Lundberg, "Quality Impact of Introducing Component-Level Test Automation and Test-Driven Development," Software Process Improvement, P. Abrahamsson, N. Baddoo, T. Margaria, and R. Messnarz, eds. vol. 4764, pp. 187-199, Springer, 2007.