The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment

Information and Software Technology

Volumn 52, Issue 2, 2010, Pages 169-184

  Madeyski, Lech ^a  

^a WROCLAW UNIVERSITY OF TECHNOLOGY   (Poland)

Author keywords

Empirical study; Test driven development; Test First programming; Unit tests

Indexed keywords

BRANCH COVERAGE; CODE COVERAGE; DEPENDENT VARIABLES; EMPIRICAL STUDIES; EXPERIMENTAL EVALUATION; MULTI VARIATE ANALYSIS; MUTATION SCORE; MUTATION TESTING; PROGRAMMING TECHNIQUE; SOFTWARE DEVELOPMENT PRACTICES; STATISTICALLY SIGNIFICANT DIFFERENCE; TEST DRIVEN DEVELOPMENT; TEST RESULTS; UNIT TESTS;

DECISION MAKING; EXPERIMENTS; FAULT DETECTION; FLUORINE CONTAINING POLYMERS; MULTIVARIANT ANALYSIS; REGRESSION ANALYSIS; SOFTWARE TESTING;

TESTING;

EID: 70450225064     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2009.08.007     Document Type: Article

References (90)

1. Beck K., and Andres C. Extreme Programming Explained: Embrace Change. second ed. (2004), Addison-Wesley
2. Beck K. Test Driven Development: By Example (2002), Addison-Wesley
3. Koskela L. Test Driven: Practical TDD and Acceptance TDD for Java Developers (2007), Manning Publications
4. D. Astels, Test Driven development: A Practical Guide, Prentice Hall, 2003.
5. Williams L., Maximilien E.M., and Vouk M. Test-driven development as a defect-reduction practice. ISSRE '03: Proceedings of the 14th International Symposium on Software Reliability Engineering (2003), IEEE Computer Society, Washington, DC, USA 34-48
6. Maximilien E.M., and Williams L.A. Assessing test-driven development at IBM. ICSE '03: Proceedings of the 25th International Conference on Software Engineering (2003), IEEE Computer Society 564-569
7. Canfora G., Cimitile A., Garcia F., Piattini M., and Visaggio C.A. Evaluating advantages of test driven development: a controlled experiment with professionals. ISESE '06: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering (2006), ACM Press, New York, NY, USA 364-371
8. Bhat T., and Nagappan N. Evaluating the efficacy of test-driven development: industrial case studies. ISESE '06: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering (2006), ACM Press, New York, NY, USA 356-363
9. Sanchez J.C., Williams L., and Maximilien E.M. On the sustained use of a test-driven development practice at ibm. AGILE '07: Proceedings of the 2007 Conference on Agile Software Development (2007), IEEE Computer Society, Washington, DC, USA 5-14
10. Nagappan N., Maximilien E.M., Bhat T., and Williams L. Realizing quality improvement through test driven development: results and experiences of four industrial teams. Empirical Software Engineering 13 3 (2008) 289-302
11. Janzen D., and Saiedian H. Does test-driven development really improve software design quality?. IEEE Software 25 2 (2008) 77-84
12. Edwards S.H. Rethinking computer science education from a test-first perspective. OOPSLA '03: Companion of the 18th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (2003), ACM, New York, NY, USA 148-155
13. Edwards S.H. Teaching software testing: automatic grading meets test-first coding. OOPSLA '03: Companion of the 18th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (2003), ACM, New York, NY, USA 318-319
14. G. Melnik, F. Maurer, A cross-program investigation of students' perceptions of agile methods, in: ICSE '05: Proceedings of the 27th International Conference on Software Engineering, 2005, pp. 481-488.
15. Müller M.M., and Hagner O. Experiment about test-first programming. IEE Procedings-Software 149 5 (2002) 131-136
16. M. Pančur, M. Ciglarič, M. Trampuš, T. Vidmar, Towards empirical evaluation of test-driven development in a university environment, in: EUROCON '03: Proceedings of the International Conference on Computer as a Tool, 2003, pp. 83-86.
17. Erdogmus H., Morisio M., and Torchiano M. On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering 31 3 (2005) 226-237
18. Flohr T., and Schneider T. Lessons learned from an XP experiment with students: test-first need more teachings. In: Münch J., and Vierimaa M. (Eds). Product Focused Software Process Improvement. Lecture Notes in Computer Science vol. 4034 (2006), Springer, Berlin, Heidelberg 305-318
19. Madeyski L. Preliminary analysis of the effects of pair programming and test-driven development on the external code quality. In: Zieliński K., and Szmuc T. (Eds). Software Engineering: Evolution and Emerging Technologies. Frontiers in Artificial Intelligence and Applications vol. 130 (2005), IOS Press 113-123.
20. Madeyski L. The impact of pair programming and test-driven development on package dependencies in object-oriented design - an experiment. In: Münch J., and Vierimaa M. (Eds). Product Focused Software Process Improvement. Lecture Notes in Computer Science vol. 4034 (2006), Springer, Berlin, Heidelberg 278-289.
21. Gupta A., and Jalote P. An experimental evaluation of the effectiveness and efficiency of the test driven development. ESEM '07: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (2007), IEEE Computer Society, Washington, DC, USA 285-294
22. Huang L., and Holcombe M. Empirical investigation towards the effectiveness of Test First programming. Information and Software Technology 51 1 (2009) 182-194
23. Jeffries R.E., Anderson A., and Hendrickson C. Extreme Programming Installed (2000), Addison-Wesley Longman Publishing Co. Inc., Boston, MA, USA
24. S. Ambler, Introduction to test driven design (TDD). , 2008 (accessed 2008).
25. B. Mattu, R. Shankar, Test driven design methodology for component-based system, in: First Annual IEEE Systems Conference, 2007, pp. 1-7.
26. B. Marick, How to misuse code coverage, in: Proceedings of the 16th International Conference on Testing Computer Software, Washington, USA. , 1999.
27. P.J. Walsh, A Measure of Test Case Completeness, Ph.D. Thesis, University of New York, 1985.
28. Frankl P.G., Weiss S.N., and Hu C. All-uses vs. mutation testing: an experimental comparison of effectiveness. Journal of Systems and Software 38 3 (1997) 235-253
29. Offutt A.J., Pan J., Tewary K., and Zhang T. An experimental evaluation of data flow and mutation testing. Software Practice and Experience 26 2 (1996) 165-176
30. Wohlin C., Runeson P., Höst M., Ohlsson M.C., Regnell B., and Wesslén A. Experimentation in Software Engineering: An Introduction (2000), Kluwer Academic Publishers, Norwell, MA, USA
31. Robson C. Real World Research: A Resource for Social Scientists and Practitioner-Researchers (2002), Blackwell Publishing Limited
32. Boehm B.W. A Spiral Model of Software Development and Enhancement. IEEE Computer 21 5 (1988) 61-72
33. IEEE Standard for Software Unit Testing, ANSI/IEEE Std 1008, 1987.
34. L. Madeyski, Empirical studies on the impact of test-first programming, Technical Report I32/09/, Wroclaw University of Technology, Institute of Informatics, , 2009.
35. Madeyski L., and Szała Ł. The impact of test-driven development on software development productivity - an empirical study. In: Abrahamsson P., Baddoo N., Margaria T., and Messnarz R. (Eds). Software Process Improvement. Lecture Notes in Computer Science vol. 4764 (2007), Springer 200-211.
36. R.A. Ynchausti, Integrating unit testing into a software development team's process, in: M. Marchesi, G. Succi (Eds.), XP 2001: Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering, Sardinia, Italy, 2001, pp. 84-87.
37. George B., and Williams L. An initial investigation of test driven development in industry. SAC '03: Proceedings of the 2003 ACM Symposium on Applied Computing (2003), ACM, New York, NY, USA 1135-1139
38. George B., and Williams L.A. A structured experiment of test-driven development. Information and Software Technology 46 5 (2004) 337-342
39. Geras A., Smith M.R., and Miller J. A prototype empirical evaluation of test driven development. IEEE METRICS'2004: Proceedings of the 10th IEEE International Software Metrics Symposium (2004), IEEE Computer Society 405-416
40. Damm L.-O., and Lundberg L. Results from introducing component-level test automation and Test-Driven Development. Journal of Systems and Software 79 7 (2006) 1001-1014
41. Damm L.-O., and Lundberg L. Quality impact of introducing component-level test automation and test-driven development. In: Abrahamsson P., Baddoo N., Margaria T., and Messnarz R. (Eds). Software Process Improvement. Lecture Notes in Computer Science vol. 4764 (2007), Springer 187-199
42. Abrahamsson P., Hanhineva A., and Jäälinoja J. Improving business agility through technical solutions: a case study on test-driven development in mobile software development. In: Baskerville R., Mathiassen L., Pries-Heje J., and DeGross J.I. (Eds). Proceedings of the IFIP TC8 WG 8.6 International Working Conference on Business Agility and Information Technology Diffusion. IFIP International Federation for Information Processing vol. 180 (2005), Springer 1-17
43. Siniaalto M., and Abrahamsson P. A comparative case study on the impact of test-driven development on program design and test coverage. ESEM '07: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (2007), IEEE Computer Society 275-284
44. Müller M.M. The effect of test-driven development on program code. XP '06: Extreme Programming and Agile Processes in Software Engineering, 7th International Conference, XP 2006, Oulu, Finland, June 17-22, 2006 (2006), Springer 94-103
45. Runeson P. Using students as experiment subjects - an analysis of graduate and freshmen student data. EASE '03: Proceedings of 7th International Conference on Empirical Assessment and Evaluation in Software Engineering (2003), British Computer Society 95-102
46. Höst M., Regnell B., and Wohlin C. Using students as subjects - a comparative study of students and professionals in lead-time impact assessment. Empirical Software Engineering 5 3 (2000) 201-214
47. Tichy W.F. Hints for reviewing empirical work in software engineering. Empirical Software Engineering 5 4 (2000) 309-312
48. Kitchenham B., Pfleeger S.L., Pickard L., Jones P., Hoaglin D.C., Emam K.E., and Rosenberg J. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28 8 (2002) 721-734
49. Jeffries R., and Melnik G. Guest editors' introduction: TDD - The art of fearless programming. IEEE Software 24 3 (2007) 24-30
50. Maxwell S.E., and Delaney H.D. Designing Experiments and Analyzing Data: A Model Comparison Perspective. second ed. (2004), Lawrence Erlbaum, Mahwah
51. Madeyski L. On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch J., and Abrahamsson P. (Eds). Product Focused Software Process Improvement. Lecture Notes in Computer Science vol. 4589 (2007), Springer 207-221.
52. Madeyski L. The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites. Wiley, Software Process: Improvement and Practice 13 3 (2008) 281-295.
53. Cook T.D., and Campbell D.T. Quasi-Experimentation: Design and Analysis Issues (1979), Houghton Mifflin Company
54. Fowler M., Beck K., Brant J., Opdyke W., and Roberts D. Refactoring: Improving the Design of Existing Code (1999), Addison-Wesley
55. E. Gamma, K. Beck, JUnit, , 2006 (accessed 2006).
56. Massol V., and Husted T. JUnit in Action. first ed. (2003), Manning Publications
57. Hamill P. Unit Test Frameworks (2004), O'Reilly
58. S. Cornett, Code coverage analysis, , 2007 (accessed 2007).
59. Atlassian Pty Ltd., Clover project, , 2008 (accessed 2008).
60. DeMillo R.A., Lipton R.J., and Sayward F.G. Hints on test data selection: help for the practicing programmer. IEEE Computer 11 4 (1978) 34-41
61. Hamlet R.G. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering 3 4 (1977) 279-290
62. Zhu H., Hall P.A.V., and May J.H.R. Software unit test coverage and adequacy. ACM Computing Surveys 29 4 (1997) 366-427
63. Offutt A.J., and Untch R.H. Mutation Testing for the New Century (2001), Kluwer Academic Publishers, Norwell, MA, USA Ch. Mutation 2000: Uniting the Orthogonal
64. N. Radyk, L. Madeyski, Judy - mutation testing tool for Java. , , 2009 (accessed 2009).
65. Ant project, , 2006 (accessed 2006).
66. Offutt A.J., Lee A., Rothermel G., Untch R.H., and Zapf C. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology 5 2 (1996) 99-118
67. Ammann P., and Offutt J. Introduction to Software Testing (2008), Cambridge University Press, Cambridge, UK
68. Ma Y.-S., Kwon Y.-R., and Offutt J. Inter-class mutation operators for Java. ISSRE '02: Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE'02) (2002), IEEE Computer Society, Washington, DC, USA 352-363
69. Ma Y.-S., Harrold M.J., and Kwon Y.-R. Evaluation of mutation testing for object-oriented programs. ICSE '06: Proceeding of the 28th International Conference on Software Engineering (2006), ACM Press, New York, NY, USA 869-872
70. Höst M., Wohlin C., and Thelin T. Experimental context classification: incentives and experience of subjects. ICSE '05: Proceedings of the 27th International Conference on Software Engineering (2005), ACM Press, New York, NY, USA 470-478
71. Howitt D., and Cramer D. Introduction to SPSS in Psychology. fourth ed. (2008), Pearson Education Limited
72. Stevens J.P. Applied Multivariate Statistics for the Social Sciences. fourth ed. (2002), Lawrence Erlbaum, Mahwah
73. G. Seshadri, Understanding JavaServer Pages Model 2 Architecture - exploring the MVC design pattern, , 1999.
74. Cactus project, , 2006 (accessed 2006).
75. Madeyski L., and Stochmiałek M. Architectural design of modern web applications. Foundations of Computing and Decision Sciences Journal 30 1 (2005) 49-60.
76. Williams L., Shukla A., and Antón A.I. An initial exploration of the relationship between pair programming and brooks' law. ADC '04: Proceedings of the Agile Development Conference (ADC'04) (2004), IEEE Computer Society, Washington, DC, USA 11-20
77. Tabachnick B.G., and Fidell L.S. Using Multivariate Statistics. fifth ed. (2006), Allyn and Bacon, Inc., Needham Heights, MA, USA
78. American Psychological Association. Publication Manual of the American Psychological Association. fifth ed. (2001), American Psychological Association, Washington, DC, USA
79. Field A. Discovering Statistics Using SPSS (2005), SAGE Publications
80. Shadish W.R., Cook T.D., and Campbell D.T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference (2002), Houghton, Mifflin
81. Rubin A., and Babbie E.R. Research Methods for Social Work. fifth ed. (2004), Wadsworth Publishing
82. Prechelt L., Malpohl G., and Phlippsen M. Finding Plagiarisms among a set of programs with JPlag. Journal of Universal Computer Science 8 11 (2002) 1016-1038. < http://www2.informatik.uni-erlangen.de/Forschung/Publikationen/download/jplag.pdf>
83. L.S. Sørumgård, Verification of Process Conformance in Empirical Studies of Software Development, Ph.D. Thesis, The Norwegian University of Science and Technology, , 1997.
84. Wang Y., and Erdogmus H. The role of process measurement in test-driven development. In: Zannier C., Erdogmus H., and Lindstrom L. (Eds). XP/Agile Universe. Lecture Notes in Computer Science vol. 3134 (2004), Springer 32-42
85. Müller M.M., and Höfer A. The effect of experience on the test-driven development process. Empirical Software Engineering 12 6 (2007) 593-615
86. M. Fowler, The new methodology, , 2007 (accessed 2007).
87. Arisholm E., and Sjøberg D.I.K. Evaluating the effect of a delegated versus centralized control style on the maintainability of object-oriented software. IEEE Transactions on Software Engineering 30 8 (2004) 521-534
88. Lipsey M.W., and Wilson D.B. Practical Meta-Analysis (2001), Sage Publications, California, USA
89. Kampenes V.B., Dybå T., Hannay J.E., and Sjøberg D.I.K. Systematic review: a systematic review of effect size in software engineering experiments. Information and Software Technology 49 11-12 (2007) 1073-1086
90. L. Madeyski, The impact of test-first programming on branch coverage and mutation score indicator of unit tests: a raw data from an experiment, (raw data from SPSS), Wroclaw University of Technology, Institute of Informatics, , , 2009.