Which process metrics can significantly improve defect prediction models? An empirical study

Software Quality Journal

Volumn 23, Issue 3, 2015, Pages 393-422

  Madeyski, Lech ^a   Jureczko, Marian ^a  

^a WROCLAW UNIVERSITY OF TECHNOLOGY   (Poland)

Author keywords

Defect prediction models; Process metrics; Product metrics; Software defect prediction; Software metrics

Indexed keywords

DEFECTS; OPEN SOURCE SOFTWARE; OPEN SYSTEMS; QUALITY ASSURANCE;

DEFECT PREDICTION MODELS; PROCESS METRICS; PRODUCT METRICS; SOFTWARE DEFECT PREDICTION; SOFTWARE METRICS;

FORECASTING;

EID: 84928707319     PISSN: 09639314     EISSN: 15731367     Source Type: Journal    
DOI: 10.1007/s11219-014-9241-7     Document Type: Article

References (72)

1. Antoniol, G., Ayari, K., Di Penta, M., Khomh, F., & Guéhéneuc, Y. G. (2008). Is it a bug or an enhancement? A text-based approach to classify change requests. In CASCON’08: Proceedings of the 2008 conference of the center for advanced studies on collaborative research (pp. 304–318) New York, NY: ACM. doi10.1145/1463788.1463819.
2. Arisholm, E., & Briand, L. C. (2006). Predicting fault-prone components in a java legacy system. In ISESE’06: Proceedings of the 2006 ACM/IEEE international symposium on empirical software engineering (pp. 8–17). New York, NY: ACM. doi10.1145/1159733.1159738.
3. Bacchelli, A., D’Ambros, M., & Lanza, M. (2010). Are popular classes more defect prone? In Rosenblum D, Taentzer G (eds) Fundamental approaches to software engineering, Lecture Notes in Computer Science, (Vol. 6013, pp. 59–73). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-12029-9_5.
4. Bansiya, J., & Davis, C. G. (2002). A hierarchical model for object-oriented design quality assessment. IEEE Transactions on Software Engineering, 28(1), 4–17. doi:10.1109/32.979986.
5. Basili, V. R., Briand, L. C., & Melo, W. L. (1996). A validation of object-oriented design metrics as quality indicators. IEEE Transactions on Software Engineering, 22(10), 751–761. doi:10.1109/32.544352.
6. Bell, R. M., Ostrand, T. J., & Weyuker, E. J. (2006). Looking for bugs in all the right places. In ISSTA’06: Proceedings of the 2006 international symposium on Software testing and analysis (pp. 61–72). New York, NY: ACM. doi:10.1145/1146238.1146246.
7. Boehm, B. W., & Papaccio, P. N. (1988). Understanding and controlling software costs. IEEE Transactions on Software Engineering, 14, 1462–1477. doi:10.1109/32.6191.
8. Catal, C., & Diri, B. (2009). A systematic review of software fault prediction studies. Expert Systems with Applications, 36(4), 7346–7354. doi:10.1016/j.eswa.2008.10.027.
9. Chidamber, S. R., & Kemerer, C. F. (1994). A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20(6), 476–493. doi:10.1109/32.295895.
10. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues. Boston, MA: Houghton Mifflin Company.
11. D’Ambros, M., Bacchelli, A., & Lanza, M. (2010a). On the Impact of Design Flaws on Software Defects. In QSIC’10: Proceedings of the 2010 10th international conference on quality software, IEEE computer society (pp. 23–31). Washington, DC. doi:10.1109/QSIC.2010.58.
12. D’Ambros, M., Lanza, M., & Robbes, R. (2010b). An extensive comparison of bug prediction approaches. In MSR’10: Proceedings of the 2010 7th IEEE working conference on mining software repositories (pp. 31–41). doi:10.1109/MSR.2010.5463279.
13. Denaro, G., & Pezzè, M. (2002). An empirical evaluation of fault-proneness models. In Proceedings of the 24th international conference on software engineering, ICSE’02 (pp. 241–251). New York, NY: ACM. doi:10.1145/581339.581371.
14. Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1(2), 170–177.
15. Endres, A., & Rombach, D. (2003). A handbook of software and systems engineering. Reading: Addison-Wesley.
16. Fenton, N., Neil, M., Marsh, W., Hearty, P., Radlinski, L., & Krause, P. (2007). Project data incorporating qualitative factors for improved software defect prediction. In ICSEW’07: Proceedings of the 29th international conference on software engineering workshops, IEEE Computer Society, Washington, DC. doi:10.1109/ICSEW.2007.171.
17. Fenton, N. E., & Neil, M. (1999). A critique of software defect prediction models. IEEE Transactions on Software Engineering, 25, 675–689, doi10.1109/32.815326, http://dl.acm.org/citation.cfm?id=325392.325401.
18. Fenton, N. E., & Ohlsson, N. (2000). Quantitative analysis of faults and failures in a complex software system. IEEE Transactions on Software Engineering, 26(8), 797–814. doi:10.1109/32.879815.
19. Fischer, M., Pinzger, M., & Gall, H. (2003). Populating a release history database from version control and bug tracking systems. In ICSM’03: Proceedings of the international conference on software maintenance (p. 23). Washington, DC: IEEE Computer Society. doi:10.1109/ICSM.2003.1235403.
20. Giger, E., Pinzger, M., & Gall, H. (2011a). Using the gini coefficient for bug prediction in eclipse. In Proceedings of the 12th international workshop on principles of software evolution and the 7th annual ERCIM workshop on software evolution, IWPSE-EVOL’11 (pp. 51–55). New York, NY: ACM. doi:10.1145/2024445.2024455.
21. Giger, E., Pinzger, M., & Gall, H. C. (2011b). Comparing fine-grained source code changes and code churn for bug prediction. In Proceedings of the 8th working conference on mining software repositories, MSR’11 (pp. 83–92). New York, NY: ACM. doi:10.1145/1985441.1985456.
22. Graves, T. L., Karr, A. F., Marron, J. S., & Siy, H. (2000). Predicting fault incidence using software change history. IEEE Transactions on Software Engineering, 26(7), 653–661. doi:10.1109/32.859533.
23. Grissom, R. J., & Kim, J. J. (2012). Effect sizes for research: Univariate and multivariate applications. Taylor and Francis: Routledge.
24. Gyimothy, T., Ferenc, R., & Siket, I. (2005). Empirical validation of object-oriented metrics on open source software for fault prediction. IEEE Transactions on Software Engineering, 31(10), 897–910. doi:10.1109/TSE.2005.112.
25. Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012). A systematic literature review on fault prediction performance in software engineering. IEEE Transactions on Software Engineering, 38(6), 1276–1304. doi:10.1109/TSE.2011.103.
26. Hassan, A. E. (2009). Predicting faults using the complexity of code changes. In ICSE’09: Proceedings of the 31st international conference on software engineering (pp. 78–88). Washington, DC: IEEE Computer Society. doi:10.1109/ICSE.2009.5070510.
27. Henderson-Sellers, B. (1996). Object-oriented metrics: Measures of complexity. Upper Saddle River, NJ: Prentice-Hall.
28. Illes-Seifert, T., & Paech, B. (2010). Exploring the relationship of a file’s history and its fault-proneness: An empirical method and its application to open source programs. Information and Software Technology, 52(5), 539–558. doi:10.1016/j.infsof.2009.11.010.
29. Jureczko, M., & Madeyski, L. (2010). Towards identifying software project clusters with regard to defect prediction. In PROMISE’2010: Proceedings of the 6th international conference on predictor models in software engineering (pp. 9:1–9:10). ACM doi:10.1145/1868328.1868342, http://madeyski.e-informatyka.pl/download/JureczkoMadeyski10f.pdf.
30. Jureczko, M., & Madeyski, L. (2011a). A review of process metrics in defect prediction studies. Metody Informatyki Stosowanej 30(5), 133–145, http://madeyski.e-informatyka.pl/download/Madeyski11.pdf.
31. Jureczko, M., & Madeyski, L. (2011b). Open source project descriptions. Report SPR 1/2014, Faculty of Computer Science and Management, Wroclaw University of Technology, http://madeyski.e-informatyka.pl/download/JureczkoMadeyskiOpenSourceProjects.pdf.
32. Jureczko, M., & Madeyski, L. (2011c). Software product metrics used to build defect prediction models. Report SPR 2/2014, Faculty of Computer Science and Management, Wroclaw University of Technology, http://madeyski.e-informatyka.pl/download/JureczkoMadeyskiSoftwareProductMetrics.pdf.
33. Jureczko, M., & Magott, J. (2012), QualitySpy: A framework for monitoring software development processes. Journal of Theoretical and Applied Computer, Science, 6(1), 35–45. www.jtacs.org/archive/2012/1/4/JTACS_2012_01_04.
34. Jureczko, M., & Spinellis, D. (2010). Using object-oriented design metrics to predict software defects, monographs of system dependability, models and methodology of system dependability (pp. 69–81). Wroclaw, Poland: Wroclaw University of Technology Press. http://www.dmst.aueb.gr/dds/pubs/conf/2010-DepCoS-RELCOMEX-ckjm-defects/html/JS10.
35. Kalinowski, M., Card, D. N., & Travassos, G. H. (2012). Evidence-based guidelines to defect causal analysis. IEEE Software, 29(4), 16–18, doi:10.1109/MS.2012.72.
36. Kampenes, V. B., Dybå, T., Hannay, J. E., & Sjøberg D. I. K. (2007). Systematic review: A systematic review of effect size in software engineering experiments. Information and Software Technology, 49(11–12), 1073–1086. doi:10.1016/j.infsof.2007.02.015.
37. Khoshgoftaar, T. M., Allen, E. B., Halstead, R., Trio, G. P., & Flass, R. M. (1998). Using process history to predict software quality. Computer, 31(4), 66–72. doi:10.1109/2.666844.
38. Kim, S., Zimmermann, T., Whitehead, Jr. E. J., & Zeller, A. (2007). Predicting Faults from Cached History. In ICSE’07: Proceedings of the 29th international conference on software engineering (pp. 489–498). Washington, DC: IEEE Computer Society. doi:10.1109/ICSE.2007.66.
39. Kitchenham, B. (2010). What’s up with software metrics? A preliminary mapping study. Journal of Systems and Software, 83(1), 37–51. doi:10.1016/j.jss.2009.06.041.
40. Layman, L., Kudrjavets, G., & Nagappan, N. (2008). Iterative identification of fault-prone binaries using in-process metrics. In ESEM’08: Proceedings of the second ACM-IEEE international symposium on empirical software engineering and Measurement (pp. 206–212), New York, NY: ACM. doi:10.1145/1414004.1414038.
41. Madeyski, L. (2006). Is external code quality correlated with programming experience or feelgood factor? Lecture Notes in Computer Science (Vol. 4044, pp. 65–74). doi:10.1007/11774129_7 Draft: http://madeyski.e-informatyka.pl/download/Madeyski06b.pdf.
42. Madeyski, L. (2010). Test-driven development: An empirical evaluation of agile practice. Heidelberg, Dordrecht, London, New York: Springer, doi:10.1007/978-3-642-04288-1, http://www.springer.com/978-3-642-04287-4.
43. Martin, R. (1994). OO design quality metrics—an analysis of dependencies. In OOPSLA’94: Proceedings of workshop pragmatic and theoretical directions in object-oriented software metrics (pp. 1–8), http://www.objectmentor.com/resources/articles/oodmetrc.
44. Matsumoto, S., Kamei, Y., Monden, A., ichi Matsumoto, K., & Nakamura, M. (2010). An analysis of developer metrics for fault prediction. In PROMISE’10: Proceedings of the sixth international conference on predictor models in software engineering (pp. 18:1–18:9). ACM. doi:10.1145/1868328.1868356.
45. McCabe, T. (1976). A complexity measure. IEEE Transactions on Software Engineering, 2, 308–320. doi:10.1109/TSE.1976.233837.
46. Moser, R., Pedrycz, W., & Succi, G. (2008). A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In ICSE’08: Proceedings of the 30th international conference on software engineering (pp. 181–190). New York, NY: ACM. doi:10.1145/1368088.1368114.
47. Nagappan, N., & Ball, T. (2005). Use of relative code churn measures to predict system defect density. In ICSE’05: Proceedings of the 27th international conference on software engineering (pp. 284–292). New York, NY: ACM. doi:10.1145/1062455.1062514
48. Nagappan, N., & Ball, T. (2007). Using software dependencies and churn metrics to predict field failures: An empirical case study. In ESEM’07: Proceedings of the first international symposium on empirical software engineering and measurement (pp. 364–373). Washington, DC: IEEE Computer Society. doi:10.1109/ESEM.2007.13.
49. Nagappan, N., Murphy, B., & Basili, V. (2008). The influence of organizational structure on software quality: an empirical case study. In ICSE’08: Proceedings of the 30th international conference on software engineering (pp. 521–530). New York, NY: ACM. doi10.1145/1368088.1368160.
50. Nagappan, N., Zeller, A., Zimmermann, T., Herzig, K., & Murphy, B. (2010). Change bursts as defect predictors. Technical report, Microsoft Research, http://research.microsoft.com/pubs/137315/bursts.
51. Ostrand, T. J., & Weyuker, E. J. (2002). The distribution of faults in a large industrial software system. In ISSTA’02: Proceedings of the 2002 ACM SIGSOFT international symposium on software testing and analysis (pp. 55–64). New York, NY: ACM. doi:10.1145/566172.566181.
52. Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2004). Where the bugs are. In ISSTA’04: Proceedings of the 2004 ACM SIGSOFT international symposium on software testing and analysis (pp. 86–96). New York, NY: ACM. doi:10.1145/1007512.1007524.
53. Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2005). Predicting the location and number of faults in large software systems. IEEE Transactions on Software Engineering, 31(4), 340–355. doi:10.1109/TSE.2005.49.
54. Petroski, H. (1985). To engineer is human: The role of failure in successful design. New York: St. Martin’s Press.
55. Purao, S., & Vaishnavi, V. (2003). Product metrics for object-oriented systems. ACM Computing Surveys, 35(2), 191–221. doi:10.1145/857076.857090.
56. Purushothaman, R., & Perry, D. E. (2005). Toward understanding the rhetoric of small source code changes. IEEE Transactions on Software Engineering, 31, 511–526. doi:10.1109/TSE.2005.74.
57. Ratzinger, J., Pinzger, M., & Gall, H. (2007). EQ-Mine: Predicting short-term defects for software evolution. In M. Dwyer & A. Lopes (Eds.), Fundamental approaches to software engineering, Lecture Notes in Computer Science, (Vol. 4422, pp. 12–26) Berlin/Heidelberg: Springer. doi:10.1007/978-3-540-71289-3_3.
58. Rosenthal, R. (1991). Meta-analytic procedures for social research (2nd ed.). Newbury Park, CA: SAGE.
59. Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. doi:10.1146/annurev.psych.52.1.59.
60. Schröter, A., Zimmermann, T., Premraj, R., & Zeller, A. (2006). If your bug database could talk. In Proceedings of the 5th international symposium on empirical software engineering, volume II: Short papers and posters (pp. 18–20).
61. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
62. Shihab, E., Jiang, Z. M., Ibrahim, W. M., Adams, B., & Hassan, A. E. (2010). Understanding the impact of code and process metrics on post-release defects: A case study on the Eclipse project. In ESEM’10: Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement (pp. 1–10). New York, NY:ACM. doi:10.1145/1852786.1852792.
63. Śliwerski, J., Zimmermann, T., & Zeller, A. (2005). When do changes induce fixes? In MSR’05: Proceedings of the 2005 international workshop on mining software repositories (pp. 1–5). New York, NY: ACM doi:10.1145/1083142.1083147.
64. Tang, M. H., Kao, M. H., & Chen, M. H. (1999). An empirical study on object-oriented metrics. In METRICS’99: Proceedings of the 6th international symposium on software metrics (p. 242). Washington, DC: IEEE Computer Society. doi: 10.1109/METRIC.1999.809745.
65. Wahyudin, D., Schatten, A., Winkler, D., Tjoa, A. M., & Biffl, S. (2008). Defect prediction using combined product and project metrics—a case study from the open source “Apache” MyFaces project family. In SEAA’08: Proceedings of the 2008 34th euromicro conference software engineering and advanced applications (pp. 207–215). Washington, DC: IEEE Computer Society. doi:10.1109/SEAA.2008.36.
66. Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2006). Adapting a fault prediction model to allow widespread usage. In PROMISE’06: Proceedings of the 4th International Workshop on Predictor Models in Software Engineering (pp. 1–5). New York, NY: ACM doi:10.1145/857076.857090.
67. Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2007). Using developer information as a factor for fault prediction. In PROMISE ’07: Proceedings of the third international workshop on predictor models in software engineering (p. 8). Washington, DC: IEEE Computer Society. doi:10.1109/PROMISE.2007.14.
68. Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2008). Do too many cooks spoil the broth? Using the number of developers to enhance defect prediction models. Empirical Software Engineering, 13(5), 539–559. doi:10.1007/s10664-008-9082-8.
69. Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2010). Programmer-based Fault Prediction. In PROMISE ’10: Proceedings of the sixth international conference on predictor models in software engineering (pp. 19:1–19:10). ACM. doi:10.1145/1868328.1868357.
70. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000). Experimentation in software engineering: An introduction. Norwell, MA: Kluwer Academic Publishers.
71. Zimmermann, T., Premraj, R., & Zeller, A. (2007). Predicting Defects for Eclipse. In PROMISE ’07: Proceedings of the third international workshop on predictor models in software engineering (p. 9), Washington, DC: IEEE Computer Society. doi:10.1109/PROMISE.2007.10.
72. Zimmermann, T., Nagappan, N., Gall, H., Giger, E., & Murphy, B. (2009). Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. In ESEC/FSE ’09: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (pp. 91–100). New York, NY: ACM. doi:10.1145/1595696.1595713.