Researcher bias: The use of machine learning in software defect prediction

IEEE Transactions on Software Engineering

Volumn 40, Issue 6, 2014, Pages 603-616

  Shepperd, Martin ^a   Bowes, David ^b   Hall, Tracy ^a  

^a BRUNEL UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF HERTFORDSHIRE   (United Kingdom)

Author keywords

meta analysis; researcher bias; Software defect prediction

Indexed keywords

ARTIFICIAL INTELLIGENCE; DEFECTS; FORECASTING; RESEARCH; REVERSE ENGINEERING;

APPLICATIONS DOMAINS; DEFECT PREDICTION MODELS; EMPIRICAL PREDICTIONS; META-ANALYSIS; PREDICTIVE PERFORMANCE; RELATIVE CONTRIBUTION; RESEARCHER BIAS; SOFTWARE DEFECT PREDICTION;

QUALITY CONTROL;

EID: 84903176990     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2014.2322358     Document Type: Article

References (95)

1. C. Catal and B. Diri, "A systematic review of software fault prediction studies," Expert Syst. Appl., vol. 36, no. 4, pp. 7346-7354, 2009.
2. T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell, "A systematic literature review on fault prediction performance in software engineering," IEEE Trans. Softw. Eng., vol. 38, no. 6, pp. 1276-1304, Nov./Dec. 2012.
3. S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, "Benchmarking classification models for software defect prediction: A proposed framework and novel findings," IEEE Trans. Softw. Eng., vol. 34, no. 4, pp. 485-496, Jul./Aug. 2008.
4. I. Myrtveit, E. Stensrud, and M. Shepperd, "Reliability and validity in comparative studies of software prediction models," IEEE Trans. Softw. Eng., vol. 31, no. 5, pp. 380-391, May 2005. (Pubitemid 41046926)
5. T. Raeder, T. Hoens, and N. Chawla, "Consequences of variability in classifier performance estimates," in Proc. IEEE 10th Int. Conf. Data Mining, 2010, pp. 421-430.
6. Q. Song, Z. Jia, M. Shepperd, S. Ying, and J. Liu, "A general software defect-proneness prediction framework," IEEE Trans. Softw. Eng., vol. 11, no. 3, pp. 356-370, May 2011.
7. D. Ince, L. Hatton, and J. Graham-Cumming, "The case for open computer programs," Nature, vol. 482, pp. 485-488, 2012.
8. P. Baldi, S. Brunak, Y. Chauvin, C. Andersen, and H. Nielsen, "Assessing the accuracy of prediction algorithms for classification: An overview," Bioinformatics, vol. 16, no. 5, pp. 412-424, 2000. (Pubitemid 30487963)
9. R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," in Proc. Int. Joint Conf. Artif. Intell., 1995, pp. 1137-1143.
10. J. D. Rodríguez, A. Pérez, and J. A. Lozano, "Sensitivity analysis of k-fold cross validation in prediction error estimation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 3, pp. 569-575, Mar. 2010.
11. T. Menzies, B. Caglayan, Z. He, E. Kocaguneli, J. Krall, F. Peters, and B. Turhan. (2012, Jun.). The PROMISE repository of empirical software engineering data [Online]. Available: http://promisedata.googlecode.com
12. D. Gray, D. Bowes, N. Davey, Y. Sun, and B. Christianson, "The misuse of the NASA metrics data program data sets for automated software defect prediction," in Proc. 15th Annu. Conf. Eval. Assessment Sofwtw. Eng., Durham, U.K., 2011, pp. 96-103.
13. M. Shepperd, Q. Song, Z. Sun, and C. Mair, "Data quality: Some comments on the nasa software defect data sets," IEEE Trans. Softw. Eng., vol. 39, no. 9, pp. 1208-1215, Sept. 2013.
14. T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, "Cross-project defect prediction: A large scale experiment on data vs. domain vs. process," in Proc. 7th Joint Meet. Eur. Softw. Eng. Conf. ACM SIGSOFT Symp. Found. Softw. Eng., 2009, pp. 91-100.
15. E. Arisholm, L. C. Briand, and E. B. Johannessen, "A systematic and comprehensive investigation of methods to build and evaluate fault prediction models," Simula, Dept. Informatics, Univ. Oslo, Oslo, Norway, Tech. Rep. TR-2008-06, 2008.
16. D. Powers, "Evaluation: From precision, recall and f-measure to roc, informedness, markedness and correlation," J. Mach. Learn. Technol., vol. 2, no. 1, pp. 37-63, 2011.
17. Y. Jiang, B. Cukic, and Y. Ma, "Techniques for evaluating fault prediction models," Empirical Softw. Eng., vol. 13, pp. 561-595, 2008.
18. J. Hernández-Orallo, P. Flach, and C. Ferri, "A unified view of performance metrics: Translating threshold choice into expected classification loss," J. Mach. Learn. Res., vol. 13, pp. 2813-2869, 2012.
19. G. Forman and M. Scholz, "Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement," ACM SIGKDD Explorations Newsletter, vol. 12, no. 1, pp 49-57, 2010.
20. B. Kitchenham and S. Charters, "Guidelines for performing systematic literature reviews in software engineering, version 2.3," Keele Univ., Staffordshire, U.K., Tech. Rep. EBSE-2007-01., 2007.
21. M. Blettner, W. Sauerbrei, B. Schlehofer, T. Scheuchenpflug, and C. Friedenreich, "Traditional reviews, meta-analyses and pooled analyses in epidemiology," Int. J. Epidemiol., vol. 28, no. 1, pp. 1-9, 1999. (Pubitemid 29086203)
22. G. V. Glass, "Primary, secondary, and meta-analysis of research," Educ. Res., vol. 5, no. 10, pp. 3-8, 1976.
23. L. Pickard, B. Kitchenham, and P. Jones, "Combining empirical results in software engineering," Inf. Softw. Technol., vol. 40, no. 14, pp. 811-821, 1998. (Pubitemid 128415104)
24. J. Miller, "Applying meta-analytical procedures to software engineering experiments," J. Syst. Softw., vol. 54, no. 1, pp. 29-39, 2000.
25. D. Cruzes and T. Dybå, "Research synthesis in software engineering: A tertiary study," Inf. Softw. Technol., vol. 53, no. 5, pp. 440-455, 2011.
26. M. Egger, M. Schneider, and G. Davey-Smith, "Meta-analysis spurious precision? Meta-analysis of observational studies," Brit. Med. J., vol. 316, pp. 140-144, 1998. (Pubitemid 28121116)
27. S. Shapiro, "Meta analysis/shmeta analysis," Amer. J. Epidemiol., vol. 140, pp. 771-778, 1994.
28. G. Glass, B. McGaw, and M. Smith, Meta-Analysis in Social Research. Beverly Hills, CA, USA: Sage, 1981.
29. K. Petersen and C. Wohlin, "Context in industrial software engineering research," in Proc. 3rd IEEE Intl. Symp. Empirical Softw. Eng. Meas., 2009, pp. 401-404.
30. T. Fawcett, "An introduction to ROC analysis," Pattern Recognit. Lett., vol. 27, no. 8, pp. 861-874, 2006.
31. M. Warrens, "On association coefficients for 2 × 2 tables and properties that do not depend on the marginal distributions," Psychometrika, vol. 73, no. 4, pp. 777-789, 2008.
32. E. Davenport and N. El-Sanhurry, "Phi/Phimax: Review and synthesis," Educ. Psychol. Meas., vol. 51, pp. 821-828, 1991.
33. D. Bowes, T. Hall, and D. Gray, "DConfusion: A technique to allow cross study performance evaluation of fault prediction studies," Autom. Softw. Eng., vol. 21, pp. 287-313, 2013.
34. S. Chidamber and C. Kemerer, "A metrics suite for object oriented design,"IEEETrans. Softw. Eng., vol. 20, no. 6, pp. 476-493, Jun.1994.
35. S. Siegel and N. Castellan, Nonparametric Statistics for the Behavioral Sciences. New York, NY, USA: Mcgraw-Hill, 1988.
36. T. Hall and D. Bowes, "The state of machine learning methodology in software fault prediction," in Proc. 11th Int. Conf. Mach. Learn. Appl., 2012, Dec. 2012, vol. 2, pp. 308-313.
37. M. Shepperd and G. Kadoda, "Comparing software prediction techniques using simulation," IEEE Trans. Softw. Eng., vol. 27, no. 11, pp. 987-998, Nov. 2001.
38. K. Krishnamoorthy, F. Lu, and T. Matthew, "A parametric bootstrap approach for ANOVA with unequal variances: Fixed and random models," Comput. Statist. Data Anal., vol. 51, no. 12, pp. 5731-5742, 2007. (Pubitemid 47125986)
39. T.-S. Lim and W.-Y. Loh, "A comparison of tests of equality of variances," Comput. Statis. Data Anal., vol. 22, no. 3, pp. 287-301, 1996. (Pubitemid 126393651)
40. J. Faraway, "Practical regression and ANOVA using R," Tech. Rep., Jul. 2002.
41. T. Levine and C. Hullett, "Eta squared, partial eta squared, and misreporting of effect size in communication research," Human Commun. Res., vol. 28, no. 4, pp. 612-625, 2002. (Pubitemid 37496224)
42. C. Pierce, R. Block, and H. Aguinis, "Cautionary note on reporting eta-squared values from multifactor ANOVA designs," Educ. Psychol. Meas., vol. 64, no. 6, pp. 916-924, 2004. (Pubitemid 39604741)
43. B. Kitchenham and E. Mendes, "Why comparative effort prediction studies may be invalid," in Proc. 5th Int. Conf. Predictor Models in Softw. Eng., 2009, p. 4.
44. M. Shepperd (2011). It doesn't matter what you do but does matter who does it! UCL, London [Online]. Available: crest.cs.ucl.ac. uk/15/slides/ MartinShepperdCOW15.pptx
45. D. Michie, D. Spiegelhalter, and C. Taylor, Eds., Machine Learning, Neural And Statistical Classification, series Ellis Horwood Series in Artificial Intelligence, Chichester, Sussex, U.K.: Ellis Horwood, 1994.
46. A.-L. Boulesteix, "Over-optimism in bioinformatics research," Bioinformatics, vol. 26, no. 3, pp. 437-439, 2010.
47. C. Mynatt, M. Doherty, and R. Tweney, "Confirmation bias in a simulated research environment: An experimental study of scientific inference," Quart. J. Exp. Psychol., vol. 29, no. 1, pp. 85-95, 1977.
48. V. Bebarta, D. Luyten, and K. Heard, "Emergency medicine animal research: Does use of randomization and blinding affect the results?" Academic Emergency Med., vol. 10, no. 6, pp. 684-687, 2003. (Pubitemid 36667073)
49. B. Kitchenham, "What's up with software metrics? - A preliminary mapping study," J. Syst. Softw., vol. 83, no. 1, pp. 37-51, Jan. 2010.
50. B. West, K. Welch, and A. Galecki, Linear Mixed Models: A Practical Guide Using Statistical Software, series Chapman Hall/CRC Press., Boca Raton, FL, USA: Ellis Horwood, 2007.
51. M. Harwell, E. Rubinstein, W. Hayes, and C. Olds, "Summarizing monte carlo results in methodological research: The one- and twofactor fixed effects ANOVA cases," J. Educational Behavioral Statist., vol. 17, no. 4, pp. 315-339, 1992.
52. C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu, "Fair and balanced?: Bias in bug-fix datasets," in Proc. 7th Joint Meeting Eur. Softw. Eng. Conf. ACM SIGSOFT Symp. Found. Softw. Eng., 2009, pp. 121-130.
53. C. Drummond, "Replicability is not reproducibility: Nor is it good science," Jun. 2009.
54. G. Calikli, A. Tosun, A. Bener, and M. Celik, "The effect of granularity level on software defect prediction," in Proc. 24th Int. Symp. Comput. Inf. Sci., Sep. 2009, pp. 531-536.
55. B. Caglayan, A. Bener, and S. Koch, "Merits of using repository metrics in defect prediction for open source projects," in Proc. ICSE Workshop Emerging Trends Free/Libre/Open Source Softw. Res. Develop., 2009, pp. 31-36.
56. O. Kutlubay, B. Turhan, and A. Bener, "A two-step model for defect density estimation," in Proc. 33rd EUROMICRO Conf. Softw. Eng. Adv. Appl., Aug. 2007, pp. 322-332.
57. B. Turhan and A. Bener, "A multivariate analysis of static code attributes for defect prediction," in Proc. 7th Int. Conf. Quality Softw., Oct. 2007, pp. 231-237.
58. A. Tosun and A. Bener, "Reducing false alarms in software defect prediction by decision threshold optimization," in Proc. 3rd Int. Symp. Empirical Softw. Eng. Meas., 2009, pp. 477-480.
59. T. Menzies, J. Greenwald, and A. Frank, "Data mining static code attributes to learn defect predictors," IEEE Trans. Softw. Eng., vol. 33, no. 1, pp. 2-13, Jan. 2007. (Pubitemid 46002165)
60. A. Kaur, P. S. Sandhu, and A. S. Bra, "Early software fault prediction using real time defect data," in Proc. 2nd Int. Conf. Mach. Vis., 2009, pp. 242-245.
61. A. Kaur and R. Malhotra, "Application of random forest in predicting fault-prone classes," in Proc. Int. Conf. Adv. Comput. Theory Eng., 2008, pp. 37-43.
62. Y. Singh, A. Kaur, and R. Malhotra, "Predicting software fault proneness model using neural network," Product-Focus. Softw. Process Improvement, vol. 5089, pp. 204-214, 2008.
63. G. Pai and J. Dugan, "Empirical analysis of software fault content and fault proneness using bayesian methods," IEEE Trans. Softw. Eng., vol. 33, no. 10, pp. 675-686, Oct. 2007. (Pubitemid 47455979)
64. C. Bird, N. Nagappan, H. Gall, B. Murphy, and P. Devanbu, "Putting it all together: Using socio-technical networks to predict failures," in Proc. IEEE 20th Int. Symp. Softw. Reliability Eng., 2009, pp. 109-119.
65. N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, and B. Murphy, "Change bursts as defect predictors," in Proc. IEEE 21st Int. Symp. Softw. Reliability Eng., 2010, pp. 309-318.
66. T. Zimmermann, R. Premraj, and A. Zeller, "Predicting defects for eclipse," in Proc. ICSE Int. Workshop Predictor Models Softw. Eng., May 2007, p. 9.
67. A. Schröter, T. Zimmermann, and A. Zeller, "Predicting component failures at design time," in Proc. ACM/IEEE Int. Symp. Empirical Softw. Eng., 2006, pp. 18-27. (Pubitemid 46634437)
68. A. Koru and H. Liu, "Building effective defect-prediction models in practice," IEEE Softw., vol. 22, no. 6, pp. 23-29, Nov./Dec. 2005. (Pubitemid 41709721)
69. G. Denaro and M. Pezzè, "An empirical evaluation of fault-proneness models," in Proc. 24th Int. Conf. Softw. Eng., 2002, pp. 241-251. (Pubitemid 35009114)
70. A. B. de Carvalho, A. Pozo, S. Vergilio, and A. Lenz, "Predicting fault proneness of classes through a multiobjective particle swarm optimization algorithm," in Proc. 20th IEEE Int. Conf. Tools Artif. Intell., 2008, vol. 2, pp. 387-394.
71. A. B. de Carvalho, A. Pozo, and S. R. Vergilio, "A symbolic fault-prediction model based on multiobjective particle swarm optimization," J. Syst. Softw., vol. 83, no. 5, pp. 868-882, 2010.
72. A. Nugroho, M. R. V. Chaudron, and E. Arisholm, "Assessing uml design metrics for predicting faultprone classes in a java system," in Proc. 7th IEEE Working Conf. Mining Softw. Repositories, 2010, pp. 21-30.
73. L. Briand, W. Melo, and J. Wust, "Assessing the applicability of fault-proneness models across object-oriented software projects," IEEE Trans. Softw. Eng., vol. 28, no. 7, pp. 706-720, Jul. 2002.
74. E. Arisholm, L. C. Briand, and M. Fuglerud, "Data mining techniques for building fault-proneness models in telecom java software," in Proc. 18th IEEE Int. Symp. Softw. Reliability, Nov. 2007, pp. 215-224.
75. E. Arisholm, L. C. Briand, and E. B. Johannessen, "A systematic and comprehensive investigation of methods to build and evaluate fault prediction models," J. Syst. Softw., vol. 83, no. 1, pp. 2-17, 2010.
76. C. Cruz and A. Erika, "Exploratory study of a UML metric for fault prediction," in Proc. 32nd ACM/IEEE Int. Conf. Softw. Eng., 2010, vol. 2, pp. 361-364.
77. L. Yi, T. M. Khoshgoftaar, and N. Seliya, "Evolutionary optimization of software quality modeling with multiple repositories," IEEE Trans. Softw. Eng., vol. 36, no. 6, pp. 852-864, Nov./Dec. 2010.
78. N. Seliya, T. Khoshgoftaar, and S. Zhong, "Analyzing software quality with limited fault-proneness defect data," in Proc. 9th IEEE Int. Symp. High-Assurance Syst. Eng., Oct. 2005, pp. 89-98.
79. T. Khoshgoftaar and N. Seliya, "Comparative assessment of software quality classification techniques: An empirical case study," Empirical Softw. Eng., vol. 9, no. 3, pp. 229-257, 2004.
80. T. Khoshgoftaar, X. Yuan, E. Allen, W. Jones, and J. Hudepohl, "Uncertain classification of fault-prone software modules," Empirical Softw. Eng., vol. 7, no. 4, pp. 297-318, 2002. (Pubitemid 35377447)
81. T. Mende and R. Koschke, "Effort-aware defect prediction models," in Proc. 14th Eur. Conf. Softw. Maintenance Reeng., 2010, pp. 107-116.
82. C. Catal, B. Diri, and B. Ozumut, "An artificial immune system approach for fault prediction in object-oriented software," in Proc. 2nd Int. Conf. Dependability Comput. Syst., Jun. 2007, pp. 238-245. (Pubitemid 350080820)
83. R. Moser, W. Pedrycz, and G. Succi, "A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction," in Proc. ACM/IEEE 30th Int. Conf. Softw. Eng., 2008, pp. 181-190.
84. T. Gyimothy, R. Ferenc, and I. Siket, "Empirical validation of object-oriented metrics on open source software for fault prediction," IEEE Trans. Softw. Eng., vol. 31, no. 10, pp. 897-910, Oct. 2005. (Pubitemid 43065752)
85. R. Vivanco, Y. Kamei, A. Monden, K. Matsumoto, and D. Jin, "Using search-based metric selection and oversampling to predict fault prone modules," in Proc. 23rd Can. Conf. Electr. Comput. Eng., 2010, pp. 1-6.
86. Y. Ma, L. Guo, and B. Cukic, "A statistical framework for the prediction of fault-proneness," in Advances in Machine Learning Application in Software Engineering, Calgary, AB, Canada: Idea Group Inc., 2006, pp. 237-265.
87. L. Guo, Y. Ma, B. Cukic, and H. Singh, "Robust prediction of fault-proneness by random forests," in Proc. 15th Int. Symp. Softw. Reliability Eng., Nov. 2004, pp. 417-428. (Pubitemid 40455712)
88. Y. Jiang, B. Cukic, and Y. Ma, "Techniques for evaluating fault prediction models," Empirical Softw. Eng., vol. 13, no. 5, pp. 561-595, 2008.
89. O. Mizuno, S. Ikami, S. Nakaichi, and T. Kikuno, "Spam filter based approach for finding fault-prone software modules," in Proc. 4th ICSE Int. Workshop Mining Softw. Repositories, May 2007, p. 4.
90. O. Mizuno and T. Kikuno, "Training on errors experiment to detect fault-prone software modules by spam filter," in Proc. 6th Joint Meeting Eur. Softw. Eng. Conf. ACM SIGSOFT Symp. Found. Softw. Eng., 2007, pp. 405-414.
91. Y. Zhou and H. Leung, "Empirical analysis of object-oriented design metrics for predicting high and low severity faults," IEEE Trans. Softw. Eng., vol. 32, no. 10, pp. 771-789, Oct. 2006. (Pubitemid 46405591)
92. O. Vandecruys, D. Martens, B. Baesens, C. Mues, M. De Backer, and R. Haesen, "Mining software repositories for comprehensible software fault prediction models," J. Syst. Softw., vol. 81, no. 5, pp. 823-839, 2008. (Pubitemid 351389560)
93. Z. Hongyu, "An investigation of the relationships between lines of code and defects," in IEEE Int. Conf. Softw. Maintenance, 2009, pp. 274-283.
94. S. Kanmani, V. Uthariaraj, V. Sankaranarayanan, and P. Thambidurai, "Object-oriented software fault prediction using neural networks," Inf. Softw. Technol., vol. 49, no. 5, pp. 483-492, 2007. (Pubitemid 46367627)
95. S. Shivaji, E. J. Whitehead, R. Akella, and K. Sunghun, "Reducing features to improve bug prediction," in Proc. 24th IEEE/ACM Int. Conf. Autom. Softw. Eng., 2009, pp. 600-604.