Comparing the performance of fault prediction models which report multiple performance measures: Recomputing the confusion matrix

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 109-118

  Bowes, David ^a   Hall, Tracy ^b   Gray, David ^a  

^a UNIVERSITY OF HERTFORDSHIRE   (United Kingdom)

^b BRUNEL UNIVERSITY   (United Kingdom)

Author keywords

Confusion matrix; Fault; Machine learning

Indexed keywords

CONFUSION MATRICES; FAULT; FAULT PREDICTION; PERFORMANCE MEASURE; PREDICTIVE PERFORMANCE; RE-COMPUTING;

IMAGE RESOLUTION; LEARNING SYSTEMS; MATHEMATICAL MODELS;

SOFTWARE ENGINEERING;

EID: 84867733834     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2365324.2365338     Document Type: Conference Paper

References (36)

1. E. Arisholm, L. C. Briand, and M. Fuglerud. Data mining techniques for building fault-proneness models in telecom java software. In Software Reliability, 2007. ISSRE '07. The 18th IEEE International Symposium on, pages 215 -224, nov. 2007.
2. E. Arisholm, L. C. Briand, and E. B. Johannessen. A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. Journal of Systems and Software, 83(1):2-17, 2010.
3. P. Baldi, S. Brunak, Y. Chauvin, C. Andersen, and H. Nielsen. Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics, 16(5):412-424, 2000.
4. G. Batista, R. Prati, and M. Monard. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explorations Newsletter, 6(1):20-29, 2004.
5. D. Bowes and D. Gray. Recomputing the confusion matrix for prediction studies reporting categorical output. Technical Report 509, University of Hertfordshire, 2011.
6. C. Catal, B. Diri, and B. Ozumut. An artificial immune system approach for fault prediction in object-oriented software. In Dependability of Computer Systems, 2007. DepCoS-RELCOMEX '07. 2nd International Conference on, pages 238 -245, june 2007.
7. N. V. Chawla, N. Japkowicz, and A. Kotcz. Editorial: special issue on learning from imbalanced data sets. SIGKDD Explorations, 6(1):1-6, 2004.
8. D. S. Cruzes and T. Dybå. Research synthesis in software engineering: A tertiary study. Inf. Softw. Technol., 53:440-455, May 2011.
9. A. B. de Carvalho, A. Pozo, S. Vergilio, and A. Lenz. Predicting fault proneness of classes trough a multiobjective particle swarm optimization algorithm. In Tools with Artificial Intelligence, 2008. ICTAI '08. 20th IEEE International Conference on, volume 2, pages 387-394, 2008.
10. A. B. de Carvalho, A. Pozo, and S. R. Vergilio. A symbolic fault-prediction model based on multiobjective particle swarm optimization. Journal of Systems and Software, 83(5):868-882, 2010.
11. G. Denaro and M. Pezzè. An empirical evaluation of fault-proneness models. In Proceedings of the 24th International Conference on Software Engineering, ICSE '02, pages 241-251, New York, NY, USA, 2002. ACM.
12. K. O. Elish and M. O. Elish. Predicting defect-prone software modules using support vector machines. Journal of Systems and Software, 81(5):649 - 660, 2008.
13. D. Gray, D. Bowes, N. Davey, Y. Sun, and B. Christianson. Further thoughts on precision. In Evaluation and Assessment in Software Engineering (EASE), 2011.
14. T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell. A systematic review of fault prediction performance in software engineering. Software Engineering, IEEE Transactions on, PP(99):1, 2011.
15. H. He and E. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, pages 1263-1284, 2008.
16. Y. Jiang, B. Cukic, and Y. Ma. Techniques for evaluating fault prediction models. Empirical Software Engineering, 13(5):561-595, 2008.
17. Y. Kamei, A. Monden, S. Matsumoto, T. Kakimoto, and K. Matsumoto. The effects of over and under sampling on fault-prone module detection. In Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on, pages 196 -204, sept. 2007.
18. A. Kaur, P. S. Sandhu, and A. S. Bra. Early software fault prediction using real time defect data. In Machine Vision, 2009. ICMV '09. Second International Conference on, pages 242-245. accept, 2009.
19. T. Khoshgoftaar and N. Seliya. Comparative assessment of software quality classification techniques: An empirical case study. Empirical Software Engineering, 9(3):229-257, 2004.
20. A. Koru and H. Liu. Building effective defect-prediction models in practice. Software, IEEE, 22(6):23 - 29, nov.-dec. 2005.
21. O. Kutlubay, B. Turhan, and A. Bener. A two-step model for defect density estimation. In Software Engineering and Advanced Applications, 2007. 33rd EUROMICRO Conference on, pages 322 -332, aug. 2007.
22. S. Lessmann, B. Baesens, C. Mues, and S. Pietsch. Benchmarking classification models for software defect prediction: A proposed framework and novel findings. Software Engineering, IEEE Transactions on, 34(4):485 -496, july-aug. 2008.
23. T. Mende and R. Koschke. Effort-aware defect prediction models. In Software Maintenance and Reengineering (CSMR), 2010 14th European Conference on, pages 107-116, 2010.
24. T. Menzies, A. Dekhtyar, J. Distefano, and J. Greenwald. Problems with precision: A response to "comments on 'data mining static code attributes to learn defect predictors'". Software Engineering, IEEE Transactions on, 33(9):637 -640, sept. 2007.
25. T. Menzies, J. Greenwald, and A. Frank. Data mining static code attributes to learn defect predictors. Software Engineering, IEEE Transactions on, 33(1):2 -13, jan. 2007.
26. T. Ostrand and E. Weyuker. How to measure success of fault prediction models. In Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meeting, pages 25-30. ACM, 2007.
27. G. Pai and J. Dugan. Empirical analysis of software fault content and fault proneness using bayesian methods. Software Engineering, IEEE Transactions on, 33(10):675 -686, oct. 2007.
28. N. Pizzi, A. Summers, and W. Pedrycz. Software quality prediction using median-adjusted class labels. In Neural Networks, 2002. IJCNN '02. Proceedings of the 2002 International Joint Conference on, volume 3, pages 2405 -2409, 2002.
29. N. Seliya, T. Khoshgoftaar, and S. Zhong. Analyzing software quality with limited fault-proneness defect data. In High-Assurance Systems Engineering, 2005. HASE 2005. Ninth IEEE International Symposium on, pages 89 -98, oct. 2005.
30. Y. Sun, C. Castellano, M. Robinson, R. Adams, A. Rust, and N. Davey. Using pre & post-processing methods to improve binding site predictions. Pattern Recognition, 42(9):1949-1958, 2009.
31. B. Turhan, G. Kocak, and A. Bener. Data mining source code for locating software bugs: A case study in telecommunication industry. Expert Systems with Applications, 36(6):9986-9990, 2009.
32. T. Wang and W.-h. Li. Naive bayes software defect prediction model. In Computational Intelligence and Software Engineering (CiSE), 2010 International Conference on, pages 1-4. accept, 2010.
33. L. Yi, T. M. Khoshgoftaar, and N. Seliya. Evolutionary optimization of software quality modeling with multiple repositories. Software Engineering, IEEE Transactions on, 36(6):852-864, 2010.
34. A. Zeller, T. Zimmermann, and C. Bird. Failure is a four-letter word: a parody in empirical research. In Proceedings of the 7th International Conference on Predictive Models in Software Engineering, Promise '11, pages 5:1-5:7, New York, NY, USA, 2011. ACM.
35. H. Zhang and X. Zhang. Comments on "data mining static code attributes to learn defect predictors". Software Engineering, IEEE Transactions on, 33(9):635 -637, sept. 2007.
36. Y. Zhou and H. Leung. Empirical analysis of object-oriented design metrics for predicting high and low severity faults. Software Engineering, IEEE Transactions on, 32(10):771 -789, oct. 2006.