DConfusion: A technique to allow cross study performance evaluation of fault prediction studies

Automated Software Engineering

Volumn 21, Issue 2, 2014, Pages 287-313

  Bowes, David ^a   Hall, Tracy ^b   Gray, David ^a  

^a UNIVERSITY OF HERTFORDSHIRE   (United Kingdom)

^b BRUNEL UNIVERSITY   (United Kingdom)

Author keywords

Confusion matrix; Fault; Machine learning

Indexed keywords

AUTOMATION; FAULTING; LEARNING SYSTEMS; SOFTWARE ENGINEERING;

CONFUSION MATRICES; FAULT PREDICTION; FAULT PREDICTION MODELS; PERFORMANCE MEASURE; PREDICTIVE PERFORMANCE;

IMAGE RESOLUTION;

EID: 84898870457     PISSN: 09288910     EISSN: 15737535     Source Type: Journal    
DOI: 10.1007/s10515-013-0129-8     Document Type: Article

References (43)

1. Arisholm, E., Briand, L.C., Fuglerud, M.: Data mining techniques for building fault-proneness models in telecom Java software. In: The 18th IEEE International Symposium on Software Reliability, 2007, ISSRE '07, pp. 215-224 (2007)
2. Arisholm, E., Briand, L.C., Johannessen, E.B.: A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. J. Syst. Softw. 83(1), 2-17 (2010)
3. Baldi, P., Brunak, S., Chauvin, Y., Andersen, C., Nielsen, H.: Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics 16(5), 412-424 (2000)
4. Batista, G., Prati, R., Monard, M.: A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explor. Newsl. 6(1), 20-29 (2004)
5. Bowes, D., Hall, T., Gray, D.: Comparing the performance of fault prediction models which report multiple performance measures: recomputing the confusion matrix. In: Proceedings of the 8th International Conference on Predictive Models in Software Engineering, PROMISE '12, New York, NY, USA, pp. 109-118. ACM, New York (2012)
6. Catal, C., Diri, B., Ozumut, B.: An artificial immune system approach for fault prediction in object-oriented software. In: 2nd International Conference on Dependability of Computer Systems, 2007, DepCoS-RELCOMEX '07, pp. 238-245 (2007, accepted)
7. Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16(1), 321-357 (2002)
8. Chawla, N.V., Japkowicz, N., Kotcz, A.: Editorial: special issue on learning from imbalanced data sets. ACM SIGKDD Explor. Newsl. 6(1), 1-6 (2004)
9. Cruzes, D.S., Dybå, T.: Research synthesis in software engineering: a tertiary study. Inf. Softw. Technol. 53, 440-455 (2011)
10. de Carvalho, A.B., Pozo, A., Vergilio, S., Lenz, A.: Predicting fault proneness of classes trough a multiobjective particle swarm optimization algorithm. In: 20th IEEE International Conference on Tools with Artificial Intelligence, 2008, ICTAI '08, vol. 2, pp. 387-394 (2008)
11. de Carvalho, A.B., Pozo, A., Vergilio, S.R.: A symbolic fault-prediction model based on multiobjective particle swarm optimization. J. Syst. Softw. 83(5), 868-882 (2010)
12. Denaro, G., Pezzè, M.: An empirical evaluation of fault-proneness models. In: Proceedings of the 24th International Conference on Software Engineering, ICSE '02, New York, NY, USA, pp. 241-251. ACM, New York (2002)
13. Elish, K., Elish, M.: Predicting defect-prone software modules using support vector machines. J. Syst. Softw. 81(5), 649-660 (2008)
14. Fenton, N., Neil, M.: A critique of software defect prediction models. IEEE Trans. Softw. Eng. 25(5), 675-689 (1999)
15. Forman, G., Scholz, M.: Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. ACM SIGKDD Explor. Newsl. 12(1), 49-57 (2010)
16. Gray, D., Bowes, D., Davey, N., Sun, Y., Christianson, B.: Further thoughts on precision. In: Evaluation and Assessment in Software Engineering (EASE) (2011)
17. Gray, D., Bowes, D., Davey, N., Sun, Y., Christianson, B.: Reflections on the nasa mdp data sets. IET Softw. 6(6), 549-558 (2012)
18. Hall, T., Bowes, D.: The state of machine learning methodology in software fault prediction. In: International Conference on Machine Learning and Applications (ICMLA) (2012)
19. Hall, T., Beecham, S., Bowes, D., Gray, D., Counsell, S.: A systematic literature review on fault prediction performance in software engineering. IEEE Trans. Softw. Eng. 38(6), 1276-1304 (2012)
20. He, H., Garcia, E.: Learning from imbalanced data. IEEE Trans. Knowl. Data Eng., 1263-1284 (2008)
21. Jiang, Y., Cukic, B., Ma, Y.: Techniques for evaluating fault prediction models. Empir. Softw. Eng. 13(5), 561-595 (2008)
22. Kamei, Y., Monden, A., Matsumoto, S., Kakimoto, T., Matsumoto, K.: The effects of over and under sampling on fault-prone module detection. In: First International Symposium on Empirical Software Engineering and Measurement, 2007, ESEM 2007, pp. 196-204 (2007)
23. Kaur, A., Sandhu, P.S., Bra, A.S.: Early software fault prediction using real time defect data. In: Second International Conference on Machine Vision, 2009, ICMV '09, pp. 242-245 (2009)
24. Khoshgoftaar, T., Seliya, N.: Comparative assessment of software quality classification techniques: an empirical case study. Empir. Softw. Eng. 9(3), 229-257 (2004)
25. Koru, A., Liu, H.: Building effective defect-prediction models in practice. IEEE Softw. 22(6), 23-29 (2005)
26. Kutlubay, O., Turhan, B., Bener, A.: A two-step model for defect density estimation. In: 33rd EUROMICRO Conference on Software Engineering and Advanced Applications, 2007, pp. 322-332 (2007)
27. Lessmann, S., Baesens, B., Mues, C., Pietsch, S.: Benchmarking classification models for software defect prediction: a proposed framework and novel findings. IEEE Trans. Softw. Eng. 34(4), 485-496 (2008)
28. Ma, Y., Guo, L., Cukic, B.: A statistical framework for the prediction of fault-proneness. In: Advances in Machine Learning Applications in Software Engineering, IGI Global, pp. 237-265 (2006)
29. Mende, T., Koschke, R.: Effort-aware defect prediction models. In: 14th European Conference on Software Maintenance and Reengineering (CSMR), 2010, pp. 107-116 (2010)
30. Menzies, T., Dekhtyar, A., Distefano, J., Greenwald, J.: Problems with precision: a response to "comments on 'data mining static code attributes to learn defect predictors'∈". IEEE Trans. Softw. Eng. 33(9), 637-640 (2007a)
31. Menzies, T., Greenwald, J., Frank, A.: Data mining static code attributes to learn defect predictors. IEEE Trans. Softw. Eng. 33(1), 2-13 (2007b)
32. Ostrand, T., Weyuker, E.: How to measure success of fault prediction models. In: Fourth International Workshop on Software Quality Assurance: in Conjunction with the 6th ESEC/FSE Joint Meeting, pp. 25-30. ACM, New York (2007)
33. Ostrand, T., Weyuker, E., Bell, R.: Where the bugs are. In: ACM SIGSOFT Software Engineering Notes, vol. 29, pp. 86-96. ACM, New York (2004)
34. Pai, G., Dugan, J.: Empirical analysis of software fault content and fault proneness using Bayesian methods. IEEE Trans. Softw. Eng. 33(10), 675-686 (2007)
35. Pizzi, N., Summers, A., Pedrycz, W.: Software quality prediction using median-adjusted class labels. In: Proceedings of the 2002 International Joint Conference on Neural Networks, 2002, IJCNN '02, vol. 3, pp. 2405-2409 (2002)
36. Seliya, N., Khoshgoftaar, T., Zhong, S.: Analyzing software quality with limited fault-proneness defect data. In: Ninth IEEE International Symposium on High-Assurance Systems Engineering, 2005, HASE 2005, pp. 89-98 (2005)
37. Sun, Y., Castellano, C., Robinson, M., Adams, R., Rust, A., Davey, N.: Using pre & post-processing methods to improve binding site predictions. Pattern Recognit. 42(9), 1949-1958 (2009)
38. Turhan, B., Kocak, G., Bener, A.: Data mining source code for locating software bugs: a case study in telecommunication industry. Expert Syst. Appl. 36(6), 9986-9990 (2009)
39. Wang, T., Li, W.: Naïve Bayes software defect prediction model. In: International Conference on Computational Intelligence and Software Engineering (CiSE), 2010, pp. 1-4 (2010). doi: 10.1109/CISE.2010.5677057
40. Yi, L., Khoshgoftaar, T.M., Seliya, N.: Evolutionary optimization of software quality modelling with multiple repositories. IEEE Trans. Softw. Eng. 36(6), 852-864 (2010)
41. Zeller, A., Zimmermann, T., Bird, C.: Failure is a four-letter word: a parody in empirical research. In: Proceedings of the 7th International Conference on Predictive Models in Software Engineering, Promise '11, New York, NY, USA, pp. 5:1-5:7. ACM, New York (2011)
42. Zhang, H., Zhang, X.: Comments on "data mining static code attributes to learn defect predictors". IEEE Trans. Softw. Eng. 33(9), 635-637 (2007)
43. Zhou, Y., Leung, H.: Empirical analysis of object-oriented design metrics for predicting high and low severity faults. IEEE Trans. Softw. Eng. 32(10), 771-789 (2006)