A permutation test based on regression error characteristic curves for software cost estimation models

Empirical Software Engineering

Volumn 17, Issue 1-2, 2012, Pages 34-61

  Mittas, Nikolaos ^a   Angelis, Lefteris ^a  

^a ARISTOTLE UNIVERSITY OF THESSALONIKI   (Greece)

Author keywords

Accuracy measures; Cost prediction models; Graphical comparison; Hypothesis test; Permutation test; REC curves

Indexed keywords

ACCURACY MEASURES; COST PREDICTION MODELS; GRAPHICAL COMPARISON; HYPOTHESIS TESTS; PERMUTATION TESTS; REC CURVES;

COST ESTIMATING; FORECASTING; MATHEMATICAL MODELS; REGRESSION ANALYSIS; SOFTWARE ENGINEERING; SOFTWARE TESTING; STATISTICAL TESTS; VISUALIZATION;

COST BENEFIT ANALYSIS;

EID: 84857374517     PISSN: 13823256     EISSN: 15737616     Source Type: Journal    
DOI: 10.1007/s10664-011-9177-5     Document Type: Article

References (38)

1. Bi J, Bennet KP (2003) Regression Error Characteristics Curves. Proceedings of the AIII 20th International Conference on Machine Learning (ICML'03), August, 43-50
2. Briand L, Langley T, Wieczorek I (2000) A replicated assessment and comparison of common software cost modeling techniques. Proceedings of the IEEE International Conference Software Engineering (ICSE 22), 377-386
3. J Demsar 2006 Statistical comparisons of classifiers over multiple data sets J Mach Learn Res 7 1 30 2274360 1222.68184 (Pubitemid 43022939)
4. Early Estimate Checker V5.0 (2007) http://www.isbsg.org/
5. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall.
6. Egan JP (1975) Signal detection theory and ROC analysis. Series in Cognition and Perception, Academic Press.
7. T Fawcett 2006 An introduction to ROC analysis Pattern Recogn Lett 27 861 874 10.1016/j.patrec.2005.10.010
8. T Foss E Stensrud B Kitchenham I Myrtveit 2003 A simulation study of the model evaluation criterion MMRE IEEE Trans Software Eng 29 11 985 995 10.1109/TSE.2003.1245300
9. Good IG (2000) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Series in Statistics 2nd edition
10. Green DM, Swets JM (1966) Signal detection theory and psychophysics. Wiley, New York
11. ISBSG dataset 10 (2007) http://www.isbsg.org
12. Jeffery R, Ruhe M, Wieczorek I (2001) Using public domain metrics to estimate software development effort. In: Proceedings of the IEEE 7th International Software Metrics Symposium (METRICS 2001), April, 16-27 (Pubitemid 32470993)
13. M Jorgensen M Shepperd 2007 A systematic review of software development cost estimation studies IEEE Trans Software Eng 33 1 33 53 10.1109/TSE.2007. 256943 (Pubitemid 46002167)
14. Kitchenham B, Mendes T (2004) A comparison of cross-company and within-company effort estimation models for web applications. In: Proceedings of the Empirical Assessment in Software Engineering (EASE), 47-55.
15. Kitchenham B, Mendes T (2009) Why comparative effort prediction studies may be invalid. In: Proceedings of the ACM 5th International Conference on Predictor Models in Software Engineering, May
16. B Kitchenham SL Pfleeger B McColl S Eagan 2002 An empirical study of maintenance and development accuracy J. Syst Softw 64 57 77 10.1016/S0164- 1212(02)00021-3
17. B Kitchenham L Pickard S MacDonell M Shepperd 2001 What accuracy statistics really measure IEE Proc Software 148 3 81 85 10.1049/ip-sen:20010506 (Pubitemid 34172674)
18. Korte M, Port D (2008) Confidence in software cost estimation results based on mmre and pred. In Proceedings of the 4th ACM International Workshop on Predictor Models in Software Engineering, 63-70
19. Liebchen G, Shepperd M (2008) Data sets and data quality in software engineering. Proceedings of the 4th ACM International Workshop on Predictor Models in Software Engineering, 39-44
20. LB Lusted 1978 General problems in medical decision making with comments on ROC analysis Semin Nucl Med 8 4 299 306 10.1016/S0001-2998(78)80015-4 (Pubitemid 9075027)
21. NA Obuchowski 2003 Receiver operating characteristic curves and their use in radiology Radiology 229 1 3 8 10.1148/radiol.2291010898 (Pubitemid 37152094)
22. Port D, Korte M (2008) Comparative studies of the model evaluation criterions mmre and pred in software cost estimation research. In Proceedings of the ACM-IEEE 2nd International Symposium on Empirical Software Engineering and Management, 51-60
23. Mair C, Shepperd M (2005) The consistency of empirical comparisons of regression and analogy-based software project cost prediction. In: Proceedings of the International Symposium on Empirical Software Engineering (ISESE'05), November, 509-518 (Pubitemid 44462635)
24. E Mendes S Di Martino F Ferrucci C Gravino 2008 Cross-company vs. single-company web effort models using the Tukutuku database: an extended study J. Syst Softw 81 5 673 690 10.1016/j.jss.2007.07.044 (Pubitemid 351389564)
25. CE Metz 1978 Basic principles of ROC analysis Semin Nucl Med 8 4 283 298 10.1016/S0001-2998(78)80014-2 (Pubitemid 9075026)
26. N Mittas L Angelis 2008 Comparing cost prediction models by resampling techniques J. Syst Softw 81 5 616 632 10.1016/j.jss.2007.07.039 (Pubitemid 351389561)
27. Mittas N, Angelis L, (2008b) Comparing software cost prediction models by a visualization tool. In Proceedings of the IEEE 34th Euromicro Conference on Software Engineering and Advanced Applications (SEAA'08), 433-440
28. N Mittas L Angelis 2010 Visual comparison of software cost estimation models by regression error characteristic analysis J. Syst Softw 83 621 637 10.1016/j.jss.2009.10.044
29. Mittas N, Angelis L (2010b) LSEbA: least squares regression and estimation by analogy in a semi-parametric model for software cost estimation.
30. N Mittas M Athanasiades L Angelis 2008 Improving analogy-based software cost estimation by a resampling method Inform Softw Technol 50 3 221 230 10.1016/j.infsof.2007.04.001 (Pubitemid 350266325)
31. Y Miyazaki M Terakado K Ozaki H Nozaki 1994 Robust regression for developing software estimation models J Syst Softw 27 3 16 10.1016/0164-1212(94) 90110-4
32. Moore DS, McCabe GP, Duckworth WD II, Sclove SL (2003) The practice of business statistics. Freeman, New York
33. I Myrtveit E Stensrud M Shepperd 2005 Reliability and validity in comparative studies of software prediction models IEEE Trans Software Eng 31 5 380 391 10.1109/TSE.2005.58 (Pubitemid 41046926)
34. NASA93 dataset (2007) http://promisedata.org/
35. Sheskin DJ (2004) Handbook of parametric and nonparametric statistical procedures, third ed. Chapman & Hall/CRC
36. Z Šidàk 1967 Rectangular confidence region for the means of multivariate normal distributions J Am Stat Assoc 62 626 633 0158.17705 10.2307/2283989
37. Stensrud E, Myrveit I (1998) Human performance estimating with analogy and regression models: An empirical validation. In: Proceedings of the IEEE 5th International Software Metrics Symposium (Metrics'98), CA, 205-213
38. MH Zweig G Campbell 1993 Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine Clin Chem 39 8 561 577 (Pubitemid 23118304)