Software effort models should be assessed via leave-one-out validation

Journal of Systems and Software

Volumn 86, Issue 7, 2013, Pages 1879-1890

  Kocaguneli, Ekrem ^a   Menzies, Tim ^a  

^a WEST VIRGINIA UNIVERSITY   (United States)

Author keywords

Bias; Prediction system; Software cost estimation; Variance

Indexed keywords

BIAS; BIAS AND VARIANCE; DETERMINISTIC ALGORITHMS; PREDICTION SYSTEMS; REPRODUCIBILITIES; SOFTWARE COST ESTIMATIONS; SOFTWARE EFFORT ESTIMATION; VARIANCE;

HARDWARE; SOFTWARE ENGINEERING;

ALGORITHMS;

EID: 84879923542     PISSN: 01641212     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jss.2013.02.053     Document Type: Article

References (47)

1. A. Albrecht, and J. Gaffney Software function, source lines of code and development effort prediction: a software science validation IEEE Transactions on Software Engineeruing 9 1983 639 648
2. E. Alpaydin Techniques for combining multiple learners Proceedings of Engineering of Intelligent Systems 1998
3. E. Alpaydin Introduction to Machine Learning 2004 MIT Press
4. M. Auer, A. Trendowicz, B. Graser, E. Haunschmid, and S. Biffl Optimal project feature weights in analogy-based cost estimation: improvement and limitations IEEE Transactions on Software Engineering 32 2006 83 92
5. Baker, D.; 2007. A Hybrid Approach to Expert and Model-based Effort Estimation. Master's Thesis. Lane Department of Computer Science and Electrical Engineering, West Virginia University.
6. A. Bakir, E. Kocaguneli, A. Tosun, A. Bener, and B. Turhan Xiruxe: an intelligent fault tracking tool AIPR'09: International Conference on Artificial Intelligence and Pattern Recognition Orlando 2009
7. N. Bettenburg, M. Nagappan, and A.E. Hassan Think locally, act globally: improving defect and effort prediction models MSR 2012 60 69
8. B.W. Boehm Software Engineering Economics 1981 Prentice Hall PTR Upper Saddle River, NJ, USA
9. B. Boehm, C. Abts, and S. Chulani Software development cost estimation approaches - a survey Annals of Software Engineering 10 2000 177 205
10. L. Breiman, J. Friedman, R. Olshen, and C. Stone Classification and Regression Trees 1984 Wadsworth and Brooks Monterey, CA
11. C.L. Chang Finding prototypes for nearest neighbor classifiers IEEE Transactions on Computers 1974 1179 1185
12. J. Demsar Statistical comparisons of classifiers over multiple data sets Journal of Machine Learning Research 7 2006 1 30
13. T.G. Dietterich Ensemble methods in machine learning MCS'00: Proceedings of the First International Workshop on Multiple Classifier Systems 2000 Springer-Verlag London, UK 1 15
14. J. Ghosh Multiclassifier systems: back to the future MCS'02: Proceedings of the Third International Workshop on Multiple Classifier Systems 2002 Springer-Verlag London, UK 1 15
15. M.A. Hall, and G. Holmes Benchmarking attribute selection techniques for discrete class data mining IEEE Transactions on Knowledge and Data Engineering 15 2003 1437 1447
16. T. Hastie, R. Tibshirani, and J. Friedman The Elements of Statistical Learning 2nd ed. 2008 Springer
17. Y. Hochberg, and A.C. Tamhane Multiple Comparison Procedures 1987 John Wiley and Sons, Inc. Hoboken, NJ
18. M. Hollander, and D.A. Wolfe Nonparametric Statistical Methods 1999 John Wiley and Sons, Inc. Hoboken, NJ
19. M. Jørgensen A review of studies on expert estimation of software development effort Journal of Systems and Software 70 2004 37 60
20. M. Jørgensen, and M. Shepperd A systematic review of software development cost estimation studies IEEE Transactions on Software Engineering 33 2007 33 53
21. G. Kadoda, M. Cartwright, and M. Shepperd On configuring a case-based reasoning software project prediction system UK CBR Workshop Cambridge, UK, Citeseer 2000 1 10
22. V.B. Kampenes, T. Dybå, J.E. Hannay, and D.I.K. Sjøberg A systematic review of effect size in software engineering experiments Information & Software Technology 49 2007 1073 1086
23. C.F. Kemerer An empirical validation of software cost estimation models Communications of the ACM 30 1987 416 429
24. J. Keung Theoretical maximum prediction accuracy for analogy-based software cost estimation APSEC'08: 15th Asia-Pacific Software Engineering Conference 2008 495 502
25. J. Keung, E. Kocaguneli, and T. Menzies Finding conclusion stability for selecting the best effort predictor in software effort estimation Automated Software Engineering 2012 1 25 10.1007/s10515-012-0108-5
26. B. Kitchenham, and K. Känsälä Inter-item correlations among function points ICSE'93: Proceedings of the 15th international Conference on Software Engineering 1993 477 480
27. J. Kittler, I.C. Society, M. Hatef, R.P.W. Duin, and J. Matas On combining classifiers IEEE Transactions on Pattern Analysis and Machine Intelligence 20 1998 226 239
28. E. Kocaguneli, T. Menzies, and J. Keung On the value of ensemble effort estimation IEEE Transactions on Software Engineering 2011 1
29. E. Kocaguneli, T. Menzies, A. Bener, and J.W. Keung Exploiting the essential assumptions of analogy-based effort estimation IEEE Transactions on Software Engineering 38 2012 425 438
30. Y. Kultur, B. Turhan, and A.B. Bener ENNA: software effort estimation using ensemble of neural networks with associative memory SIGSOFT'08/FSE-16: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering 2008 ACM New York, NY, USA 330 338
31. S. Lessmann, B. Baesens, C. Mues, and S. Pietsch Benchmarking classification models for software defect prediction: a proposed framework and novel findings IEEE Transactions on Software Engineering 34 2008 485 496
32. Y. Li, M. Xie, and T. Goh A study of project selection and feature weighting for analogy based software cost estimation Journal of Systems and Software 82 2009 241 252
33. K. Lum, T. Menzies, and D. Baker 2CEE, a twenty first century effort estimation methodology ISPA/SCEA'08: Joint Conference of International Society of Parametric Analysts and Society of Cost Estimating and Analysis 2008 12 14
34. K.D. Maxwell Applied Statistics for Software Managers 2002 Prentice Hall, PTR
35. E. Mendes, I. Watson, C. Triggs, N. Mosley, and S. Counsell A comparative study of cost estimation models for web hypermedia applications Empirical Software Engineering 8 2003 163 196
36. T. Menzies, and M. Shepperd Special issue on repeatable results in software engineering prediction Empirical Software Engineering 17 2012 1 17
37. T. Menzies, Z. Chen, J. Hihn, and K. Lum Selecting best practices for effort estimation IEEE Transactions on Software Engineering 32 2006 883 895
38. T. Menzies, A. Butcher, A. Marcus, T. Zimmermann, and D.R. Cok Local vs. global models for effort estimation and defect prediction ASE 2011 343 351
39. Y. Miyazaki, M. Terakado, K. Ozaki, and H. Nozaki Robust regression for developing software estimation models Journal of Systems and Software 27 1994 3 16
40. A.M. Molinaro, R. Simon, and R.M. Pfeiffer Prediction error estimation: a comparison of resampling methods Bioinformatics (Oxford, England) 21 2005 3301 3307
41. I. Myrtveit, E. Stensrud, and M. Shepperd Reliability and validity in comparative studies of software prediction models IEEE Transactions on Software Engineering 31 2005 380 391
42. G. Seni, and J. Elder Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions 2010 Morgan and Claypool Publishers
43. M. Shepperd, and G.F. Kadoda Comparing software prediction techniques using simulation IEEE Transactions on Software Engineering 27 2001 1014 1022
44. M. Shepperd, and C. Schofield Estimating Software Project Effort Using Analogies IEEE Transactions on Software Engineering 23 1997 736 743
45. M. Shepperd, C. Schofield, and B. Kitchenham Effort estimation using analogy ICSE'96: Proceedings of the 18th International Conference on Software Engineering 1996 IEEE Computer Society Washington, DC, USA 170 178
46. R. Valerdi Heuristics for systems engineering cost estimation IEEE Systems Journal 5 2011 91 98
47. A.R. Venkatachalam Software cost estimation using artificial neural networks Proceedings of International Joint Conference on Neural Networks 1993 987 990