A Simulation Study of the Model Evaluation Criterion MMRE

IEEE Transactions on Software Engineering

Volumn 29, Issue 11, 2003, Pages 985-995

  Foss, Tron ^a   Stensrud, Erik ^b   Kitchenham, Barbara ^c   Myrtveit, Ingunn ^a  

^a BI NORWEGIAN BUSINESS SCHOOL   (Norway)

^b Myrtveit og Stensrud ANS   (Norway)

^c KEELE UNIVERSITY   (United Kingdom)

Author keywords

Empirical software engineering; Mean magnitude of relative error; Prediction accuracy; Prediction models; Regression analysis; Simulation; Software cost estimation; Software engineering; Software metrics

Indexed keywords

COMPUTER SIMULATION; COST ACCOUNTING; DATA REDUCTION; ERROR STATISTICS; LEAST SQUARES APPROXIMATIONS; LINEAR SYSTEMS; MATHEMATICAL MODELS; MATHEMATICAL TRANSFORMATIONS; REGRESSION ANALYSIS;

MEAN MAGNITUDE OF RELATIVE ERROR (MMRE); PREDICTION ACCURACY; SOFTWARE METRICS;

SOFTWARE ENGINEERING;

EID: 0346896359     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2003.1245300     Document Type: Article

References (52)

1. L. Angelis and I. Stamelos, "A Simulation Tool for Efficient Analogy Based Cost Estimation," Empirical Software Eng., vol. 5, pp. 35-68, 2000.
2. L.C. Briand, V.R. Basili, and W.M. Thomas, "A Pattern Recognition Approach for Software Engineering Data Analysis," IEEE Trans. Software Eng., vol. 18, no. 11, pp. 931-942, Nov. 1992.
3. L.C. Briand, V.R. Basili, and C.J. Hetmanski, "Developing Interpretable Models with Optimized Set Reduction for Identifying High-Risk Software Components," IEEE Trans. Software Eng., vol. 19, no. 11, pp. 1028-1044, Nov. 1993.
4. L.C. Briand, K. El-Emam, and F. Bomarius, "COBRA: A Hybrid Method for Software Cost Estimation, Benchmarking and Risk Assessment," Proc. 20th Int'l Conf. Software Eng., pp. 390-399, 1998.
5. L.C. Briand, K. El-Emam, and I. Wieczorek, "A Case Study in Productivity Benchmarking: Methods and Lessons Learned," Proc. Ninth European Software Control and Metrics Conf., pp. 4-14, 1998.
6. L.C. Briand, K. El-Emam, and I. Wieczorek, "Explaining the Cost of European Space and Military Projects," Proc. 21st Int'l Conf. Software Eng., pp. 303-312, 1999.
7. L.C. Briand, K. El-Emam, K. Maxwell, D. Surmann, and I. Wieczorek, "An Assessment and Comparison of Common Cost Software Project Estimation Methods," Proc. 21st Int'l Conf. Software Eng., pp. 313-322, 1999.
8. L.C. Briand, T. Langley, and I. Wieczorek, "A Replicated Assessment and Comparison of Common Software Cost Modeling Techniques," Proc. Int'l Conf. Software Eng., pp. 377-386, 2000.
9. L.C. Briand and D. Pfahl, "Using Simulation for Assessing the Real Impact of Test Coverage on Defect Coverage," IEEE Trans. Reliability, vol. 49, no. 1, pp. 60-70, 2000.
10. L.C. Briand and I. Wieczorek, "Resource Modeling in Software Engineering," Encyclopedia of Software Eng., second ed., J. Marciniak, ed., Wiley, in press.
11. A.F. Chalmers, What is This Thing Called Science? second ed. Buckingham: Open Univ. Press, 1982.
12. The COCOMO II Suite, http://sunset.usc.edu/research/cocomosuite/index.html, 2002.
13. S.D. Conte, H.E. Dunsmore, and V.Y. Shen, Software Engineering Metrics and Models. Menlo Park, Calif.: Benjamin/Cummings, 1986.
14. A.M.E. Cuelenare, M.J.I. van Genuchten, and F.J. Heemstra, "Calibrating a Software Cost Estimating Model: Why and How," Information and Software Technology, vol. 29, no. 10, pp. 558-569, 1987.
15. J.M. Desharnais, "Analyse Statistique de la Productivite des Projects de Developpement en Informatique a Partir de la Technique des Points de Fonction," Master's thesis, Univ. du Quebec a Montreal, 1989.
16. J.J. Dolado, "On the Problem of the Software Cost Function," Information Software Technology, vol. 43, no. 1, pp. 61-72, 2001.
17. Encyclopedia of Statistical Sciences, S. Kotz, N.L. Johnson, and C.B. Read, eds., New York: Wiley, 1982-1998.
18. D.V. Ferens, "Software Cost Estimation in the DoD Environment, " Am. Programmer, pp. 28-34, July 1996.
19. G.R. Finnie, G.E. Wittig, and J.M. Desharnais, "A Comparison of Software Effort Estimation Techniques: Using Function Points with Neural Networks, Case-Based Reasoning and Regression Models," J. Systems and Software, vol. 39, no. 3, pp. 281-289, 1997.
20. T. Foss, I. Myrtveit, and E. Stensrud, "MRE and Heteroscedasticity," Proc. 12th European Software Control and Metrics Conf., pp. 157-164, 2001.
21. T. Foss, I. Myrtveit, and E. Stensrud, "A Comparison of LAD and OLS Regression for Effort Prediction of Software Projects," Proc. 12th European Software Control and Metrics Conf., pp. 9-15, 2001.
22. K. El-Emam, "The Predictive Validity Criterion for Evaluating Binary Classifiers," Proc. Second Int'l Software Metrics Symp., pp. 235-244, 1998.
23. T. Foss, I. Myrtveit, and E. Stensrud, "A Comparison of LAD and OLS Regression for Effort Prediction of Software Projects," Proc. 12th European Software Control and Metrics Conf., pp. 9-15, 2001.
24. A.R. Gray and S.G. MacDonell, "Software Metrics Data Analysis - Exploring the Relative Performance of Some Commonly Used Modeling Techniques," Empirical Software Eng., vol. 4, pp. 297-316, 1999.
25. D.N. Gujarati, Basic Econometrics, third ed. New York: McGraw-Hill, 1995.
26. R. Jeffery, M. Ruhe, and I. Wieczorek, "Using Public Domain Metrics to Estimate Software Development Effort," Proc. Fifth Int'l Software Metrics Symp., pp. 16-27, 2001.
27. R. Jeffery and F. Walkerden, "Analogy, Regression and Other Methods for Estimating Effort and Software Quality Attributes," Proc. European Conf. Optimizing Software Development and Maintenance, pp. 37-46, 1999.
28. R.L. Jenson and J.W. Bartley, "Parametric Estimation of Programming Effort: An Object-Oriented Model," J. Systems and Software, vol. 15, pp. 107-114, 1991.
29. M. Jørgensen, "Experience With the Accuracy of Software Maintenance Task Effort Prediction Models," IEEE Trans. Software Eng., vol. 21, no. 8, pp. 674-681, 1995.
30. C.F. Kemerer, "An Empirical Validation of Cost Estimation Models," Comm. ACM, vol. 30, no. 5, pp. 416-429, 1987.
31. B.A. Kitchenham, S.G. MacDonell, L.M. Pickard, and M.J. Shepperd, "What Accuracy Statistics Really Measure," IEE Proc. Software, vol. 148, no. 3, pp. 81-85, 2001.
32. B.A. Kitchenham and K. Kansala, "Inter-Item Correlations Among Function Points," Proc. First METRICS Conf., pp. 11-14, 1993.
33. B.A. Kitchenham, "A Procedure for Analyzing Unbalanced Datasets," IEEE Trans. Software Eng., vol. 24, no. 4, pp. 278-301, Apr. 1998.
34. A. Koutsoyiannis, Theory of Econometrics, second ed. London: MacMillan, 1977.
35. S. Makridakis, S.C. Wheelwright, and R.J. Hyndman, Forecasting Methods and Applications, third ed. John Wiley & Sons Inc., 1998.
36. R. Martin, "Evaluation of Current Software Costing Tools," ACM SICSOFT Software Eng. Notes, vol. 13, no. 3, pp. 49-51, 1988.
37. Minitab Statistical Software Release 13, www.minitab.com, 2000.
38. Y. Miyazaki, M. Terakado, K. Ozaki, and H. Nozaki, "Robust Regression for Developing Software Estimation Models," J. Systems and Software, vol. 27, pp. 3-16, 1994.
39. T. Mukhopadhyay, S.S. Vicinanza, and M.J. Prietula, "Examining the Feasibility of a Case-Based Reasoning Model for Software Effort Estimation," MIS Quarterly, pp. 155-171, June 1992.
40. I. Myrtveit and E. Stensrud, "A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 510-525, 1999.
41. I. Myrtveit and E. Stensrud, "Benchmarking COTS Projects Using Data Envelopment Analysis," Proc. Sixth Int'l Software Metrics Symp., pp. 269-278, 1999.
42. P. Nesi and T. Querci, "Effort Estimation and Prediction for Object-Oriented Systems," J. Systems and Software, vol. 42, pp. 89-102, 1998.
43. L. Pickard, B. Kitchenham, and S.J. Linkman, "Using Simulated Data Sets to Compare Data Analysis Techniques Used for Software Cost Modelling, " IEE Proc. Software, vol. 148, no. 6, pp. 165-174, Dec. 2001.
44. J. Rosenberg, "Some Misconceptions about Lines of Code," Proc. First Int'l Software Metrics Symp., pp. 137-142, 1997.
45. B. Samson, D. Ellison, and P. Dugard, "Software Cost Estimation Using and Albus Perceptron (CMAC)," Information and Software Technology, vol. 39, pp. 55-60, 1997.
46. M. Shepperd and G. Kadoda, "Using Simulation to Evaluate Predition Techniques," Proc. Proc. Fifth Int'l Software Metrics Symp., pp. 349-359, 2001.
47. M. Shepperd and C. Schofield, "Estimating Software Project Effort Using Analogies," IEEE Trans. Software Eng., vol. 23, no. 12, pp. 736-743, Dec. 1997.
48. R. Srinivasan and D. Fisher, "Machine Learning Approaches to Estimating Software Development Effort," IEEE Trans. Software Eng., vol. 21, no. 2, pp. 126-137, Feb. 1995.
49. E. Stensrud and I. Myrtveit, "Human Performance Estimating with Analogy and Regression Models: An Empirical Validation," Proc. Second Int'l Software Metrics Symp., pp. 205-213, 1998.
50. E. Stensrud, T. Foss, B. Kitchenham, and I. Myrtveit, "A Further Empirical Investigation of the Relationship between MRE and Project Size," Empirical Software Eng., 2002.
51. K. Strike, K. El-Emam, and N. Madhavji, "Software Cost Estimation with Incomplete Data," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 890-908, Oct. 2001.
52. F. Walkerden and R. Jeffery, "An Empirical Study of Analogy-based Software Effort Estimation," Empirical Software Eng., vol. 4, no. 2, pp. 135-158, 1999.