Robust Statistical Methods for Empirical Software Engineering

Empirical Software Engineering

Volumn 22, Issue 2, 2017, Pages 579-630

  Kitchenham, Barbara ^a   Madeyski, Lech ^b   Budgen, David ^c   Keung, Jacky ^d   Brereton, Pearl ^a   Charters, Stuart ^e   Gibbs, Shirley ^e   Pohthong, Amnart ^f  

^a KEELE UNIVERSITY   (United Kingdom)

^b WROCLAW UNIVERSITY OF TECHNOLOGY   (Poland)

^c DURHAM UNIVERSITY   (United Kingdom)

^d CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^e LINCOLN UNIVERSITY   (New Zealand)

^f PRINCE OF SONGKLA UNIVERSITY   (Thailand)

Author keywords

Empirical software engineering; Robust methods; Robust statistical methods; Statistical methods

Indexed keywords

SOFTWARE ENGINEERING; STATISTICS;

EMPIRICAL SOFTWARE ENGINEERING; NON-PARAMETRIC STATISTICAL TESTS; NONPARAMETRIC METHODS; PARAMETRIC -ANALYSIS; ROBUST METHODS; ROBUST STATISTICAL METHOD; SOFTWARE ENGINEERING DATA; SOFTWARE ENGINEERING EXPERIMENTS;

STATISTICAL METHODS;

EID: 84975113024     PISSN: 13823256     EISSN: 15737616     Source Type: Journal    
DOI: 10.1007/s10664-016-9437-5     Document Type: Article

References (70)

1. Acion L, Peterson JJ, Temple S, Arndt S (2006) Probabilistic index: an intuitive non-parametric approach to measuring the size of treatment effects. Stat Med 25(4):591–602. doi:http://dx.doi.org/10.1002/sim.2256
2. Agresti A, Pendergast J (1986) Comparing mean ranks for repeated measures data. Communications in Statistics - Theory and Methods 15(5):1417–1433
3. Akritas MG, Arnold SF (1994) Fully nonparametric hypotheses for factorial designs i: multivariate repeated measures designs. J Am Stat Assoc 89(425):336–343. doi:10.1080/01621459.1994.10476475
4. Akritas MG, Arnold SF, Brunner E (1997) Nonparametric hypotheses and rank statistics for unbalanced factorial designs. J Am Stat Assoc 92(437):258–265. doi:10.1080/01621459.1997.10473623
5. Arcuri A, Briand L (2011) A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: ACM/IEEE international conference on software engineering (ICSE), IEEE. doi:10.1145/1985793.1985795, pp 1–10
6. Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24(3):219–250. doi:10.1002/stvr.1486
7. Behrens JT (1997) Principles and procedures of exploratory data analysis. Psychol Methods 2(2):131–160
8. Bergmann R, Ludbrook J, Spooren WPJM (2000) Different outcomes of the Wilcoxon-Mann-Whitney test from different statistics packages. Am Stat 54(1):72–77
9. Boehm BW (1981) Software engineering economics. Prentice-Hall
10. Borenstein M, Hedges LV, Higgins JP, Hannah RR (2009) Introduction to meta-analysis. Wiley
11. Box GEP (1954) Some theorems on quadratic forms applied in the study of analysis of variance problems, i. Effect of inequality of variance in the One-Way classification. Ann Math Stat 25(2):290–302. doi:10.1214/aoms/1177728786
12. Braver SL, Thoemmes FJ, Rosenthal R (2014) Continuously cumulating meta-analysis and replicability. Perspect Psychol Sci 9(3):333–342. doi:10.1177/1745691614529796
13. Brunner E, Munzel U, Puri ML (2002) The multivariate nonparametric Behrens–Fisher problem. Journal of Statistical Planning and Inference 108(1–2):37–53. doi:10.1016/S0378-3758(02)00269-0
14. Budgen D, Kitchenham BA, Charters SM, Turner M, Brereton P, Linkman SG (2008) Presenting software engineering results using structured abstracts: a randomised experiment. Empir Softw Eng 13(4):435–468. doi:10.1007/s10664-008-9075-7
15. Budgen D, Burn AJ, Kitchenham B (2011) Reporting computing projects through structured abstracts: a quasi-experiment. Empir Softw Eng 16(2):244–277. doi:10.1007/s10664-010-9139-3
16. Budgen D, Kitchenham B, Charters S, Gibbs S, Pohthong A, Keung J, Brereton P (2013) Lessons from conducting a distributed quasi-experiment. In: 2013 ACM/IEEE international symposium on empirical software engineering and measurement. doi:10.1109/ESEM.2013.12, pp 143–152
17. Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494–509
18. Cohen JW (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Hillsdale, New York
19. Cohen JW (1992) A power primer. Psychol Bull 112(1):155–159
20. Conover W, Imam RL (1981) Rank transformations as a bridge between parametric and nonparametric statistics. Am Stat 35(3):124–129
21. D’Agostino RB, Belanger A, D’Agostino J, Ralph B (1990) A suggestion for using powerful and informative tests of normality. Am Stat 44(4):316–321
22. Dejaeger K, Verbeke W, Martens D, Baesens B (2012) Data mining techniques for software effort estimation: a comparative study. IEEE Trans Softw Eng 38(2):357–397
23. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30
24. Dybå T, Kampenes VB, Sjøberg DIK (2006) A systematic review of statistical power in software engineering experiments. Inf Softw Technol 48(8):745–755. doi:10.1016/j.infsof.2005.08.009
25. El-Attar M (2014) Using SMCD to reduce inconsistencies in misuse case models: a subject-based empirical evaluation. J Syst Softw 87:104–118. doi:10.1016/j.jss.2013.10.017
26. El-Attar M, Elish M, Mahmood S, Miller J (2012) Is in-depth object-oriented knowledge necessary to develop quality robustness diagrams? Journal of Software 7 (11):2538–2552. doi:10.4304/jsw.7.11.2538-2552
27. Erceg-Hurn DM, Mirosevich VM (2008) Modern robust statistical methods an easy way to maximize the accuracy and power of your research. Am Psychol 63 (7):591–601
28. Gandrud C (2015) Reproducible research with R and R studio. CRC Press
29. Goodall C (1983) Understanding robust and exploratory data analysis. John Wiley and Sons Inc., chap M-Estimators of Location: An outline of the theory, pp 339–403
30. Grissom RJ (1996) The magical number.7 ±.2: meta-meta-analysis of the probability of superior outcome in comparisons involving therapy, placebo, and control. J Consult Clin Psychol 64 (5):973–982. doi:10.1037/0022-006X.64.5.973
31. Hoaglin DC, Mosteller F, Tukey JW (eds) (1983) Understanding robust and exploratory data analysis. Wiley
32. Huijgens H, van Solingen R, van Deursen A (2013) How to build a good practice software project portfolio? Tech. Rep. TUD-SERG-2013-019, Delft University of Technology
33. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23 (5):524–532. doi:10.1177/0956797611430953
34. Jureczko M, Madeyski L (2015) Cross–project defect prediction with respect to code ownership model: an empirical study. e-Informatica Software Engineering Journal 9(1):21–35. doi:10.5277/e-Inf150102
35. Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49(11-12):1073–1086. doi:10.1016/j.infsof.2007.02.015
36. Kitchenham B (1996) Software metrics: measurement for software process improvement. Blackwell Publishers Inc.
37. Kitchenham B (2015) Robust statistical methods: why, what and how: keynote. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering (EASE 2015). doi:10.1145/2745802.2747956, pp 1:1–1:6
38. Kitchenham B, Känsälä K (1983) Inter-item correlations among function points. In: Proceedings ICSE 15. IEEE Computer Society Press, pp 477–480
39. Kraemer HC, Kupfer DJ (2006) Size of treatment effects and their importance to clinical research and practice. Biol Psychiatry 59(11):990–996. doi:10.1016/j.biopsych.2005.09.014
40. Kromrey JD, Hogarty KY, Ferron JM, Hines CV, Hess MR (2005) Robustness in meta-analysis: an empirical comparison of point and interval estimates of standardized mean differences and Cliff’s delta. In: Proceedings of the joint statistical meetings, Minneapolis
41. Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage Publications, California
42. Madeyski L (2010) Test-driven development: an empirical evaluation of agile practice. Springer, Heidelberg
43. Madeyski L (2015) Reproducer: reproduce statistical analyses and meta-analyses. http://madeyski.e-informatyka.pl/reproducible-research, R package (http://CRAN.R-project.org/package=reproducer)
44. Madeyski L, Jureczko M (2015) Which process metrics can significantly improve defect prediction models? An empirical study. Softw Qual J 23(3):393–422. doi:10.1007/s11219-014-9241-7
45. Madeyski L, Orzeszyna W, Torkar R, Józala M (2012) Appendix to the paper “Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation”. http://madeyski.e-informatyka.pl/download/app/AppendixTSE.pdf
46. Madeyski L, Orzeszyna W, Torkar R, Józala M (2014) Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans Softw Eng 40(1):23–42. doi:10.1109/TSE.2013.44
47. Micceri T (1989) The unicorn, the normal curve, and other improbable creatures. Psychol Bull 105(1):156–166
48. Mosteller F, Tukey JW (1977) Data analysis and regression: a second course in statistics. Addison-Wesley
49. Mudholkar GS, Marchetti CE, Lin CT (2002) Independence characterizations and testing normality against restricted skewness–kurtosis alternatives. Journal of Statistical Planning and Inference 104(2):485– 501
50. Price RM, Bonett DG (2001) Estimating the variance of the sample median. J Stat Comput Simul 68(3):295–305. doi:10.1080/00949650108812071
51. R Core Team (2015) R: a language and environment for statistical computing. R foundation for statistical computing, Vienna, Austria
52. Ramsey PH (1980) Exact type 1 error rates for robustness of student’s t test with unequal variances. J Educ Behav Stat 5(4):337–349. doi:10.3102/10769986005004337
53. Razali NM, Wah YB (2011) Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. Journal of Statistical Modeling and Analytics 2(1):21–33
54. Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin, Boston
55. Shapiro SS, Wilk M, Chen HJ (1968) A comparative study of various tests for normality. J Am Stat Assoc 63(324):1343–1372
56. Shrout P, Fleiss J (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86(2):420–428. doi:10.1037/0033-2909.86.2.420
57. Stout DE, Ruble TL (1995) Assessing the practical significance of empirical results in accounting education research: the use of effect size information. Journal of Accounting Education 13(3):281– 298
58. Tappenden AF, Miller J (2014) Automated cookie collection testing. ACM Trans Softw Eng Methodol 23(1):3:1–3:40. doi:10.1145/2559936
59. Tian T, Wilcox R (2007) A comparison of two rank tests for repeated measures designs. Journal of Modern Applied Statistical Methods 6(1):331–335
60. Urdan TC (2005) Statistics in plain english, 2nd edn. Routledge, Oxon, UK
61. Vargha A, Delaney HD (2000) A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J Educ Behav Stat 25(2):101–132. doi:10.3102/10769986025002101
62. Viechtbauer W (2010) Conducting meta-analyses in R with the metafor package. J Stat Softw 36(3):1–48
63. Welch BL (1938) The significance of the difference between two means when the population variances are unequal. Biometrika 29(3-4):350–362. doi:10.1093/biomet/29.3-4.350
64. Whigham PA, Owen C, MacDonell S (2015) A baseline model for software effort estimation. ACM Trans Softw Eng Methodol 24(3):20:1–20:11
65. Wilcox RR (1998) How many discoveries have been lost by ignoring modern statistical methods? Am Psychol 53(3):300–314
66. Wilcox RR (2012) Introduction to robust estimation & hypothesis testing, 3rd edn. Elsevier
67. Wilcox RR, Keselman HJ (2003) Modern robust data analysis methods: measures of central tendency. Psychol Methods 8(3):254–274
68. Yuen KK (1974) The two-sample trimmed t for unequal population variances. Biometrika 61(1):165–170
69. Zimmerman DW (2000) Statistical significance levels of nonparametric tests biased by heterogeneous variances of treatment groups. J Gen Psychol 127(4):354–364. doi:10.1080/00221300009598589
70. Zimmerman DW, Zumbo BD (1993) Rank transformations and the power of the Student t test and Welch t test for non-normal populations with unequal variances. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale 47(3):523– 539