Incorrect results in software engineering experiments: How to improve research practices

Journal of Systems and Software

Volumn 116, Issue , 2016, Pages 133-145

  Jørgensen, Magne ^a,b   Dybå, Tore ^b,c   Liestøl, Knut ^b   Sjøberg, Dag I K ^b  

^a SIMULA RESEARCH LABORATORY   (Norway)

^b UNIVERSITY OF OSLO   (Norway)

^c SINTEF   (Norway)

Author keywords

Controlled experiments; Empirical software engineering; Statistical hypothesis testing

Indexed keywords

PUBLISHING; SOFTWARE ENGINEERING; SURVEYS; TESTING;

CONTROLLED EXPERIMENT; EMPIRICAL SOFTWARE ENGINEERING; PUBLICATION BIAS; RESEARCH RESULTS; SIGNIFICANCE LEVELS; SOFTWARE ENGINEERING EXPERIMENTS; STATISTICAL HYPOTHESIS TESTING; STATISTICAL POWER;

SOFTWARE TESTING;

EID: 84927142981     PISSN: 01641212     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jss.2015.03.065     Document Type: Article

References (56)

1. Babbage C. Reflections on the Decline of Science in England, and on Some of Its Causes 1830 B. Fellowes, London
2. Bakker M., van Dijk A., Wicherts J.M. The rules of the game called psychological science Perspect. Psychol. Sci. 7 6 2012 543 554
3. Bofetta P. et al. False-positive results in cancer epidemiology: a plea for epistemological modesty J. Nat. Canc. Inst. 100 14 2008 988 995
4. Button K.S. et al. Power failure: why small sample size undermines the reliability of neuroscience Nat. Rev. Neurosci. 14 5 2013 365 376
5. Ciolkowski M. What do we know about perspective-based reading? an approach for quantitative aggregation in software engineering Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement 2009 IEEE Computer Society
6. Cohen J. Things I have learned (so far) Am. Psychol. 45 12 1990 1304
7. Cohen J. A power primer Psychol. Bull. 112 1 1992 155
8. Cumming G. The new statistics why and how Psychol. Sci. 25 1 2014 7 29
9. Dahl F.A. et al. Data splitting as a countermeasure against hypothesis fishing: With a case study of predictors for low back pain Europ. J. Epidemiol. 23 4 2008 237 242
10. Dingell J.D. Misconduct in medical research N. Eng. J. Med. 328 22 1993 1610 1615
11. Dybå T., Kampenes V.B., Sjøberg D.I. A systematic review of statistical power in software engineering experiments Inf. Softw. Technol. 48 8 2006 745 755
12. Dybå T., Kitchenham B., Jørgensen M. Evidence-based software engineering for practitioners IEEE Softw. 22 1 2005 58 65
13. Fanelli D. How many scientists fabricate and falsify research? a systematic review and meta-analysis of survey data PloS One 4 5 2009 e5738
14. Fanelli D. "Positive" results increase down the hierarchy of the sciences PloS One 5 4 2010 e10068
15. Fanelli D. Do pressures to publish increase scientists' bias? an empirical support from US States Data PloS One 5 4 2010 e10271
16. Fanelli D. Negative results are disappearing from most disciplines and countries Scientometrics 90 3 2012 891 904
17. Farthing M.J.G. Research misconduct: A grand global challenge for the 21st century J. Gastroen. Hepatol. 29 3 2014 422 427
18. Francis G. Too good to be true: publication bias in two prominent studies from experimental psychology Psychon. Bull. Rev. 19 2 2012 151 156
19. Goodman S., Greenland S. Why most published research findings are false: problems in the analysis PLoS Med. 4 4 2007 773
20. Hannay J.E. et al. Effects of personality on pair programming IEEE Trans. Softw. Eng. 36 1 2010 61 80
21. Hedges L.V. Estimation of effect size under nonrandom sampling: the effects of censoring studies yielding statistically insignificant mean differences J. Educ. Behav. Stat. 9 1 1984 61 85
22. Ingre M. Why small low-powered studies are worse than large high-powered studies and how to protect against trivial findings in research: comment on Friston (2012) Neuroimage 81 2013 496 498
23. Ioannidis J., Hozo I., Djulbegovic B. Optimal type I and type II error pairs when the available sample size is fixed J. Clin. Epidemiol. 66 8 2013 903 910
24. Ioannidis J.P.A. Why most published research findings are false PLoS Med. 2 8 2005 e124
25. Ioannidis J.P.A. Why most discovered true associations are inflated Epidemiology 19 5 2008 640 648
26. Ioannidis J.P.A., Trikalinos T.A. An exploratory test for an excess of significant findings Clin. Trials 4 3 2007 245 253
27. John L.K., Loewenstein G., Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling Psychol. Sci. 23 5 2012 524 532
28. Juristo N., Gomez O.S. Replication of software engineering experiments Empirical Software Engineering and Verification 2012 Springer 60 88
29. Juristo N., Vegas S. The role of non-exact replications in software engineering experiments Empir. Softw. Eng. 16 3 2011 295 324
30. Kampenes V.B. et al. A systematic review of effect size in software engineering experiments Inf. Softw. Technol. 49 11 2007 1073 1086
31. Kass R.E., Raftery A.E. Bayes factors J. Am. Stat. Assoc. 90 430 1995 773 795
32. Kattenbraker M. Health Education Research and Publication: Ethical Considerations and the Response of Health Educators [Thesis] 2007 School of Graduate, Southern Illinois University Carbondale Carbondale, IL 4
33. Kepes S., McDaniel M.A. How trustworthy is the scientific literature in industrial and organizational psychology? Ind. Organ. Psychol. 6 3 2013 252 268
34. Kincaid J.P. et al. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel DTIC Document. 1975
35. Kirk R.E. Practical significance: a concept whose time has come Educ. Psychol. Meas. 56 5 1996 746 759
36. Kitchenham B.A. et al. Preliminary guidelines for empirical research in software engineering IEEE Trans. Softw. Eng. 28 8 2002 721 734
37. Lane D.M., Dunlap W.P. Estimating effect size: bias resulting from the significance criterion in editorial decisions Br. J. Math. Stat. Psychol. 31 2 1978 107 112
38. Levine M., Ensom M.H. Post hoc power analysis: an idea whose time has passed? Pharmacother. J. Hum. Pharmacol. Drug Ther. 21 4 2001 405 409
39. Martinson B.C., Anderson M.S., De Vries R. Scientists behaving badly Nature 435 7043 2005 737 738
40. Masicampo E., Lalande D.R. A peculiar prevalence of p values just below.05 Q. J. Exp. Psychol. 65 11 2012 2271 2279
41. Miller J. et al. Statistical power and its subcomponents-missing and misunderstood concepts in empirical software engineering research Inf. Softw. Technol. 39 4 1997 285 295
42. Murayama K., Pekrun R., Fiedler K. Research practices that can prevent an inflation of false-positive rates Pers. Soc. Psychol. Rev. 18 2014 107 118 10.1177/1088868313496330 ISSN 1532-7957
43. Perry D.E., Porter A.A., Votta L.G. Empirical studies of software engineering: a roadmap Proceedings of the Conference on The Future of Software Engineering 2000 ACM
44. Prinz F., Schlange T., Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10 9 2011 712
45. Schimmack U. The ironic effect of significant results on the credibility of multiple-study articles Psychol. Methods 17 4 2012 551 566
46. Shepperd M., Bowes D., Hall T. Researcher bias: the use of machine learning in software defect prediction IEEE Trans. Softw. Eng. 40 6 2014 603 616
47. Simmons J.P., Nelson L.D., Simonsohn U. False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant Psychol. Sci. 22 11 2011 1359 1366
48. Sjøberg D. et al. A survey of controlled experiments in software engineering IEEE Trans. Softw. Eng. 31 9 2005 733 753
49. Sjøberg D.I., Dybå T., Jørgensen M. The future of empirical methods in software engineering research Future of Software Engineering, 2007. FOSE'07 2007 IEEE
50. Tannock I.F. False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons J. Natl. Cancer Inst. 88 3-4 1996 206 207
51. Tversky A., Kahneman D. Belief in the law of small numbers Psychol. Bull. 76 2 1971 105
52. van Assen M.A. et al. Why publishing everything is more effective than selective publishing of statistically significant results PloS One 9 1 2014 e84896
53. Wacholder S. et al. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies J. Natl. Cancer Inst. 96 6 2004 434 442
54. Wetzels R. et al. Statistical evidence in experimental psychology an empirical comparison using 855 t tests Perspect. Psychol. Sci. 6 3 2011 291 298
55. Wohlin C. et al. Experimentation in Software Engineering 2012 Springer
56. Zeller A., Zimmermann T., Bird C. Failure is a four-letter word: a parody in empirical research Proceedings of the 7th International Conference on Predictive Models in Software Engineering 2011 ACM