The prevalence of statistical reporting errors in psychology (1985–2013)

Behavior Research Methods

Volumn 48, Issue 4, 2016, Pages 1205-1226

  Nuijten, Michèle B ^a   Hartgerink, Chris H J ^a   van Assen, Marcel A L M ^a   Epskamp, Sacha ^b   Wicherts, Jelte M ^a  

^a TILBURG UNIVERSITY   (Netherlands)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

False positives; NHST; p values; Publication bias; Questionable research practices; Reporting errors; Significance

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN; PREVALENCE; STATISTICAL BIAS; STATISTICS AND NUMERICAL DATA;

BEHAVIORAL RESEARCH; BIAS (EPIDEMIOLOGY); HUMANS; PREVALENCE;

EID: 84945157481     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-015-0664-2     Document Type: Article

References (40)

1. Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011). Public availability of published research data in high-impact journals. PLoS One, 6(9), e24357. doi:10.1371/journal.pone.0024357
2. American Psychological Association. (1983). Publication Manual of the American Psychological Association (3rd ed.). Washington, DC: American Psychological Association.
3. American Psychological Association. (2010). Publication Manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.
4. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678. doi:10.3758/s13428-011-0089-5
5. Bakker, M., & Wicherts, J. M. (2014). Outlier removal and the relation with reporting errors and quality of research. Manuscript submitted for publication.
6. Berle, D., & Starcevic, V. (2007). Inconsistencies between reported test statistics and p-values in two psychiatry journals. International Journal of Methods in Psychiatric Research, 16(4), 202–207. doi:10.1002/mpr.225
7. Caperos, J. M., & Pardo, A. (2013). Consistency errors in p-values reported in Spanish psychology journals. Psicothema, 25(3), 408–414.
8. Chamberlain, S., Boettiger, C., & Ram, K. (2014). rplos: Interface to PLoS Journals search API. R package version 0.4.0. http://CRAN.R-project.org/package=rplos
9. Cohen, J. (1994). The earth is round (P less-than.05). American Psychologist, 49(12), 997–1003.
10. Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A.,.. Wilson, S. (2007). Statistical reform in psychology: Is anything changing? Psychological science, 18(3), 230–232.
11. Epskamp, S., & Nuijten, M. B. (2015). statcheck: Extract statistics from articles and recompute p values. R package version 1.0.1. http://CRAN.R-project.org/package=statcheck
12. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. doi:10.1007/s11192-011-0494-7
13. Fanelli, D. (2013). Why Growing Retractions Are (Mostly) a Good Sign. Plos Medicine, 10(12). doi: 10.1371/journal.pmed.1001563
14. Fanelli, D. (2014). Rise in retractions is a signal of integrity. Nature, 509(7498), 33–33.
15. Fidler, F., & Cumming, G. (2005). Teaching confidence intervals: Problems and potential solutions. Proceedings of the 55th International Statistics Institute Session.
16. Fiedler, K., & Schwarz, N. (2015). Questionable Research Practices Revisited.
17. Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015). Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108(2), 275–297.
18. Garcia-Berthou, E., & Alcaraz, C. (2004). Incongruence between test statistics and P values in medical papers. Bmc Medical Research Methodology, 4, 13. doi:10.1186/1471-2288-4-13
19. Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571. doi:10.1177/1745691612457576
20. Gotzsche, P. C., Hrobjartsson, A., Maric, K., & Tendal, B. (2007). Data extraction errors in meta-analyses that use standardized mean differences. Jama-Journal of the American Medical Association, 298(4), 430–437.
21. Hartgerink, C. H. J., van Assen, M. A. L. M., & Wicherts, J. M. (2015). Too Good to be False: Non-Significant Results Revisited. Retrieved from osf.io/qpfnw.
22. Hubbard, R., & Ryan, P. A. (2000). The historical growth of statistical significance testing in psychology-and its future prospects. Educational and Psychological Measurement, 60, 661–681.
23. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological science, 23, 524–532. doi:10.1177/0956797611430953
24. Krueger, J. (2001). Null hypothesis significance testing - On the survival of a flawed method. American Psychologist, 56(1), 16–26.
25. Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. (2013). The life of p:“Just significant” results are on the rise. The Quarterly Journal of Experimental Psychology, 66(12), 2303–2309.
26. Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612–625. doi:10.1111/j.1468-2958.2002.tb00828.x
27. Nosek, B. A., Spies, J., & Motyl, M. (2012). Scientific Utopia: II - Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058
28. R Core Team. (2014). R: A Language and Environment for Statistical Computing. http://www.R-project.org/
29. Rozeboom, W. W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416–428.
30. Sakaluk, J., Williams, A., & Biernat, M. (2014). Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science. Perspectives on Psychological Science, 9(6), 652–660.
31. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547.
32. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance - Or vice versa. Journal of the American Statistical Association, 54, 30–34.
33. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited - The effect of the outcome of statistical tests on the decision to publish and vice-versa. American Statistician, 49(1), 108–112.
34. Van Assen, M. A. L. M., Van Aert, R. C. M., & Wicherts, J. M. (2014). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods.
35. Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., Van Assen, M. A. L. M., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science PLoS One.
36. Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi:10.3758/BF03194105
37. Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7. doi:10.1038/480007a
38. Wicherts, J. M. (2013). Science revolves around the data. Journal of Open Psychology Data, 1(1), e1.
39. Wicherts, J. M., & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence. doi:10.1016/j.intell.2012.01.004
40. Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6(11), e26828.