Distributions of p-values smaller than .05 in psychology: What is going on?

PeerJ

Volumn 2016, Issue 4, 2016, Pages

  Hartgerink, Chris H J ^a   Van Aert, Robbie C M ^a   Nuijten, Michèle B ^a   Wicherts, Jelte M ^a   Van Assen, Marcel A L M ^a,b  

^a TILBURG UNIVERSITY   (Netherlands)

^b UTRECHT UNIVERSITY   (Netherlands)

Author keywords

Caliper test; Data peeking; NHST; p values; QRP

Indexed keywords

ARTICLE; CALIPER; COMPUTER ANALYSIS; COMPUTER SIMULATION; MATHEMATICAL PARAMETERS; NONHUMAN; PREVALENCE; PSYCHOLOGY; PUBLISHING; SAMPLE SIZE; STATISTICAL SIGNIFICANCE;

EID: 84966333692     PISSN: None     EISSN: 21678359     Source Type: Journal    
DOI: 10.7717/peerj.1935     Document Type: Article

References (45)

1. American Psychological Association. 1983. Publication manual of the American Psychological Association. 3rd edition. Washington, D.C.: American Psychological Association.
2. American Psychological Association. 2001. Publication manual of the American psychological association. 5th edition. Washington, D.C.: American Psychological Association.
3. American Psychological Association. 2010. Publication manual of the American Psychological Association. 6th edition. Washington, D.C.: American Psychological Association.
4. Armitage P, McPherson CK, Rowe BC. 1969. Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A 132(2):235-244.
5. Asendorpf JB, Conner M, De Fruyt F, De Houwer J, Denissen JJA, Fiedler K, Fiedler S, Funder DC, Kliegl R, Nosek BA, Perugini M, Roberts BW, Schmitt M, Van Aken MAG, Weber H, Wicherts JM. 2013. Recommendations for increasing replicability in psychology. European Journal of Personality 27(2):108-119 DOI 10.1002/per.1919.
6. Bakker M, Wicherts JM. 2011. The (mis)reporting of statistical results in psychology journals. Behavior Research Methods 43(3):666-678 DOI 10.3758/s13428-011-0089-5.
7. Bakker M, Wicherts JM. 2014. Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: the power of alternatives and recommendations. Psychological Methods 19(3):409-427 DOI 10.1037/met0000014.
8. Benjamini Y, Hechtlinger Y. 2014. Discussion: an estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek. Biostatistics 15(1):13-16 DOI 10.1093/biostatistics/kxt032.
9. Bishop DV, Thompson PA. 2016. Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ 4:e1715 DOI 10.7717/peerj.1715.
10. Chamberlain S, Boettiger C, Ram K. 2015. rplos: interface to the search 'API' for 'PLoS' journals. Available at https://cran.r-project.org/web/packages/rplos/index.html.
11. De Winter JC, Dodou D. 2015. A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ 3:e733 DOI 10.7717/peerj.733.
12. Epskamp S, Nuijten M. 2015. statcheck: extract statistics from articles and recompute p-values. Available at https://cran.r-project.org/web/packages/statcheck/statcheck.pdf.
13. Ferguson CJ. 2015. Everybody knows psychology is not a real science: public perceptions of psychology and how we can improve our relationship with policymakers, the scientific community, and the general public. The American Psychologist 70(6):527-542 DOI 10.1037/a0039405.
14. Franco A, Malhotra N, Simonovits G. 2014. Publication bias in the social sciences: unlocking the file drawer. Science 345(6203):1502-1505 DOI 10.1126/science.1255484.
15. Franco A, Malhotra N, Simonovits G. 2015. Underreporting in psychology experiments: evidence from a study registry. Social Psychological and Personality Science DOI 10.1177/1948550615598377.
16. García-Berthou E, Alcaraz C. 2004. Incongruence between test statistics and P values in medical papers. BMC Medical Research Methodology 4:13 DOI 10.1186/1471-2288-4-13.
17. Gelman A, O'Rourke K. 2014. Discussion: difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics 15(1):18-23 DOI 10.1093/biostatistics/kxt034.
18. Gerber AS, Malhotra N, Dowling CM, Doherty D. 2010. Publication bias in two political behavior literatures. American Politics Research 38(4):591-613 DOI 10.1177/1532673X09350979.
19. Ginsel B, Aggarwal A, Xuan W, Harris I. 2015. The distribution of probability values in medical abstracts: an observational study. BMC Research Notes 8(1):721 DOI 10.1186/s13104-015-1691-x.
20. Goodman SN. 2014. Discussion: an estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15(1):23-27 DOI 10.1093/biostatistics/kxt035.
21. Hartgerink CHJ. 2015. Reanalyzing Head et al. (2015): no widespread p-hacking after all? Available at https://www.authorea.com/users/2013/articles/31568.
22. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. 2015. The extent and consequences of p-hacking in science. PLoS Biology 13(3):e1002106 DOI 10.1371/journal.pbio.1002106.
23. Ioannidis JPA. 2014. Discussion: Why "an estimate of the science-wise false discovery rate and application to the top medical literature" is false. Biostatistics 15(1):28-36 DOI 10.1093/biostatistics/kxt036.
24. Jager LR, Leek JT. 2014. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15(1):1-12 DOI 10.1093/biostatistics/kxt007.
25. John LK, Loewenstein G, Prelec D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23(5):524-532 DOI 10.1177/0956797611430953.
26. Krawczyk M. 2015. The search for significance: a few peculiarities in the distribution of p values in experimental psychology literature. PLoS ONE 10(6):e0127872 DOI 10.1371/journal.pone.0127872.
27. Kühberger A, Fritz A, Scherndl T. 2014. Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size. PLoS ONE 9(9):e105825 DOI 10.1371/journal.pone.0105825.
28. Lakens D. 2015a. On the challenges of drawing conclusions from p-values just below 0.05. PeerJ 3:e1142 DOI 10.7717/peerj.1142.
29. Lakens D. 2015b. What p-hacking really looks like: a comment on masicampo and LaLande (2012). Quarterly Journal of Experimental Psychology 68(4):829-832 DOI 10.1080/17470218.2014.982664.
30. Leggett NC, Thomas NA, Loetscher T, Nicholls MER. 2013. The life of p: "just significant" results are on the rise. Quarterly Journal of Experimental Psychology 66(12):2303-2309 DOI 10.1080/17470218.2013.863371.
31. Masicampo EJ, Lalande DR. 2012. A peculiar prevalence of p values just below.05. Quarterly Journal of Experimental Psychology 65(11):2271-2279 DOI 10.1080/17470218.2012.711335.
32. Nuijten MB, Hartgerink CHJ, Van Assen MALM, Epskamp S, Wicherts JM. 2015. The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods Epub ahead of print Oct 23 2015 DOI 10.3758/s13428-015-0664-2.
33. Panel on Scientific Responsibility and the Conduct of Research. 1992. Responsible science, volume I: ensuring the integrity of the research process. Washington, D.C.: National Academies Press.
34. Pashler H, Wagenmakers E-J. 2012. Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science 7(6):528-530.
35. Pautasso M. 2010. Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85(1):193-202 DOI 10.1007/s11192-010-0233-5.
36. Ridley J, Kolm N, Freckelton RP, Gage MJG. 2007. An unexpected influence of widely used significance thresholds on the distribution of reported p-values. Journal of Evolutionary Biology 20(3):1082-1089 DOI 10.1111/j.1420-9101.2006.01291.x.
37. Simmons JP, Nelson LD, Simonsohn U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22(11):1359-1366 DOI 10.1177/0956797611417632.
38. Simonsohn U, Nelson LD, Simmons JP. 2014. P-curve: a key to the file-drawer. Journal of Experimental Psychology: General 143(2):534-547 DOI 10.1037/a0033242.
39. Simonsohn U, Simmons JP, Nelson LD. 2015. Better p-curves. Journal of Experimental Psychology: General 144:1146-1152 DOI 10.1037/xge0000104.
40. Ulrich R, Miller J. 2015. p-hacking by post hoc selection with multiple opportunities: Detectability by skewness test? Comment on Simonsohn, Nelson, and Simmons (2014). Journal of Experimental Psychology. General 144(6):1137-1145 DOI 10.1037/xge0000086.
41. Van Assen MALM, Van Aert RCM, Wicherts JM. 2015. Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods 20:293-309 DOI 10.1037/met0000025.
42. Veldkamp CLS, Nuijten MB, Dominguez-Alvarez L, Van Assen MALM, Wicherts JM. 2014. Statistical reporting errors and collaboration on statistical analyses in psychological science. PLoS ONE 9(12):e114876 DOI 10.1371/journal.pone.0114876.
43. Vermeulen I, Beukeboom CJ, Batenburg A, Avramiea A, Stoyanov D, De Velde BV, Oegema D. 2015. Blinded by the light: how a focus on statistical "significance" may cause p-value misreporting and an excess of p-values just below.05 in communication science. Communication Methods and Measures 9(4):253-279 DOI 10.1080/19312458.2015.1096333.
44. Wagenmakers E-J. 2007. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 14(5):779-804 DOI 10.3758/BF03194105.
45. Wagenmakers E-J, Wetzels R, Borsboom D, Van der Maas HLJ, Kievit RA. 2012. An agenda for purely confirmatory research. Perspectives on Psychological Science: a Journal of the Association for Psychological Science 7(6):632-638 DOI 10.1177/1745691612463078.