Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value

PeerJ

Volumn 2016, Issue 2, 2016, Pages

  Bishop, Dorothy V M ^a   Thompson, Paul A ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Correlation; Ghost variables; P curve; P hacking; Power; Reproducibility; Simulation; Text mining

Indexed keywords

DEPENDENT VARIABLE; HEAD; MINING; MODEL; STATISTICAL SIGNIFICANCE;

EID: 84963983996     PISSN: None     EISSN: 21678359     Source Type: Journal    
DOI: 10.7717/peerj.1715     Document Type: Article

References (27)

1. Academy of Medical Sciences, BBSRC, MRC, Wellcome Trust. 2015. Reproducibility and reliability of biomedical research: improving research practice. London: Academy of Medical Sciences. Available at http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research/.
2. Altman DG. 1991. Statistics in medical journals: developments in the 1980s. Statistics in Medicine 10:1897-1913 DOI 10.1002/sim.4780101206.
3. Begg CB, Berlin JA. 1988. Publication bias: a problem in interpreting medical data. Journal of the Royal Statistical Society: Series A 151(3):419-463 DOI 10.2307/2982993.
4. Bishop DV, Thompson PA. 2015. Problems in using text-mining and p-curve analysis to detect rate of p-hacking. PeerJ PrePrints 3:e1643 DOI 10.7287/peerj.preprints.1266v2.
5. Chernick MR, Liu CY. 2002. The saw-toothed behavior of power versus sample size and software solutions. The American Statistician 56:149-155 DOI 10.1198/000313002317572835.
6. Cohen J. 1992. Statistical power analysis. Current Directions in Psychological Science 1(3):98-101 DOI 10.1111/1467-8721.ep10768783.
7. De Groot AD. 2014. The meaning of ''significance'' for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L.J. van der Maas]. Acta Psychologica 148(0):188-194 DOI 10.1016/j.actpsy.2014.02.001.
8. De Winter JCF, Dodou D. 2015. A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ 3:e733 DOI 10.7717/peerj.733.
9. Gelman A, O'Rourke K. 2014. Discussion: difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics 15:18-23 DOI 10.1093/biostatistics/kxt034.
10. Greenwald AG. 1975. Consequences of prejudice against the null. Psychological Bulletin 82:1-20.
11. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. 2015. The extent and consequences of p-hacking in science. PLoS Biology 13:e1002106 DOI 10.1371/journal.pbio.1002106.
12. Ioannidis JPA. 2005. Why most published research findings are false. PLoS Medicine 2:e124 DOI 10.1371/journal.pmed.0020124.
13. Ioannidis JPA. 2014. How to make more published research true. PLoS Medicine 11:e1001747 DOI 10.1371/journal.pmed.1001747.
14. Ioannidis JPA, Munafò MR, Fusar-Poli P, Nosek BA, David SP. 2014. Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences 18:235-241 DOI 10.1016/j.tics.2014.02.010.
15. Jager LR, Leek JT. 2013. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15:28-36 DOI 10.1093/biostatistics/kxt007.
16. Kraemer HC. 2013. Statistical power: issues and proper applications. In: Comer JS, Kendall PC, eds. The Oxford handbook of research strategies for clinical psychology. Oxford: Oxford University Press.
17. Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. 2009. Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience 12:535-540 DOI 10.1038/nn.2303.
18. Lakens D. 2014. What p-hacking really looks like: a comment on Masicampo & Lalande (2012). Quarterly Journal of Experimental Psychology A 68:829-832.
19. Lakens D. 2015. On the challenges of drawing conclusions from p-values just below 0.05. PeerJ 3:e1142 DOI 10.7717/peerj.1142.
20. Masicampo EJ, Lalande DR. 2012. A peculiar prevalence of p values just below.05. Quarterly Journal of Experimental Psychology 65:2271-2279 DOI 10.1080/17470218.2012.711335.
21. Meehl PE. 1990. Why summaries of research on psychological theories are often uninterpretable. Psychological Reports 66:195-244 DOI 10.2466/pr0.1990.66.1.195.
22. Motulsky HJ. 2015. Common misconceptions about data analysis and statistics. British Journal of Pharmacology 172:2126-2132 DOI 10.1111/bph.12884.
23. Newcombe RG. 1987. Towards a reduction in publication bias. BMJ 295:656-659 DOI 10.1136/bmj.295.6599.656.
24. Reinhart A. 2015. Statistics done wrong: a woefully complete guide. San Francisco: No Starch Press.
25. Simonsohn U, Nelson LD, Simmons JP. 2014. P-curve: a key to the file-drawer. Journal of Experimental Psychology: General 143:534-547 DOI 10.1037/a0033242.
26. Simonsohn U, Simmons JP, Nelson LD. 2015. Better p-curves: making p-curve analysis more robust to errors, fraud, and ambitious p-hacking, a reply to Ulrich and Miller (2015). Journal of Experimental Psychology: General 144(6):1146-1152 DOI 10.1037/xge0000104.
27. Vul E, Harris C, Winkielman P, Pashler H. 2009. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science 4:274-290.