P-curve and p-hacking in observational research

PLoS ONE

Volumn 11, Issue 2, 2016, Pages

  Bruns, Stephan B ^a   Ioannidis, John P A ^b  

^a UNIVERSITY OF KASSEL   (Germany)

^b STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CALIBRATION; ECONOMIC DEVELOPMENT; EXPERIMENTAL STUDY; MALARIA; MEDICAL LITERATURE; MEDICAL RESEARCH; OBSERVATIONAL STUDY; P CURVE; P HACKING; PREVALENCE; PUBLICATION; RANDOMIZATION; SIMULATION; STATISTICAL ANALYSIS; STATISTICAL PARAMETERS; STATISTICAL SIGNIFICANCE; ECONOMICS; HUMAN; RESEARCH; SAMPLE SIZE; STATISTICS; VIBRATION;

HUMANS; MALARIA; OBSERVATIONAL STUDIES AS TOPIC; PREVALENCE; RESEARCH; SAMPLE SIZE; STATISTICS AS TOPIC; VIBRATION;

EID: 84960454925     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0149144     Document Type: Article

References (40)

1. Simonsohn U, Nelson LD, Simmons JP. P-curve: A key to the file-drawer. J Exp Psychol Gen 2014; 143: 534-547. doi: 10.1037/a0033242 PMID: 23855496
2. Jager LR, Leek JT. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 2014; 15: 1-12. doi: 10.1093/biostatistics/kxt007 PMID: 24068246
3. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The Extent and Consequences of P-Hacking in Science. PLoS Biol 2015; 13: E1002106. doi: 10.1371/journal.pbio.1002106 PMID: 25768323
4. de Winter JCF, Dodou D. A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ 2015; 3: E733. doi: 10.7717/peerj.733 PMID: 25650272
5. Lakens D. On the challenges of drawing conclusions from p-values just below 0.05. PeerJ 2015; 3: E1142. doi: 10.7717/peerj.1142 PMID: 26246976
6. Ioannidis JPA. Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry 2011; 68: 773-780. doi: 10.1001/archgenpsychiatry.2011.28 PMID: 21464342
7. Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, Howells DW, et al. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol 2013; 11: E1001609. doi: 10. 1371/journal.pbio.1001609 PMID: 23874156
8. Dal-Ré R, Ioannidis JP, Bracken MB, Buffler PA, Chan AW, Franco EL, et al. Making prospective registration of observational research a reality. Sci Transl Med 2014; 6: 224cm1. doi: 10.1126/scitranslmed. 3007513 PMID: 24553383
9. John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 2012; 23: 524-532. doi: 10.1177/0956797611430953 PMID: 22508865
10. Fanelli D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One 2009; 4: E5738. doi: 10.1371/journal.pone.0005738 PMID: 19478950
11. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011; 22: 1359-1366. doi: 10. 1177/0956797611417632 PMID: 22006061
12. Greene WH. Econometric Analysis. 7th ed. Upper Saddle River, NJ: Prentice Hall; 2012.
13. Schuemie MJ, Ryan PB, Dumouchel W, Suchard MA, Madigan D. Interpreting observational studies: Why empirical calibration is needed to correct p-values. Stat Med 2014; 33: 209-218. doi: 10.1002/sim. 5925 PMID: 23900808
14. Young SS, Karr A. Deming, data and observational studies. Significance 2011; 8: 116-120.
15. White H. A reality check for data snooping. Econometrica 2000; 68: 1097-1126.
16. Lovell MC. Data Mining. Rev Econ Stat 1983; 65: 1-12.
17. Ioannidis JP. Why most discovered true associations are inflated. Epidemiology 2008; 19: 640-648. doi: 10.1097/EDE.0b013e31818131e7 PMID: 18633328
18. Leamer EE. Let's take the con out of econometrics. Am Econ Rev 1983; 73: 31-43.
19. Hendry DF. Econometrics-alchemy or science?. Economica 1980; 47: 387-406.
20. Bruns SB. The fragility of meta-regression models in observational research. MAGKS Joint Discussion Paper Series in Economics 03-2016.
21. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
22. Durlauf SN, Johnson PA, Temple JR. Growth econometrics. In: Aghion P, Durlauf SN, editors. Handbook of economic growth 2005, volume 1, Part A. pp. 555-677.
23. Sala-i-Martin X, Doppelhofer G, Miller RI. Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach. Am Econ Rev 2004; 94: 813-835.
24. Rockey J, Temple J. Growth Econometrics for Agnostics and True Believers. Eur Econ Rev 2015; in press.
25. Patel CJ, Burford B, Ioannidis JP. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J Clin Epidemiol. Forthcoming 2015.
26. Schuemie MJ, Ryan PB, Suchard MA, Shahn Z, Madigan D. Discussion: An estimate of the sciencewise false discovery rate and application to the top medical literature. Biostatistics 2014; 15: 36-39. doi: 10.1093/biostatistics/kxt037 PMID: 24068252
27. Serghiou S, Patel CJ, Tan YT, Koay P, Ioannidis JP. Field-wide meta-analyses of observational associations can map selective availability of risk factors and the impact of model specifications. J Clin Epidemiol. Forthcoming 2015.
28. Boccia S, Rothman KJ, Panic N, Flacco ME, Rosso A, Pastorino R., et al. Registration practices for observational studies on clinicaltrials.gov indicated low adherence. J Clin Epidemiol. Forthcoming 2015.
29. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005; 294: 218-228. PMID: 16014596
30. Khoury MJ, Ioannidis JP. Big data meets public health. Science 2014; 346: 1054-1055. doi: 10.1126/science.aaa2709 PMID: 25430753
31. Bishop DV, Thompson PA. Problems in using text-mining and p-curve analysis to detect rate of p-hacking. PeerJ PrePrints 2015 [cited 2015 Aug 27]; 3: E1550.
32. Lakens D. What p-hacking really looks like: A comment on Masicampo and LaLande (2012). Q J Exp Psychol 2015; 68: 829-832.
33. Savovic J, Jones HE, Altman DG, Harris RJ, Jüni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012; 157: 429-438. PMID: 22945832
34. Ulrich R, Miller J. p-hacking by post hoc selection with multiple opportunities: Detectability by skewness test?: Comment on Simonsohn, Nelson, and Simmons (2014). J Exp Psychol Gen 2015; 144: 1137-1145. doi: 10.1037/xge0000086 PMID: 26595841
35. Saquib N, Saquib J, Ioannidis JP. Practices and impact of primary outcome adjustment in randomized controlled trials: Meta-epidemiologic study. Br Med J2013; 347: F4313.
36. Ioannidis JP. Why most published research findings are false. PLoS Med 2005; 2: E124. PMID: 16060722
37. Gelman A, O'Rourke K. Discussion: Difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics 2014; 15: 18-23. doi: 10.1093/biostatistics/kxt034 PMID: 24068249
38. Ioannidis JP. Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false. Biostatistics 2014; 15: 28-36. doi: 10.1093/biostatistics/kxt036 PMID: 24068251
39. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 2015; 349: Aac4716. doi: 10.1126/science.aac4716 PMID: 26315443
40. Klein RA, Ratliff KA, Vianello M, Adams RB Jr, Bahnik S, Bernstein MJ, et al. Investigating variation in replicability. Soc Psychol (Gott) 2015; 45: 142-152.