The Extent and Consequences of P-Hacking in Science

PLoS Biology

Volumn 13, Issue 3, 2015, Pages

  Head, Megan L ^a   Holman, Luke ^a   Lanfear, Rob ^a,b   Kahn, Andrew T ^a   Jennions, Michael D ^a  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b MACQUARIE UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CONSENSUS; DATA MINING; EFFECT SIZE; MEDICAL LITERATURE; META ANALYSIS (TOPIC); P HACKING; PUBLICATION; PUBLISHING; STATISTICAL SIGNIFICANCE; SYSTEMATIC ERROR; ETHICS; HUMAN; SCIENCE; STATISTICS; STATISTICS AND NUMERICAL DATA;

HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; SCIENCE; STATISTICS AS TOPIC;

EID: 84926391432     PISSN: 15449173     EISSN: 15457885     Source Type: Journal    
DOI: 10.1371/journal.pbio.1002106     Document Type: Article

References (66)

1. Barch DM, Yarkoni T, (2013) Introduction to the special issue on reliability and replication in cognitive and affective neuroscience research. Cogn Affect Behav Neurosci 13: 687–689. doi: 10.3758/s13415-013-0201-7 23922199
2. Ioannidis JPA, (2005) Why most published research findings are false. PLoS Med 2: e124. 16060722
3. Jager LR, Leek JT, (2014) An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15: 1–12. doi: 10.1093/biostatistics/kxt007 24068246
4. Begg CB, Berlin JA, (1988) Publication bias—a problem in interpreting medical data. J R Stat Soc Ser A Stat Soc 151: 419–463.
5. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, et al. (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3: e3081. doi: 10.1371/journal.pone.0003081 18769481
6. Fanelli D, (2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90: 891–904.
7. Rosenthal R, (1979) The file drawer problem and tolerance for null results. Psychol Bull 86: 638–641.
8. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ, (2000) Publication and related biases. Health technology assessment (Winchester, England) 4: 1–115.
9. Sterling TD, (1959) Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J Am Stat Assoc 54: 30–34.
10. Stern JM, Simes RJ, (1997) Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. Br Med J 315: 640–645. 9310565
11. Laurance WF, Useche DC, Laurance SG, Bradshaw CJA, (2013) Predicting publication success for biologists. Bioscience 63: 817–823.
12. Brodeur A, Le M, Sangnier M, Zylberberg Y (2012) Star Wars: The empirics strike back. Paris School of Economics Working Paper 2012. http://ssrn.com/abstract=2089580.
13. Cumming G, (2014) The new statistics: Why and how. Psychol Sci 25: 7–29. doi: 10.1177/0956797613504966 24220629
14. Simmons JP, Nelson LD, Simonsohn U, (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22: 1359–1366. doi: 10.1177/0956797611417632 22006061
15. Gadbury GL, Allison DB, (2014) Inappropriate fiddling with statistical analyses to obtain a desirable p-value: Tests to detect its presence in published literature. PLoS ONE 7: e46363.
16. John LK, Loewenstein G, Prelec D, (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23: 524–532. doi: 10.1177/0956797611430953 22508865
17. Hutton JL, Williamson PR, (2000) Bias in meta-analysis due to outcome variable selection within studies. J R Stat Soc Ser C Appl Stat 49: 359–370.
18. Bastardi A, Uhlmann EL, Ross L, (2011) Wishful thinking: Belief, desire, and the motivated evaluation of scientific evidence. Psychol Sci 22: 731–732. doi: 10.1177/0956797611406447 21515736
19. Nosek BA, Spies JR, Motyl M, (2012) Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect Psychol Sci 7: 615–631.
20. Fisher RA, (1925) Statistical methods for research workers. London: Oliver & Boyd.
21. Benjamini Y, Hechtlinger Y, (2014) Discussion: An estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek. Biostatistics 15: 13–16. doi: 10.1093/biostatistics/kxt032 24068247
22. Goodman SN, (1999) Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med 130: 995–1004. 10383371
23. Sterne JAC, Smith GD, (2001) Sifting the evidence—what's wrong with significance tests? Br Med J 322: 226–231.
24. Nickerson RS, (2000) Null hypothesis significance testing: A review of an old and continuing controversy. Psychol Methods 5: 241–301. 10937333
25. Trafimow D, (2003) Hypothesis testing and theory evaluation at the boundaries: Surprising insights from Bayes's theorem. Psychol Rev 110: 526–535. 12885113
26. Rohlf FJ, Sokal RR, (1995) Statistical tables. New York: W.H. Freeman.
27. Ridley J, Kolm N, Freckleton RP, Gage MJG, (2007) An unexpected influence of widely used significance thresholds on the distribution of reported P-values. J Evol Biol 20: 1082–1089. 17465918
28. Nakagawa S, Cuthill IC, (2007) Effect size, confidence interval and statistical significance: A practical guide for biologists. Biol Rev Camb Philos Soc 82: 591–605. 17944619
29. Anderson DR, Burnham KP, Thompson WL, (2000) Null hypothesis testing: Problems, prevalence, and an alternative. J Wildl Manage 64: 912–923.
30. Loftus GR, (1996) Psychology will be a much better science when we change the way we analyze data. Curr Dir Psychol Sci 5: 161–171.
31. Benjamini Y, Yekutieli D, (2005) False discovery rate-adjusted multiple confidence intervals for selected parameters J Am Stat Assoc 100: 71–81.
32. Simonsohn U (2014c) Posterior-hacking: Selective reporting invalidates Bayesian results also. http://ssrncom/abstract=2374040.
33. Koricheva J, Gurevitch J, Koricheva J, Gurevitch J, Mengersen K, (2013) Place of meta-analysis among other methods of research synthesis. In: Handbook of met-analysis in ecology and evolution. Princeton, New Jersey: Princeton University Press.
34. Pereira TV, Ioannidis JPA, (2011) Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. J Clin Epidemiol 64: 1060–1069. doi: 10.1016/j.jclinepi.2010.12.012 21454050
35. Jennions MD, Moller AP, Hunt J, (2004) Meta-analysis can "fail": Reply to Kotiaho and Tomkins. Oikos 104: 191–193.
36. Kotiaho JS, Tomkins JL, (2002) Meta-analysis, can it ever fail? Oikos 96: 551–553.
37. Palmer AR, (2000) Quasireplication and the contract of error: Lessons from sex ratios, heritabilities and fluctuating asymmetry. Annu Rev Ecol Sys 31: 441–480.
38. Kelly CD, (2006) Replicating empirical research in behavioural ecology: How and why it should be done but rarely ever is. The Quarterly Review of Biology 81: 221–236. 17051829
39. Cumming G, (2008) Replication and p intervals p values predict the future only vaguely, but confidence Intervals do much better. Perspectives on Psychological Science 3: 286–300.
40. Mariscampo EJ, Lalande DR, (2012) A peculiar prevalence of p values just below .05. Q Rev Biol 65: 2271–2279.
41. Simonsohn U, Nelson LD, Simmons JP, (2014a) P-curve: A key to the file drawer. J Exp Psychol Gen 143: 534–547. doi: 10.1037/a0033242 23855496
42. Wallis WA, (1942) Compounding probabilities from independent significance tests. Econometrica 10: 229–248.
43. Gerber AS, Malhotra N, (2008) Publication bias in empirical sociological research—Do arbitrary significance levels distort published results? Sociol Methods Res 37: 3–30.
44. Ho DE, (2013) Foreword: Conference bias. J Empir Leg Stud 10: 603–611.
45. Leggett NC, Thomas NA, Loetscher T, Nicholls MER, (2013) The life of p: "Just significant" results are on the rise. Q J Exp Psychol 66: 2303–2309. doi: 10.1080/17470218.2013.863371 24205936
46. Ioannidis JPA, (2014) Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false. Biostatistics 15: 28–36. doi: 10.1093/biostatistics/kxt036 24068251
47. Clopper CJ, Pearson ES, (1934) The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26: 404–413.
48. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD (2015) Data from: The extent and consequences of p-hacking in science. Dryad Digital Repository. http://dx.doi.org/10.5061/dryad.79d43.
49. Nuzzo R, (2014) Scientific method: Statistical errors. Nature 506: 150–152. doi: 10.1038/506150a 24522584
50. Klein RA, Ratliff KA, Vianello M, R.B. A Bahník S, et al. (2014) Data from investigating variation in replicability: A "many labs" replication project. J Open Psychol Data 2: e4.
51. Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F, (2000) Methods for meta-analysis in medical research. New york: John Wiley & Sons.
52. Gildersleeve K, Haselton MG, Fales MR, (2014) Meta-Analyses and p-curves support robust cycle shifts in women's mate preferencs: Reply to Wood & Carden (2014) and Harris, Pashler, and Mickes (2014). Psychol Bull 140: 1272–1280. doi: 10.1037/a0037714 25180805
53. Akcay E, Roughgarden J, (2007) Extra-pair paternity in birds: Review of the genetic benefits. Evol Ecol Res 9: 855–868.
54. Cleasby IR, Nakagawa S, (2012) The influence of male age on within-pair and extra-pair paternity in passerines. Ibis 154: 318–324.
55. de Jong K, Forsgren E, Sandvik H, Amundsen T, (2012) Measuring mating competition correctly: available evidence supports operational sex ratio theory. Behav Ecol 23: 1170–1177.
56. Jiang Y, Bolnick DI, Kirkpatrick M, (2013) Assortative mating in animals. Am Nat 181: E125–E138. doi: 10.1086/670160 23669548
57. Kelly CD, (2008) The interrelationships between resource-holding potential, resource-value and reproductive success in territorial males: How much variation can we explain? Behav Ecol Sociobiol 62: 855–871.
58. Kraaijeveld K, Kraaijeveld-Smit FJL, Maan ME, (2011) Sexual selection and speciation: The comparative evidence revisited. Biol Rev Camb Philos Soc 86: 367–377. doi: 10.1111/j.1469-185X.2010.00150.x 20659104
59. Prokop ZM, Michalczyk L, Drobniak SM, Herdegen M, Radwan J, (2012) Meta-analysis suggests choosy females get sexy sons more than "good genes". Evolution 66: 2665–2673. doi: 10.1111/j.1558-5646.2012.01654.x 22946794
60. Santos ESA, Scheck D, Nakagawa S, (2011) Dominance and plumage traits: Meta-analysis and metaregression analysis. Anim Behav 82: 3–19.
61. Weir LK, Grant JWA, Hutchings JA, (2011) The influence of operational sex ratio on the intensity of competition for mates. Am Nat 177: 167–176. doi: 10.1086/657918 21460553
62. Simonsohn U, Nelson LD, Simmons JP (2014b) P-Curve and effect size: Correcting for publication bias using only significant results. http://ssrncom/abstract=2377290
63. van Assen MALM, van Aert RCM, Wicherts JM, (2014) Meta-analysis using effect size distributions of only statistically significant studies. Psychol Methods Advance Online Publication.
64. Altman DG, Gore SM, Gardner MJ, Pocock SJ, (1983) Statistical guidelines for contributors to medical journals. Br Med J 286: 1489–1493. 6405856
65. Wagenmakers E-J, (2007) A practical solution to the pervasive problems of p values. Psychon Bull Rev 14: 779–804. 18087943
66. Hahn S, Williamson PR, Hutton JL, (2002) Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract 8: 353–359. 12164983