A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too)

PeerJ

Volumn 2015, Issue 1, 2015, Pages

  De Winter, Joost C F ^a   Dodou, Dimitra ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Bias; Biological sciences; Physical sciences; Science policy; Significant differences; Social sciences

Indexed keywords

ALGORITHM; ARTICLE; BIOMEDICINE; COMPARATIVE STUDY; DATA EXTRACTION; IDIOSYNCRATIC EXPRESSION; LONGITUDINAL TREND; MATHEMATICAL PARAMETERS; NATURAL SCIENCE; NEGATIVE FACTOR; NEGATIVE RESULT; NO SIGNIFICANT DIFFERENCE; POSITIVE RESULT; PUBLICATION; SCOPUS; SIGNAL DETECTION; SIGNIFICANT DIFFERENCE; SOCIOLOGY; STATISTICAL SIGNIFICANCE; VENN DIAGRAM; WEB BROWSER;

SCOPUS;

EID: 84926443410     PISSN: None     EISSN: 21678359     Source Type: Journal    
DOI: 10.7717/peerj.733     Document Type: Article

References (104)

1. Asendorpf JB, Conner M, De Fruyt F, De Houwer J, Denissen JJ, Fiedler K, Wicherts JM. 2013. Recommendations for increasing replicability in psychology. European Journal of Personality 27:108-119 DOI 10.1002/per.1919.
2. Atkin PA. 2002. A paradigm shift in the medical literature. British Medical Journal 325:1450-1451 DOI 10.1136/bmj.325.7378.1450.
3. Bakker M, Wicherts JM. 2014. Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: the power of alternatives and recommendations. Psychological Methods 19:409-427 DOI 10.1037/met0000014.
4. Basu S, Park HU. 2014. Publication bias in recent empirical accounting research. Available at http://ssrn.com/abstract=2379889.
5. Benjamini Y, Hechtlinger Y. 2014. Discussion: an estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek. Biostatistics 15:13-16 DOI 10.1093/biostatistics/kxt032.
6. Binfield P. 2009. PLoS One: background, future development, and article-level metrics. In: Rethinking Electronic Publishing: Innovation in Communication Paradigms and Technologies. Milan: Nuova Culture, 69-86.
7. Boutron I, Dutton S, Ravaud P, Altman DG. 2010. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. Journal of the American Medical Association 303:2058-2064 DOI 10.1001/jama.2010.651.
8. Brodeur A, Lé M, Sangnier M, Zylberberg Y. 2013. Star wars: The empirics strike back (Discussion Paper No. 7268), Paper Series. Bonn: Forschungsinstitut zur Zukunft der Arbeit.
9. Cacioppo JT. 2007. Psychology is a hub science. APS Observer 20. Available at http://aps.psychologicalscience.org/index.php/publications/observer/2007/september-07/psychology-is-a-hub-science.html.
10. Camfield L, Duvendack M, Palmer-Jones R. 2014. Things you wanted to know about bias in evaluations but never dared to think. IDS Bulletin 45:49-64 DOI 10.1111/1759-5436.12112.
11. Chabris CF, Hebert BM, Benjamin DJ, Beauchamp J, Cesarini D, Van der Loos M, Johannesson M, Magnusson PKE, Lichtenstein P, Atwood CS, Freese J, Hauser TS, Hauser RM, Christakis N, Laibson D. 2012. Most reported genetic associations with general intelligence are probably false positives. Psychological Science 23:1314-1323 DOI 10.1177/0956797611435528.
12. Chan A-W, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, Krumholz HM, Ghersi D, Van derWorp HB. 2014. Increasing value and reducing waste: addressing inaccessible research. The Lancet 383:257-266 DOI 10.1016/S0140-6736(13)62296-5.
13. Cumming G, Finch S. 2005. Inference by eye: confidence intervals and how to read pictures of data. American Psychologist 60:170-180 DOI 10.1037/0003-066X.60.2.170.
14. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, Schroeder TV, Sox HC, Van DerWeyden MB. 2004. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351:1250-1251 DOI 10.1056/NEJMe048225.
15. De Rond M, Miller AN. 2005. Publish or perish. Bane or boon of academic life? Journal of Management Inquiry 14:321-329 DOI 10.1177/1056492605276850.
16. DeWinter JCF, Zadpoor AA, Dodou D. 2014. The expansion of Google Scholar versus Web of Science: a longitudinal study. Scientometrics 98:1547-1565 DOI 10.1007/s11192-013-1089-2.
17. Dirnagl U, Lauritzen M. 2010. Fighting publication bias: introducing the Negative Results section. Journal of Cerebral Blood Flow and Metabolism 30:1263-1264 DOI 10.1038/jcbfm.2010.51.
18. Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, Adams JR, Kuderer NM, Lyman GH. 2000. The uncertainty principle and industry-sponsored research. The Lancet 356:635-638 DOI 10.1016/S0140-6736(00)02605-2.
19. Duval S, Tweedie R. 2000. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56:455-463 DOI 10.1111/j.0006-341X.2000.00455.x.
20. Dwan K, Altman DG, Arnaiz JA, BloomJ, Chan A-W, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JPA, Simes J, Williamson PR. 2008. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3:e3081 DOI 10.1371/journal.pone.0003081.
21. Elsevier. 2014. Printing and exporting citation overviews. Available at http://help.scopus.com/Content/h citovrdoc.htm.
22. Fanelli D. 2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4:e5738 DOI 10.1371/journal.pone.0005738.
23. Fanelli D. 2010. "Positive" results increase down the hierarchy of the sciences. PLoS ONE 5:e10068 DOI 10.1371/journal.pone.0010068.
24. Fanelli D. 2012. Negative results are disappearing from most disciplines and countries. Scientometrics 90:891-904 DOI 10.1007/s11192-011-0494-7.
25. Fanelli D. 2014. Reply to deWinter and Dodou (2014): growing bias and the hierarchy are actually supported, despite different design, errors, and disconfirmation-biases. Available at https://peerj.com/preprints/447/#feedback.
26. Fanelli D, Ioannidis JP. 2013. US studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences of the United States of America 110:15031-15036 DOI 10.1073/pnas.1302997110.
27. Ferguson CJ, Heene M. 2012. A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7:555-561 DOI 10.1177/1745691612459059.
28. Fern EF, Monroe KB. 1996. Effect-size estimates: Issues and problems in interpretation. Journal of Consumer Research 23:89-105 DOI 10.1086/209469.
29. Fiedler K, Kutzner F, Krueger JI. 2012. The long way from a-error control to validity proper. Problems with a short-sighted false-positive debate. Perspectives on Psychological Science 7:661-669 DOI 10.1177/1745691612462587.
30. Finkel EJ, Eastwick PW, Reis HT. Best research practices in psychology: illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology In Press.
31. Francis G. 2013. Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology 57:153-169 DOI 10.1016/j.jmp.2013.02.003.
32. Francis G. 2014. The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review 21:1180-1187 DOI 10.3758/s13423-014-0601-x.
33. Franco A, Malhotra N, Simonovits G. 2014. Publication bias in the social sciences: unlocking the file drawer. Science 345:1502-1505 DOI 10.1126/science.1255484.
34. Gadbury GL, Allison DB. 2012. Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature. PLoS ONE 7:e46363 DOI 10.1371/journal.pone.0046363.
35. Gerber AS, Malhotra N. 2008a. Publication bias in empirical sociological research. Do arbitrary significance levels distort published results? Sociological Methods & Research 37:3-30 DOI 10.1177/0049124108318973.
36. Gerber A, Malhotra N. 2008b. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3:313-326 DOI 10.1561/100.00008024.
37. Gibson G. 2010. Hints of hidden heritability in GWAS. Nature Genetics 42:558-560 DOI 10.1038/ng0710-558.
38. Goodman SN. 2014. Discussion: an estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15:23-27 DOI 10.1093/biostatistics/kxt035.
39. Google Scholar. 2014. Search tips. Available at http://scholar.google.com/intl/en/scholar/help.html#export.
40. Gøtzsche PC. 2006. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 333:231-234 DOI 10.1136/bmj.38895.410451.79.
41. Hand DJ. 2004. Measurement: theory and practice. London: Arnold.
42. Hand D. 2014. The improbability principle: why coincidences, miracles and rare events happen all the time. London: Bantam Press.
43. Hankins M. 2014. Also search was abstract only (!), p to 3dp only (not 2 or 4+), 'p=' but not 'p <' or 'p >' [Tweet]. Available at https://twitter.com/mc hankins/status/493125467290238976.
44. Hankins M. 2013. Still not significant [blog post]. Available at http://mchankins.wordpress.com/2013/04/21/still-not-significant-2/(accessed 21 April 2013).
45. Hasselman F. 2014. Feedback on DeWinter and Dodou (2014). A surge of p-values between 0.040 and 0.049 in recent decades (but negative results are increasing rapidly too). Available at https://peerj.com/preprints/447/#feedback.
46. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. 2009. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews 1 DOI 10.1002/14651858.MR000006.pub3.
47. Hunter JE, Schmidt FL. 1996. Cumulative research knowledge and social policy formulation: the critical role of meta-analysis. Psychology, Public Policy, and Law 2:324-347 DOI 10.1037/1076-8971.2.2.324.
48. Ioannidis J. 2003. Genetic associations: false or true? Trends in Molecular Medicine 9:135-138 DOI 10.1016/S1471-4914(03)00030-3.
49. Ioannidis JP. 2005. Why most published research findings are false. PLoS Medicine 2:e124 DOI 10.1371/journal.pmed.0020124.
50. Ioannidis J. 2008. Calibration of credibility of agnostic genome-wide associations. American Journal of Medical Genetics Part B: Neuropsychiatric Genetics 147:964-972 DOI 10.1002/ajmg.b.30721.
51. Ioannidis JP. 2014. Discussion: why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false. Biostatistics 15:28-36 DOI 10.1093/biostatistics/kxt036.
52. Ioannidis J, Munafò MR, Fusar-Poli P, Nosek BA, David SP. 2014. Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences 18:235-241 DOI 10.1016/j.tics.2014.02.010.
53. Jacsó P. 2008. Google scholar revisited. Online Information Review 32:102-114 DOI 10.1108/14684520810866010.
54. Jager LR, Leek JT. 2014. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15:1-12 DOI 10.1093/biostatistics/kxt007.
55. Jennings RG, Van Horn JD. 2012. Publication bias in neuroimaging research: implications for meta-analyses. Neuroinformatics 10:67-80 DOI 10.1007/s12021-011-9125-y.
56. Jennions MD, Møller AP. 2002. Publication bias in ecology and evolution: an empirical assessment using the 'trim and fill' method. Biological Reviews of the Cambridge Philosophical Society 77:211-222 DOI 10.1017/S1464793101005875.
57. Joober R, Schmitz N, Annable L, Boksa P. 2012. Publication bias: what are the challenges and can they be overcome? Journal of Psychiatry & Neuroscience 37:149-152 DOI 10.1503/jpn.120065.
58. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. 2010. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340:c365 DOI 10.1136/bmj.c365.
59. Kühberger A, Fritz A, Scherndl T. 2014. Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size. PLoS ONE 9:e105825 DOI 10.1371/journal.pone.0105825.
60. Kyzas PA, Denaxa-Kyza D, Ioannidis J. 2007. Almost all articles on cancer prognostic markers report statistically significant results. European Journal of Cancer 43:2559-2579 DOI 10.1016/j.ejca.2007.08.030.
61. Lakens D. What p-hacking really looks like: a comment onMasicampo & Lalande (2012). Quarterly Journal of Experimental Psychology In Press.
62. Larsen PO, Von Ins M. 2010. The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84:575-603 DOI 10.1007/s11192-010-0202-z.
63. Laws KR. 2013. Negativland-a home for all findings in psychology. BMC Psychology 1:2 DOI 10.1186/2050-7283-1-2.
64. Leggett NC, Thomas NA, Loetscher T, Nicholls ME. 2013. The life of p: "just significant" results are on the rise. The Quarterly Journal of Experimental Psychology 66:2303-2309 DOI 10.1080/17470218.2013.863371.
65. Lexchin J, Bero LA, Djulbegovic B, Clark O. 2003. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326:1167-1170 DOI 10.1136/bmj.326.7400.1167.
66. Leydesdorff L, DeMoya-Anegón F, De Nooy W. Aggregated journal-journal citation relations in Scopus andWeb-of-Science matched and compared in terms of networks, maps, and interactive overlays. Journal of the Association of Information Science and Technology In press.
67. Leydesdorff L, Wagner C. 2009. Is the United States losing ground in science? A global perspective on the world science system. Scientometrics 78:23-36 DOI 10.1007/s11192-008-1830-4.
68. Low N. 2014. @PeerJPrePrints @thePeerJ #nearlysignificant Might it just be because people don't use lookup tables for p values but give exact values? [Tweet]. Available at https://twitter.com/nicolamlow/status/491460395216171008 (accessed 21 July 2014).
69. Lyons L. 2013. Discovering the Significance of 5s. ArXiv preprint. arXiv:arXiv:1310.1284.
70. Masicampo EJ, Lalande DR. 2012. A peculiar prevalence of p values just below.05. The Quarterly Journal of Experimental Psychology 65:2271-2279 DOI 10.1080/17470218.2012.711335.
71. Mathôt S. 2013. No particular prevalence of p values just below.05 [blog post]. Available at http://www.cogsci.nl/blog/miscellaneous/221-no-particular-prevalence-of-p-values-just-below-05.
72. Meehl PE. 1978. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 46:806-834 DOI 10.1037/0022-006X.46.4.806.
73. Michel J-B, Shen YK, Aiden AP, Veres A, Gray MK, Pickett JP, Hoiberg D, Clancy D, Norvig P, Orwant J, Pinker S, Nowak MA, Aiden EL. 2011. Quantitative analysis of culture using millions of digitized books. Science 331:176-182 DOI 10.1126/science.1199644.
74. Mulaik SA, Raju NS, Harshman RA. 1997. There is a time and place for significance testing. In: Harlow LL, Mulaik SA, Steiger JH, eds. What if there were no significance tests? Hillsdale, NJ: Erlbaum, 65-116.
75. Nuijten MB, Van Assen MA, Van Aert RC, Wicherts JM. 2014. Standard analyses fail to show that US studies overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences of the United States of America 111:E712-E713 DOI 10.1073/pnas.1322149111.
76. Pan Z, Trikalinos TA, Kavvoura FK, Lau J, Ioannidis JP. 2005. Local literature bias in genetic epidemiology: an empirical evaluation of the Chinese literature. PLoS Medicine 2:e334 DOI 10.1371/journal.pmed.0020334.
77. Pashler H, Harris CR. 2012. Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7:531-536 DOI 10.1177/1745691612463401.
78. Pautasso M. 2010. Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85:193-202 DOI 10.1007/s11192-010-0233-5.
79. Pocock SJ, Collier TJ, Dandreo KJ, De Stavola BL, Goldman MB, Kalish LA, McCormack VA. 2004. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 329:883 DOI 10.1136/bmj.38250.571088.55.
80. Popper KR. 1959. The logic of scientific discovery. London: Hutchinson.
81. 2O in applied magnetic fields. Hyperfine Interactions 116:83-89 DOI 10.1023/A:1012611928991.
82. Ridley J, Kolm N, Freckelton RP, Gage MJG. 2007. An unexpected influence of widely used significance thresholds on the distribution of reported P-values. Journal of Evolutionary Biology 20:1082-1089 DOI 10.1111/j.1420-9101.2006.01291.x.
83. Ripple AM, Mork JG, Knecht LS, Humphreys BL. 2011. A retrospective cohort study of structured abstracts in MEDLINE, 1992-2006. Journal of the Medical Library Association 99:160-163 DOI 10.3163/1536-5050.99.2.009.
84. Rothstein HR, Sutton AJ, Borenstein M (eds.) 2006. Publication bias in meta-analysis: prevention, assessment and adjustments. West Sussex: John Wiley & Sons.
85. Rücker G, Schwarzer G, Carpenter J. 2008. Arcsine test for publication bias in meta-analyses with binary outcomes. Statistics in Medicine 27:746-763 DOI 10.1002/sim.2971.
86. Sackett DL. 1979. Bias in analytic research. Journal of Chronic Diseases 32:51-63 DOI 10.1016/0021-9681(79)90012-2.
87. Schmidt FL, Hunter JE. 1996. Measurement error in psychological research: lessons from 26 research scenarios. Psychological Methods 1:199-233 DOI 10.1037/1082-989X.1.2.199.
88. Sherman R. 2014. I thought p <.10 was "marginally sig". Why not look for increases in p >.10? That would be more convincing. [Tweet]. Available at https://twitter.com/RyneSherman/status/494112263200968704 (accessed 29 July 2014).
89. Simmons JP, Nelson LD, Simonsohn U. 2011. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:1359-1366 DOI 10.1177/0956797611417632.
90. Simonsohn U, Nelson LD, Simmons JP. 2014. P-curve: a key to the file-drawer. Journal of Experimental Psychology: General 143:534-547 DOI 10.1037/a0033242.
91. Sismondo S. 2008. Pharmaceutical company funding and its consequences: a qualitative systematic review. Contemporary Clinical Trials 29:109-113 DOI 10.1016/j.cct.2007.08.001.
92. Smart RG. 1964. The importance of negative results in psychological research. Canadian Psychologist/Psychologie Canadienne 5:225-232 DOI 10.1037/h0083036.
93. Sterling TD, Rosenbaum WL, Weinkam JJ. 1995. Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician 49:108-112.
94. Sterne JA. 2002. Teaching hypothesis tests-time for significant change? Statistics in Medicine 21:985-994 DOI 10.1002/sim.1129.
95. Strube MJ. 2006. SNOOP: a program for demonstrating the consequences of premature and repeated null hypothesis testing. Behavior Research Methods 38:24-27 DOI 10.3758/BF03192746.
96. Terrin N, Schmid CH, Lau J, Olkin I. 2003. Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine 22:2113-2126 DOI 10.1002/sim.1461.
97. The Thomson Cooperation. 2008. Essential Science Indicators. Available at http://esi.webofknowledge.com/help/h datjrn.htm.
98. Thomson Reuters. 2014. Methodology for Sciencewatch.com. Available at http://archive.sciencewatch.com/about/met/.
99. Thornton A, Lee P. 2000. Publication bias in meta-analysis: its causes and consequences. Journal of Clinical Epidemiology 53:207-216 DOI 10.1016/S0895-4356(99)00161-4.
100. Vickers A, Goyal N, Harland R, Rees R. 1998. Do certain countries produce only positive results? A systematic review of controlled trials. Controlled Clinical Trials 19:159-166 DOI 10.1016/S0197-2456(97)00150-5.
101. Wagenmakers EJ. 2007. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 14:779-804 DOI 10.3758/BF03194105.
102. Wigner EP. 1960. The unreasonable effectiveness of mathematics in the natural sciences. In: Richard courant lecture in mathematical sciences delivered at New York University, May 11, 1959, Communications on pure and applied mathematics, vol. 13. 1-14.
103. Yong E. 2012. Replication studies: bad copy. Nature 485:298-300 DOI 10.1038/485298a.
104. Young NS, Ioannidis JP, Al-Ubaydli O. 2008. Why current publication practices may distort science. PLoS Medicine 5:e201 DOI 10.1371/journal.pmed.0050201.