The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research

PeerJ

Volumn 2017, Issue 7, 2017, Pages

  Amrhein, Valentin ^a,b,c   Korner Nievergelt, Fränzi ^c,d   Roth, Tobias ^a,b  

^a UNIVERSITY OF BASEL   (Switzerland)

^b Research Station Petite Camargue Alsacienne   (France)

^c SWISS ORNITHOLOGICAL INSTITUTE   (Switzerland)

^d oikostat GmbH   (Switzerland)

Author keywords

Graded evidence; Nonsignificant; P value; Publication bias; Replicability; Reproducibility; Significant; Threshold; Truth inflation; Winner's curse

Indexed keywords

ALTERNATIVE HYPOTHESIS; CONFIDENCE INTERVAL; DECISION MAKING; DREDGING; EFFECT SIZE; LICENCE; NULL HYPOTHESIS; PUBLISHING; RELIABILITY; REPRODUCIBILITY; SAMPLE SIZE; STATISTICAL SIGNIFICANCE; STATISTICS;

EID: 85021855749     PISSN: None     EISSN: 21678359     Source Type: Journal    
DOI: 10.7717/peerj.3544     Document Type: Article

References (212)

1. Academy of Medical Sciences. 2015. Reproducibility and reliability of biomedical research: improving research practice. Symposium report. Academy of Medical Sciences, BBSRC, MRC, Wellcome Trust.
2. Anderson DR, Burnham KP, Thompson WL. 2000. Null hypothesis testing: problems, prevalence, and an alternative. Journal of Wildlife Management 64:912-923 DOI 10.2307/3803199.
3. Badenes-Ribera L, Frias-Navarro D, Iotti B, Bonilla-Campos A, Longobardi C. 2016. Misconceptions of the p-value among Chilean and Italian academic psychologists. Frontiers in Psychology 7:Article 1247 DOI 10.3389/fpsyg.2016.01247.
4. Baker M. 2016. Is there a reproducibility crisis? Nature 533:452-454 DOI 10.1038/533452a.
5. Barber JJ, Ogle K. 2014. To P or not to P? Ecology 95:621-626 DOI 10.1890/13-1402.1.
6. Bayarri MJ, Benjamin DJ, Berger JO, Sellke TM. 2016. Rejection odds and rejection ratios: a proposal for statistical practice in testing hypotheses. Journal of Mathematical Psychology 72:90-103 DOI 10.1016/j.jmp.2015.12.007.
7. Berger JO, Berry DA. 1988. Statistical analysis and the illusion of objectivity. American Scientist 76:159-165.
8. Berkson J. 1938. Some difficulties of interpretation encountered in the application of the chi-square test. Journal of the American Statistical Association 33:526-536 DOI 10.2307/2279690.
9. Berry DA. 2016. P-values are not what theyre cracked up to be. Supplemental material to the ASA statement on p-values and statistical significance. The American Statistician 70(2):129-133 DOI 10.1080/00031305.2016.1154108.
10. Bishop DVM, Thompson PA. 2016. Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ 4:e1715 DOI 10.7717/peerj.1715.
11. Boos DD, Stefanski LA. 2011. P-value precision and reproducibility. American Statisti- cian 65:213-221 DOI 10.1198/tas.2011.10129.
12. Boring EG. 1919. Mathematical vs. scientific significance. Psychological Bulletin 16:335-338 DOI 10.1037/h0074554.
13. Box GEP. 1980. Sampling and Bayes inference in scientific modeling and robustness. Journal of the Royal Statistical Society, Series A 143:383-430 DOI 10.2307/2982063.
14. Branch M. 2014. Malignant side effects of null-hypothesis significance testing. Theory & Psychology 24:256-277 DOI 10.1177/0959354314525282.
15. Brembs B, Button K, Munafo M. 2013. Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience 7:Article 291 DOI 10.3389/fnhum.2013.00291.
16. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafo MR. 2013. Confidence and precision increase with high statistical power. Nature Reviews Neuroscience 14:585 DOI 10.1038/nrn3475-c4.
17. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafo MR. 2013. Power failure: why small sample size undermines the reliabilityof neuroscience. Nature Reviews Neuroscience 14:365-376 DOI 10.1038/nrn3475.
18. Campbell JP. 1982. Editorial: some remarks from the outgoing editor. Journal of Applied Psychology 67:691-700 DOI 10.1037/h0077946.
19. Capen EC, Clapp RV, Campbell WM. 1971. Competitive bidding in high-risk situations. Journal of Petroleum Technology 23:641-653 DOI 10.2118/2993-PA.
20. Carver RP. 1978. Case against statistical significance testing. Harvard Educational Review 48:378-399 DOI 10.17763/haer.48.3.t490261645281841.
21. Chavalarias D, Wallach JD, Li AH, Ioannidis JP. 2016. Evolution of reporting P values in the biomedical literature, 1990-2015. JAMA-Journal of the American Medical Association 315:1141-1148 DOI 10.1001/jama.2016.1952.
22. Cohen J. 1962. The statistical power of abnormal-social psychological research: a review. Journal of Abnormal Psychology 65:145-153 DOI 10.1037/h0045186.
23. Cohen J. 1990. Things I have learned (so far). American Psychologist 45:1304-1312 DOI 10.1037//0003-066x.45.12.1304.
24. Cohen J. 1994. The earth is round (p < :05). American Psychologist 49:997-1003 DOI 10.1037/0003-066x.50.12.1103.
25. Colquhoun D. 2014. An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science 1:140216 DOI 10.1098/rsos.140216.
26. Coulson M, Healey M, Fidler F, Cumming G. 2010. Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Frontiers in Psychology 1:Article 26 DOI 10.3389/fpsyg.2010.00026.
27. Cowles M, Davis C. 1982a. Is the .05 level subjectively reasonable? Canadian Journal of Behavioural Science 14:248-252 DOI 10.1037/h0081256.
28. Cowles M, Davis C. 1982b. On the origins of the .05 level of statistical significance. American Psychologist 37:553-558 DOI 10.1037/0003-066x.37.5.553.
29. Csada RD, James PC, Espie RHM. 1996. The ''file drawer problem'' of non-significant results: does it apply to biological research? Oikos 76:591-593 DOI 10.2307/3546355.
30. Cumming G. 2007. Inference by eye: pictures of confidence intervals and thinking about levels of confidence. Teaching Statistics 29:89-93 DOI 10.1111/j.1467-9639.2007.00267.x.
31. Cumming G. 2008. Replication and p intervals. Perspectives on Psychological Science 3:286-300 DOI 10.1111/j.1745-6924.2008.00079.x.
32. Cumming G. 2012. Understanding the new statistics. New York: Routledge.
33. Cumming G. 2014. The new statistics: why and how. Psychological Science 25:7-29 DOI 10.1177/0956797613504966.
34. Delong JB, Lang K. 1992. Are all economic hypotheses false? Journal of Political Economy 100:1257-1272 DOI 10.1086/261860.
35. De Winter JCF, Dodou D. 2015. A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ 3:e733 DOI 10.7717/peerj.733.
36. Dienes Z. 2014. Using Bayes to get the most out of non-significant results. Frontiers in Psychology 5:Article 781 DOI 10.3389/fpsyg.2014.00781.
37. Dixon P. 2003. The p-value fallacy and how to avoid it. Canadian Journal of Experimental Psychology 57:189-202 DOI 10.1037/h0087425.
38. Drummond GB. 2016. Most of the time, P is an unreliable marker, so we need no exact cut-off. British Journal of Anaesthesia 116:893-893 DOI 10.1093/bja/aew146.
39. Dwan K, Gamble C, Williamson PR, Kirkham JJ, the Reporting Bias Group. 2013. Systematic review of the empirical evidence of study publication bias and outcome reporting bias-an updated review. PLOS ONE 8:e66844 DOI 10.1371/journal.pone.0066844.
40. Edgington ES. 1970. Hypothesis testing without fixed levels of significance. Journal of Psychology 76:109-115 DOI 10.1080/00223980.1970.9916827.
41. Edwards W, Lindman H, Savage LJ. 1963. Bayesian statistical inference for psychological research. Psychological Review 70:193-242 DOI 10.1037/h0044139.
42. Eysenck HJ. 1960. The concept of statistical significance and the controversy about onetailed tests. Psychological Review 67:269-271 DOI 10.1037/h0048412.
43. Fabrigar LR, Wegener DT. 2016. Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66:68-80 DOI 10.1016/j.jesp.2015.07.009.
44. Fanelli D. 2012. Negative results are disappearing from most disciplines and countries. Scientometrics 90:891-904 DOI 10.1007/s11192-011-0494-7.
45. Fanelli D, Costas R, Ioannidis JPA. 2017. Meta-assessment of bias in science. Proceedings of the National Academy of Sciences of the United States of America 114:3714-3719 DOI 10.1073/pnas.1618569114.
46. Ferguson CJ, Heene M. 2012. A vast graveyard of undead theories: publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7:555-561 DOI 10.1177/1745691612459059.
47. Fidler F, Burgman MA, Cumming G, Buttrose R, Thomason N. 2006. Impact of criticism of null-hypothesis significance testing on statistical reporting practices in conservation biology. Conservation Biology 20:1539-1544 DOI 10.1111/j.1523-1739.2006.00525.x.
48. Fidler F, Geoff C, Mark B, Neil T. 2004. Statistical reform in medicine, psychology and ecology. The Journal of Socio-Economics 33:615-630 DOI 10.1016/j.socec.2004.09.035.
49. Finch S, Cumming G, Thomason N. 2001. Reporting of statistical inference in the Journal of Applied Psychology: little evidence of reform. Educational and Psychological Measurement 61:181-210 DOI 10.1177/00131640121971167.
50. Fisher RA. 1922. On the mathematical foundations of theoretical statistics. Philo- sophical Transactions of the Royal Society of London, Series A 222:309-368 DOI 10.1098/rsta.1922.0009.
51. Fisher RA. 1925. Statistical methods for research workers. Edinburgh: Oliver and Boyd.
52. Fisher RA. 1926. The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain 33:503-513.
53. Fisher RA. 1929. The statistical method in psychical research. Proceedings of the Society for Psychical Research 39:189-192.
54. Fisher RA. 1935. Statistical tests. Nature 136:474-474.
55. Fisher RA. 1937. The design of experiments. 2nd edition. Edinburgh: Oliver and Boyd.
56. Fisher R. 1955. Statistical methods and scientific induction. Journal of the Royal Statistical Society Series B-Statistical Methodology 17:69-78.
57. Fisher RA. 1956. Statistical methods and scientific inference. Edinburgh: Oliver and Boyd.
58. Fisher RA. 1958. Statistical methods for research workers. 13th edition. Edinburgh: Oliver and Boyd.
59. Foreman P, Murnighan JK. 1996. Learning to avoid the winner's curse. Organizational Behavior and Human Decision Processes 67:170-180 DOI 10.1006/obhd.1996.0072.
60. Forstmeier W, Schielzeth H. 2011. Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse. Behavioral Ecology and Sociobiology 65:47-55 DOI 10.1007/s00265-010-1038-5.
61. Francis G. 2013. Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology 57:153-169 DOI 10.1016/j.jmp.2013.02.003.
62. Gallistel CR. 2009. The importance of proving the null. Psychological Review 116:439-453 DOI 10.1037/a0015251.
63. Garner C. 2007. Upward bias in odds ratio estimates from genome-wide association studies. Genetic Epidemiology 31:288-295 DOI 10.1002/gepi.20209.
64. Gaudart J, Huiart L, Milligan PJ, Thiebaut R, Giorgi R. 2014. Reproducibility issues in science, is P value really the only answer? Proceedings of the National Academy of Sciences of the United States of America 111:E1934-E1934 DOI 10.1073/pnas.1323051111.
65. Gelman A. 2013. Interrogating p-values. Journal of Mathematical Psychology 57:188-189 DOI 10.1016/j.jmp.2013.03.005.
66. Gelman A. 2013. The problem with p-values is how they're used. Available at http: //www.stat.columbia.edu/~gelman (accessed on 6 June 2017).
67. Gelman A. 2015. The connection between varying treatment effects and the crisis of unreplicable research: a Bayesian perspective. Journal of Management 41:632-643 DOI 10.1177/0149206314525208.
68. Gelman A. 2016. The problems with p-values are not just with p-values. Supplemental material to the ASA statement on p-values and statistical significance. The American Statistician 70(2):129-133 DOI 10.1080/00031305.2016.1154108.
69. Gelman A, Carlin J. 2014. Beyond power calculations: assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science 9:641-651 DOI 10.1177/1745691614551642.
70. Gelman A, Hennig C. 2017. Beyond subjective and objective in statistics. Journal of the Royal Statistical Society, Series A (Statistics in Society) In Press.
71. Gelman A, Loken E. 2014. The statistical crisis in science. American Scientist 102:460-465 DOI 10.1511/2014.111.460.
72. Gelman A, Robert CP. 2014. Revised evidence for statistical standards. Proceedings of the National Academy of Sciences of the United States of America 111:E1933-E1933 DOI 10.1073/pnas.1322995111.
73. Gelman A, Stern H. 2006. The difference between ''significant'' and ''not significant'' is not itself statistically significant. American Statistician 60:328-331 DOI 10.1198/000313006x152649.
74. Gerber AS, Malhotra N. 2008. Publication bias in empirical sociological research-Do arbitrary significance levels distort published results? Sociological Methods & Research 37:3-30 DOI 10.1177/0049124108318973.
75. Gigerenzer G. 1993. The superego, the ego, and the id in statistical reasoning. In: Hillsdale NJ, ed. A handbook for data analysis in the behavioral sciences. Hillsdale: Lawrence Erlbaum Associates, 311-339.
76. Gigerenzer G, Krauss S, Vitouch O. 2004. The null ritual: what you always wanted to know about significance testing but were afraid to ask. In: Kaplan D, ed. The Sage handbook of quantitative methodology for the social sciences. Thousand Oaks: Sage Publications, 391-408.
77. Gigerenzer G, Marewski JN. 2015. Surrogate science: the idol of a universal method for scientific inference. Journal of Management 41:421-440 DOI 10.1177/0149206314547522.
78. Gigerenzer G, Swijtink Z, Porter T, Daston L, Beatty J, Krüger L. 1989. The empire of chance: how probability changed science and everyday life. New York: Cambridge University Press.
79. Gill J. 1999. The insignificance of null hypothesis significance testing. Political Research Quarterly 52:647-674 DOI 10.2307/449153.
80. Giner-Sorolla R. 2016. Approaching a fair deal for significance and other concerns. Journal of Experimental Social Psychology 65:1-6 DOI 10.1016/j.cjesp.2016.01.010.
81. Goodman SN. 1992. A comment on replication, p-values and evidence. Statistics in Medicine 11:875-879 DOI 10.1002/sim.4780110705.
82. Goodman SN. 1993. P values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate. American Journal of Epidemiology 137:485-496 DOI 10.1093/oxfordjournals.aje.a116700.
83. Goodman SN. 2016. The next questions: who, what, when, where, and why? Supplemental material to the ASA statement on p-values and statistical significance. The American Statistician 70(2):129-133 DOI 10.1080/00031305.2016.1154108.
84. Goodman SN, Fanelli D, Ioannidis JPA. 2016. What does research reproducibility mean? Science Translational Medicine 8:341 DOI 10.1126/scitranslmed.aaf5027.
85. Gorard S. 2016. Damaging real lives through obstinacy: re-emphasising why significance testing is wrong. Sociological Research Online 21(1):2 DOI 10.5153/sro.3857.
86. Göring HHH, Terwilliger JD, Blangero J. 2001. Large upward bias in estimation of locus-specific effects from genomewide scans. American Journal of Human Genetics 69:1357-1369 DOI 10.1086/324471.
87. Greenland S. 2012. Nonsignificance plus high power does not imply support for the null over the alternative. Annals of Epidemiology 22:364-368 DOI 10.1016/j.annepidem.2012.02.007.
88. Greenland S, Poole C. 2013. Living with statistics in observational research. Epidemiology 24:73-78 DOI 10.1097/EDE.0b013e3182785a49.
89. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. 2016. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology 31:337-350 DOI 10.1007/s10654-016-0149-3.
90. Greenwald AG. 1975. Consequences of prejudice against the null hypothesis. Psychologi- cal Bulletin 82:1-19 DOI 10.1037/h0076157.
91. Greenwald AG, Gonzalez R, Harris RJ, Guthrie D. 1996. Effect sizes and p values: what should be reported and what should be replicated? Psychophysiology 33:175-183 DOI 10.1111/j.1469-8986.1996.tb02121.x.
92. Hagen RL. 1997. In praise of the null hypothesis statistical test. American Psychologist 52:15-24 DOI 10.1037/0003-066x.52.1.15.
93. Hagen RL. 1998. A further look at wrong reasons to abandon statistical testing. American Psychologist 53:801-803 DOI 10.1037/0003-066x.53.7.801.
94. Halsey LG, Curran-Everett D, Vowler SL, Drummond GB. 2015. The fickle P value generates irreproducible results. Nature Methods 12:179-185 DOI 10.1038/nmeth.3288.
95. Higginson AD, Munafo MR. 2016. Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLOS Biology 14:e2000995 DOI 10.1371/journal.pbio.2000995.
96. Higgs MD. 2013. Do we really need the s-word? American Scientist 101:6-9 DOI 10.1511/2013.100.6.
97. Hoekstra R, Finch S, Kiers HAL, Johnson A. 2006. Probability as certainty: dichotomous thinking and the misuse of p values. Psychonomic Bulletin & Review 13:1033-1037 DOI 10.3758/bf03213921.
98. Hoekstra R, Johnson A, Kiers HAL. 2012. Confidence intervals make a difference: effects of showing confidence intervals on inferential reasoning. Educational and Psychological Measurement 72:1039-1052 DOI 10.1177/0013164412450297.
99. Hubbard R, Bayarri MJ. 2003. Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing. American Statistician 57:171-178 DOI 10.1198/0003130031856.
100. Hung HMJ, Oneill RT, Bauer P, Kohne K. 1997. The behavior of the p-value when the alternative hypothesis is true. Biometrics 53:11-22 DOI 10.2307/2533093.
101. Hurlbert SH, Lombardi CM. 2009. Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici 46:311-349 DOI 10.5735/086.046.0501.
102. IntHout J, Ioannidis JPA, Borm GF. 2016. Obtaining evidence by a single well-powered trial or several modestly powered trials. Statistical Methods in Medical Research 25:538-552 DOI 10.1177/0962280212461098.
103. Ioannidis JPA. 2005. Contradicted and initially stronger effects in highly cited clinical research. JAMA-Journal of the American Medical Association 294:218-228 DOI 10.1001/jama.294.2.218.
104. Ioannidis JPA. 2008. Why most discovered true associations are inflated. Epidemiology 19:640-648 DOI 10.1097/EDE.0b013e31818131e7.
105. Ioannidis JPA. 2010. Meta-research: the art of getting it wrong. Research Synthesis Methods 1:169-184 DOI 10.1002/jrsm.19.
106. Ioannidis JPA. 2014. How to make more published research true. PLOS Medicine 11:e1001747 DOI 10.1371/journal.pmed.1001747.
107. Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. 2014. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383:166-175 DOI 10.1016/s0140-6736(13)62227-8.
108. Jennions MD, Møller AP. 2002. Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proceedings of the Royal Society B-Biological Sciences 269:43-48 DOI 10.1098/rspb.2001.1832.
109. Jennions MD, Møller AP. 2003. A survey of the statistical power of research in behavioral ecology and animal behavior. Behavioral Ecology 14:438-445 DOI 10.1093/beheco/14.3.438.
110. John LK, Loewenstein G, Prelec D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23:524-532 DOI 10.1177/0956797611430953.
111. Johnson DH. 1999. The insignificance of statistical significance testing. Journal of Wildlife Management 63:763-772 DOI 10.2307/3802789.
112. Johnson VE. 2013. Revised standards for statistical evidence. Proceedings of the Na- tional Academy of Sciences of the United States of America 110:19313-19317 DOI 10.1073/pnas.1313476110.
113. Johnson VE. 2014. Reply to Gelman, Gaudart, Pericchi: more reasons to revise standards for statistical evidence. Proceedings of the National Academy of Sciences of the United States of America 111:E1936-E1937 DOI 10.1073/pnas.1400338111.
114. Kelly CD. 2006. Replicating empirical research in behavioral ecology: how and why it should be done but rarely ever is. Quarterly Review of Biology 81:221-236 DOI 10.1086/506236.
115. Kline RB. 2013. Beyond significance testing: statistics reform in the behavioral sciences. Washington: American Psychological Association.
116. Korner-Nievergelt F, Hüppop O. 2016. Kurze Einführung in Bayes-Statistik mit R für Ornithologen. Vogelwarte 54:181-194.
117. Korner-Nievergelt F, Roth T, Felten S, Guélat J, Almasi B, Korner-Nievergelt P. 2015. Bayesian data analysis in ecology using linear models with R, BUGS, and Stan. London: Academic Press.
118. Krueger J. 2001. Null hypothesis significance testing-on the survival of a flawed method. American Psychologist 56:16-26 DOI 10.1037//0003-066x.56.1.16.
119. Labovitz S. 1968. Criteria for selecting a significance level: a note on the sacredness of .05. American Sociologist 3:220-222.
120. Lai J, Fidler F, Cumming G. 2012. Subjective p intervals-researchers underestimate the variability of p values over replication. Methodology 8:51-62 DOI 10.1027/1614-2241/a000037.
121. Lavine M. 2014. Comment on Murtaugh. Ecology 95:642-645 DOI 10.1890/13-1112.1.
122. Lazzeroni LC, Lu Y, Belitskaya-Levy I. 2014. P-values in genomics: apparent precision masks high uncertainty. Molecular Psychiatry 19:1336-1340 DOI 10.1038/mp.2013.184.
123. Lecoutre B, Poitevineau J. 2014. The significance test controversy revisited. Heidelberg: Springer.
124. Lecoutre M-P, Poitevineau J, Lecoutre B. 2003. Even statisticians are not immune to misinterpretations of null hypothesis significance tests. International Journal of Psychology 38:37-45 DOI 10.1080/00207590244000250.
125. Leek JT, Jager LR. 2017. Is most published research really false? Annual Review of Statis- tics and Its Application 4:109-122 DOI 10.1146/annurev-statistics-060116-054104.
126. Lehmann EL. 2011. Fisher, Neyman, and the creation of classical statistics. New York: Springer.
127. Lemoine NP, Hoffman A, Felton AJ, Baur L, Chaves F, Gray J, Yu Q, Smith MD. 2016. Underappreciated problems of low replication in ecological field studies. Ecology 97:2554-2561 DOI 10.1002/ecy.1506.
128. Lenhard J. 2006. Models and statistical inference: the controversy between Fisher and Neyman-Pearson. British Journal for the Philosophy of Science 57:69-91 DOI 10.1093/bjps/axi152.
129. Lertzman KP. 1995. Notes on writing papers and theses. Bulletin of the Ecological Society of America 76:86-90.
130. Levine TR, Weber R, Park HS, Hullett CR. 2008. A communication researchers' guide to null hypothesis significance testing and alternatives. Human Communication Research 34:188-209 DOI 10.1111/j.1468-2958.2008.00318.x.
131. Lew MJ. 2016. Three inferential questions, two types of p-value. Supplemental material to the ASA statement on p-values and statistical significance. The American Statistician 70(2):129-133 DOI 10.1080/00031305.2016.1154108.
132. Lisse JR, Perlman M, Johansson G, Shoemaker JR, Schechtman J, Skalky CS, Dixon ME, Polls AB, Mollen AJ, Geba GP. 2003. Gastrointestinal tolerability and effectiveness of rofecoxib versus naproxen in the treatment of osteoarthritis-A randomized, controlled trial. Annals of Internal Medicine 139:539-546 DOI 10.7326/0003-4819-139-7-200310070-00005.
133. Loftus GR. 1993. A picture is worth 1000 p-values: On the irrelevance of hypothesistesting in the microcomputer age. Behavior Research Methods Instruments & Computers 25:250-256 DOI 10.3758/bf03204506.
134. Lovasich JL, Neyman J, Scott EL, Wells MA. 1971. Hypothetical explanations of negative apparent effects of cloud seeding in whitetop experiment. Proceedings of the National Academy of Sciences of the United States of America 68:2643-2646 DOI 10.1073/pnas.68.11.2643.
135. Madden LV, Shah DA, Esker PD. 2015. Does the P value have a future in plant pathology? Phytopathology 105:1400-1407 DOI 10.1094/phyto-07-15-0165-le.
136. Maxwell SE, Lau MY, Howard GS. 2015. Is psychology suffering from a replication crisis? What does "failure to replicate'' really mean? American Psychologist 70:487-498 DOI 10.1037/a0039400.
137. McCarthy MA. 2007. Bayesian methods for ecology. Cambridge: Cambridge University Press.
138. McCormack J, Vandermeer B, Allan GM. 2013. How confidence intervals become confusion intervals. BMC Medical Research Methodology 13:134 DOI 10.1186/1471-2288-13-134.
139. McShane BB, Gal D. 2016. Blinding us to the obvious? The effect of statistical training on the evaluation of evidence. Management Science 62:1707-1718 DOI 10.1287/mnsc.2015.2212.
140. McShane BB, Gal D. 2017. Statistical significance and the dichotomization of evidence: the relevance of the ASA statement on statistical significance and p-values for statisticians. Journal of the American Statistical Association In Press.
141. Meehl PE. 1967. Theory-testing in psychology and physics: a methodological paradox. Philosophy of Science 34:103-115 DOI 10.1086/288135.
142. Meehl PE. 1990. Why summaries of research on psychological theories are often uninterpretable. Psychological Reports 66:195-244 DOI 10.2466/pr0.66.1.195-244.
143. Miller J. 2009. What is the probability of replicating a statistically significant effect? Psychonomic Bulletin & Review 16:617-640 DOI 10.3758/pbr.16.4.617.
144. Morey RD, Rouder JN. 2011. Bayes factor approaches for testing interval null hypotheses. Psychological Methods 16:406-419 DOI 10.1037/a0024377.
145. Munafo MR, Flint J. 2010. How reliable are scientific studies? British Journal of Psychiatry 197:257-258 DOI 10.1192/bjp.bp.109.069849.
146. Munafo MR, Stothart G, Flint J. 2009. Bias in genetic association studies and impact factor. Molecular Psychiatry 14:119-120 DOI 10.1038/mp.2008.77.
147. Mundry R. 2011. Issues in information theory-based statistical inference-a commentary from a frequentist's perspective. Behavioral Ecology and Sociobiology 65:57-68 DOI 10.1007/s00265-010-1040-y.
148. Murdoch DJ, Tsai Y-L, Adcock J. 2008. P-values are random variables. American Statistician 62:242-245 DOI 10.1198/000313008x332421.
149. Murtaugh PA. 2014. In defense of P values. Ecology 95:611-617 DOI 10.1890/13-0590.1.
150. Murtaugh PA. 2014. Rejoinder. Ecology 95:651-653 DOI 10.1890/13-1858.1.
151. Nakagawa S. 2004. A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral Ecology 15:1044-1045 DOI 10.1093/beheco/arh107.
152. Neyman J. 1977. Frequentist probability and frequentist statistics. Synthese 36:97-131 DOI 10.1007/bf00485695.
153. Neyman J, Pearson ES. 1933. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A 231:289-337 DOI 10.1098/rsta.1933.0009.
154. Neyman J, Pearson ES. 1933. The testing of statistical hypotheses in relation to probabilities a priori. Proceedings of the Cambridge Philosophical Society 29:492-510 DOI 10.1017/S030500410001152X.
155. Nosek BA, Errington TM. 2017. Making sense of replications. eLife 6:e23383 DOI 10.7554/eLife.23383.
156. Nuzzo R. 2015. Fooling ourselves. Nature 526:182-185 DOI 10.1038/526182a.
157. Oakes M. 1986. Statistical inference: commentary for the social and behavioural sciences. Chichester: Wiley.
158. Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349:aac4716 DOI 10.1126/science.aac4716.
159. Orlitzky M. 2012. How can significance tests be deinstitutionalized? Organizational Research Methods 15:199-228 DOI 10.1177/1094428111428356.
160. Parker TH, Forstmeier W, Koricheva J, Fidler F, Hadfield JD, Chee YE, Kelly CD, Gurevitch J, Nakagawa S. 2016. Transparency in ecology and evolution: real problems, real solutions. Trends in Ecology & Evolution 31:711-719 DOI 10.1016/j.tree.2016.07.002.
161. Patil P, Peng RD, Leek JT. 2016. What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science 11:539-544 DOI 10.1177/1745691616646366.
162. Pearson ES. 1962. Some thoughts on statistical inference. Annals of Mathematical Statistics 33:394-403 DOI 10.1214/aoms/1177704566.
163. Pericchi L, Pereira CAB, Perez M-E. 2014. Adaptive revised standards for statistical evidence. Proceedings of the National Academy of Sciences of the United States of America 111:E1935-E1935 DOI 10.1073/pnas.1322191111.
164. Poitevineau J, Lecoutre B. 2001. Interpretation of significance levels by psychological researchers: the .05 cliff effect may be overstated. Psychonomic Bulletin & Review 8:847-850 DOI 10.3758/bf03196227.
165. Pritschet L, Powell D, Horne Z. 2016. Marginally significant effects as evidence for hypotheses: changing attitudes over four decades. Psychological Science 27:1036-1042 DOI 10.1177/0956797616645672.
166. Reinhart A. 2015. Statistics done wrong. San Francisco: No Starch Press.
167. Rosenthal R. 1979. The ''file drawer problem'' and tolerance for null results. Psychological Bulletin 86:638-641 DOI 10.1037//0033-2909.86.3.638.
168. Rosnow RL, Rosenthal R. 1989. Statistical procedures and the justification of knowledge in psychological science. American Psychologist 44:1276-1284 DOI 10.1037//0003-066x.44.10.1276.
169. Rothman KJ. 2014. Six persistent research misconceptions. Journal of General Internal Medicine 29:1060-1064 DOI 10.1007/s11606-013-2755-z.
170. Rozeboom WW. 1960. The fallacy of the null-hypothesis significance test. Psychological Bulletin 57:416-428 DOI 10.1037/h0042040.
171. Sackrowitz H, Samuel-Cahn E. 1999. P values as random variables-expected P values. American Statistician 53:326-331 DOI 10.2307/2686051.
172. Salsburg D. 2001. The lady tasting tea: how statistics revolutionized science in the twentieth century. New York: Henry Holt & Co.
173. Sauley KS, Bedeian AG. 1989. .05: a case of the tail wagging the distribution. Journal of Management 15:335-344 DOI 10.1177/014920638901500209.
174. Savalei V, Dunn E. 2015. Is the call to abandon p-values the red herring of the replicability crisis? Frontiers in Psychology 6:Article 245 DOI 10.3389/fpsyg.2015.00245.
175. Savitz DA. 2013. Reconciling theory and practice-what is to be done with P values? Epidemiology 24:212-214 DOI 10.1097/EDE.0b013e318281e856.
176. Schatz P, Jay KA, McComb J, McLaughlin JR. 2005. Misuse of statistical tests in Archives of Clinical Neuropsychology publications. Archives of Clinical Neuropsychology 20:1053-1059 DOI 10.1016/j.acn.2005.06.006.
177. Schmidt FL. 1992. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist 47:1173-1181 DOI 10.1037/0003-066x.47.10.1173.
178. Schmidt FL. 1996. Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers. Psychological Methods 1:115-129 DOI 10.1037//1082-989x.1.2.115.
179. Schneider JW. 2015. Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations. Scientometrics 102:411-432 DOI 10.1007/s11192-014-1251-5.
180. Sedlmeier P, Gigerenzer G. 1989. Do studies of statistical power have an effect on the power of studies? Psychological Bulletin 105:309-316 DOI 10.1037//0033-2909.105.2.309.
181. Sellke T, Bayarri MJ, Berger JO. 2001. Calibration of p values for testing precise null hypotheses. American Statistician 55:62-71 DOI 10.1198/000313001300339950.
182. Senn S. 2002. A comment on replication, p-values and evidence. Statistics in Medicine 21:2437-2444 DOI 10.1002/sim.1072.
183. Sharpe D. 2013. Why the resistance to statistical innovations? Bridging the communication gap. Psychological Methods 18:572-582 DOI 10.1037/a0034177.
184. Simmons JP, Nelson LD, Simonsohn U. 2011. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:1359-1366 DOI 10.1177/0956797611417632.
185. Simonsohn U. 2014. Posterior-hacking: selective reporting invalidates Bayesian results also. Available at https:// papers.ssrn.com/ sol3/ papers.cfm?abstract_id=2374040 (accessed on 6 June 2017).
186. Simonsohn U. 2015. Small telescopes: detectability and the evaluation of replication results. Psychological Science 26:559-569 DOI 10.1177/0956797614567341.
187. Siontis KCM, Evangelou E, Ioannidis JPA. 2011. Magnitude of effects in clinical trials published in high-impact general medical journals. International Journal of Epidemiology 40:1280-1291 DOI 10.1093/ije/dyr095.
188. Skipper JK, Guenther AL, Nass G. 1967. The sacredness of .05: a note concerning the uses of statistical levels of significance in social science. American Sociologist 2:16-18.
189. Smaldino PE, McElreath R. 2016. The natural selection of bad science. Royal Society Open Science 3:Article 160384 DOI 10.1098/rsos.160384.
190. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. 2010. Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment 14(8):iii, ix-xi DOI 10.3310/hta14080.
191. Stahel WA. 2016. Statistical issues in reproducibility. In: Atmanspacher H, Maasen S, eds. Reproducibility: principles, problems, practices, and prospects. Hoboken: Wiley, 87-114.
192. Stanley DJ, Spence JR. 2014. Expectations for replications: are yours realistic? Perspectives on Psychological Science 9:305-318 DOI 10.1177/1745691614528518.
193. Sterling TD. 1959. Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Associa- tion 54:30-34 DOI 10.2307/2282137.
194. Sterne JAC, Smith GD. 2001. Sifting the evidence-what's wrong with significance tests? British Medical Journal 322:226-231 DOI 10.1136/bmj.322.7280.226.
195. Stoehr AM. 1999. Are significance thresholds appropriate for the study of animal behaviour? Animal Behaviour 57:F22-F25 DOI 10.1006/anbe.1998.1016.
196. Thompson B. 1998. In praise of brilliance: where that praise really belongs. American Psychologist 53:799-800 DOI 10.1037//0003-066x.53.7.799.
197. Thompson B. 1999. Why ''encouraging'' effect size reporting is not working: the etiology of researcher resistance to changing practices. Journal of Psychology 133:133-140 DOI 10.1080/00223989909599728.
198. Trafimow D, Marks M. 2015. Editorial. Basic and Applied Social Psychology 37:1-2 DOI 10.1080/01973533.2015.1012991.
199. Tryon WW. 2001. Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: an integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods 6:371-386 DOI 10.1037//1082-989x.6.4.371.
200. Tukey JW. 1991. The philosophy of multiple comparisons. Statistical Science 6:100-116 DOI 10.1214/ss/1177011945.
201. Van Assen MALM, Van Aert RCM, Nuijten MB, Wicherts JM. 2014. Why publishing everything is more effective than selective publishing of statistically significant results. PLOS ONE 9:e84896 DOI 10.1371/journal.pone.0084896.
202. Van Helden J. 2016. Confidence intervals are no salvation from the alleged fickleness of the P value. Nature Methods 13:605-606 DOI 10.1038/nmeth.3932.
203. Vankov I, Bowers J, Munafo MR. 2014. On the persistence of low power in psychological science. Quarterly Journal of Experimental Psychology 67:1037-1040 DOI 10.1080/17470218.2014.885986.
204. Wasserstein RL, Lazar NA. 2016. The ASA's statement on p-values: context, process, and purpose. The American Statistician 70:129-133 DOI 10.1080/00031305.2016.1154108.
205. Weinberg CR. 2001. It's time to rehabilitate the p-value. Epidemiology 12:288-290 DOI 10.1097/00001648-200105000-00004.
206. Weiss KM. 2011. The 5% solution-how do we make decisions in science? Evolutionary Anthropology 20:81-84 DOI 10.1002/evan20304.
207. Whittingham MJ, Stephens PA, Bradbury RB, Freckleton RP. 2006. Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology 75:1182-1189 DOI 10.1111/j.1365-2656.2006.01141.x.
208. Wolf IS. 1961. Perspectives in psychology-XVI. Negative findings. Psychological Record 11:91-95.
209. Young NS, Ioannidis JPA, Al-Ubaydi O. 2008. Why current publication practices may distort science. PLOS Medicine 5:e201 DOI 10.1371/journal.pmed.0050201.
210. Yu EC, Sprenger AM, Thomas RP, Dougherty MR. 2014. When decision heuristics and science collide. Psychonomic Bulletin & Review 21:268-282 DOI 10.3758/s13423-013-0495-z.
211. Ziliak ST, McCloskey DN. 2008. The cult of statistical significance: how the standard error costs us jobs, justice, and lives. Ann Arbor: University of Michigan Press.
212. Zöllner S, Pritchard JK. 2007. Overcoming the winner's curse: estimating penetrance parameters from case-control data. American Journal of Human Genetics 80:605-615 DOI 10.1086/512821.