Detecting and avoiding likely false-positive findings – a practical guide

Biological Reviews

Volumn 92, Issue 4, 2017, Pages 1941-1968

  Forstmeier, Wolfgang ^a   Wagenmakers, Eric Jan ^b   Parker, Timothy H ^c  

^a MAX PLANCK INSTITUTE FOR ORNITHOLOGY   (Germany)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

^c WHITMAN COLLEGE   (United States)

Author keywords

confirmation bias; HARKing; hindsight bias; overfitting; P hacking; power; preregistration; replication; researcher degrees of freedom; Type I error

Indexed keywords

BIOLOGICAL MODEL; FALSE POSITIVE RESULT; HUMAN; INFORMATION PROCESSING; MEDICAL RESEARCH; PROCEDURES; STANDARDS;

BIOMEDICAL RESEARCH; DATA COLLECTION; FALSE POSITIVE REACTIONS; HUMANS; MODELS, BIOLOGICAL;

EID: 85006002074     PISSN: 14647931     EISSN: 1469185X     Source Type: Journal    
DOI: 10.1111/brv.12315     Document Type: Article

References (118)

1. Anderson, M. S., Martinson, B. C. & De Vries, R. (2007). Normative dissonance in science: results from a national survey of US scientists. Journal of Empirical Research on Human Research Ethics 2, 3–14.
2. Armitage, P., McPherson, C. K. & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society, Series A (General) 132, 235–244.
3. Baker, M. (2016). Is there a reproducibility crisis? Nature 533, 452–454.
4. Barber, T. X. (1976). Pitfalls in Human Research: Ten Pivotal Points. Pergamon Press Inc., New York.
5. Barnett, A. G., van der Pols, J. C. & Dobson, A. J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology 34, 215–220.
6. Barto, E. K. & Rillig, M. C. (2012). Dissemination biases in ecology: effect sizes matter more than quality. Oikos 121, 228–235.
7. Bates, D., Maechler, M., Bolker, B. & Walker, S. (2014). lme4: linear mixed-effects models using Eigen and S4. R package, version 1.1-7.
8. Bates, D. & Vazquez, A. I. (2014). pedigreemm: pedigree-based mixed-effects models. R package, version 0.3-3, http://CRAN.R-project.org/package=pedigreemm Accessed 15.4.2015.
9. Begley, C. G. & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature 483, 531–533.
10. Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate – a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological) 57, 289–300.
11. Bishop, D. V. M. & Thompson, P. A. (2016). Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ 4, e1715.
12. Bissell, M. (2013). The risks of the replication drive. Nature 503, 333–334.
13. Bruns, S. B. & Ioannidis, J. P. A. (2016). P-Curve and p-Hacking in observational research. PLoS One 11, e0149144.
14. Burley, N. & Bartels, P. J. (1990). Phenotypic similarities of sibling zebra finches. Animal Behaviour 39, 174–180.
15. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J. & Munafo, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews. Neuroscience 14, 365–376.
16. Chambers, C. D. (2013). Registered reports: a new publishing initiative at cortex. Cortex 49, 609–610.
17. Cliff, A. D. & Ord, J. K. (1981). Spatial Processes: Models & Applications. Pion, London.
18. Crawley, M. J. (2002). Statistical Computing. An Introduction to Data Analysis using S-Plus. Wiley, Chichester.
19. Cumming, G., Williams, J. & Fidler, F. (2004). Replication and researchers' understanding of confidence intervals and standard error bars. Understanding Statistics 3, 299–311.
20. Dale, M. R. T. & Fortin, M.-J. (2014). Spatial Analysis: A Guide for Ecologists. Cambridge University Press, Cambridge, UK.
21. Danchin, E., Wajnberg, E. & Wagner, R. H. (2014). Avoiding pitfalls in estimating heritability with the common options approach. Scientific Reports 4, 3974.
22. De Groot, A. D. (1956/2014). The meaning of “significance” for different types of research. Translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas. Acta Psychologica 148, 188–194.
23. Dienes, Z. (2016). How Bayes factor change scientific practice. Journal of Mathematical Psychology 72, 78–89.
24. Drotar, D. (2010). Editorial: a call for replications of research in Pediatric Psychology and guidance for authors. Journal of Pediatric Psychology 35, 801–805.
25. Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association 56, 52–64.
26. Egger, M., Smith, G. D., Schneider, M. & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 315, 629–634.
27. Etz, A. & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: psychology. PLoS One 11, e0149794.
28. Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS One 5, e10068.
29. Faul, F., Erdfelder, E., Buchner, A. & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behavior Research Methods 41, 1149–1160.
30. Felsenstein, J. (1985). Phylogenies and the comparative method. The American Naturalist 125, 1–15.
31. Ferguson, C. J. & Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7, 555–561.
32. Feynman, R. P. (1974). Cargo cult science. Engineering and Science 37, 10–13.
33. Field, A. (2005). Discovering Statistics Using SPSS. Sage, London.
34. Fischhoff, B. (1975). Hindsight not equal to foresight – effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance 1, 288–299.
35. Fisher, R. A. (1925). Statistical Methods for Research Workers. Genesis Publishing Pvt Ltd, Guildford, UK.
36. Forstmeier, W., Mueller, J. C. & Kempenaers, B. (2010). A polymorphism in the oestrogen receptor gene explains covariance between digit ratio and mating behaviour. Proceedings of the Royal Society of London B: Biological Sciences 277, 3353–3361.
37. Forstmeier, W. & Schielzeth, H. (2011). Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse. Behavioral Ecology and Sociobiology 65, 47–55.
38. Franco, A., Malhotra, N. & Simonovits, G. (2014). Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505.
39. Freckleton, R. P., Harvey, P. H. & Pagel, M. (2002). Phylogenetic analysis and comparative data: a test and review of evidence. The American Naturalist 160, 712–726.
40. Gelman, A. & Loken, E. (2014). The statistical crisis in science. The American Scientist 102, 460–465.
41. Gelman, A. & O'Rourke, K. (2014). Discussion: difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics 15, 18–23.
42. Gelman, A. & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician 60, 328–331.
43. Gelman, A. & Weakliem, D. (2009). Of beauty, sex and power: statistical challenges in estimating small effects. The American Scientist 97, 310–316.
44. Gintis, H., Smith, E. A. & Bowles, S. (2001). Costly signaling and cooperation. Journal of Theoretical Biology 213, 103–119.
45. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin 82, 1–20.
46. Hadfield, J. (2015). There's Madness in our Methods: improving inference in ecology and evolution. Methods.blog, https://methodsblog.wordpress.com/2015/11/26/madness-in-our-methods Accessed 3.1.2016.
47. Hadfield, J. D., Wilson, A. J., Garant, D., Sheldon, B. C. & Kruuk, L. E. B. (2010). The misuse of BLUP in ecology and evolution. The American Naturalist 175, 116–125.
48. Haselton, M. G., Nettle, D. & Murray, D. R. (2005). The evolution of cognitive bias. In The Handbook of Evolutionary Psychology (ed. D. M. Buss), pp. 968–987. Wiley, New York.
49. Hereford, J., Hansen, T. F. & Houle, D. (2004). Comparing strengths of directional selection: how strong is strong? Evolution 58, 2133–2143.
50. Holman, L., Head, M. L., Lanfear, R. & Jennions, M. D. (2015). Evidence of experimental bias in the Life Sciences: why we need blind data recording. PLoS Biology 13, e1002190.
51. Holman, C., Piper, S. K., Grittner, U., Diamantaras, A. A., Kimmelman, J., Siegerink, B. & Dirnagl, U. (2016). Where have all the rodents gone? The effects of attrition in experimental research on cancer and stroke. PLoS Biology 14, e1002331.
52. Horton, R. (2015). Offline: What is medicine's 5 sigma? The Lancet 385, 1380.
53. Hurlbert, S. H. (1984). Pseudoreplication and the design of ecological field experiments. Ecological Monographs 54, 187–211.
54. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine 2, 696–701.
55. Ioannidis, J. P. A. & Trikalinos, T. A. (2007). The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. Canadian Medical Association Journal 176, 1091–1096.
56. Jager, L. R. & Leek, J. T. (2014). An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15, 1–12.
57. Jennions, M. D. & Møller, A. P. (2002). Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proceedings of the Royal Society of London B: Biological Sciences 269, 43–48.
58. Jennions, M. D. & Møller, A. P. (2003). A survey of the statistical power of research in behavioral ecology and animal behavior. Behavioral Ecology 14, 438–445.
59. John, L. K., Loewenstein, G. & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23, 524–532.
60. Jung, K., Shavitt, S., Viswanathan, M. & Hilbe, J. M. (2014). Female hurricanes are deadlier than male hurricanes. Proceedings of the National Academy of Sciences of the United States of America 111, 8782–8787.
61. Kardish, M. R., Mueller, U. G., Amador-Vargas, S., Dietrich, E. I., Ma, R., Barrett, B. & Fang, C.-C. (2015). Blind trust in unblinded observation in ecology, evolution and behavior. Frontiers in Ecology and Evolution 3, 51.
62. Kelly, C. D. (2006). Replicating empirical research in behavioral ecology: how and why it should be done but rarely ever is. The Quarterly Review of Biology 81, 221–236.
63. Kelly, C. & Price, T. D. (2005). Correcting for regression to the mean in behavior and ecology. The American Naturalist 166, 700–707.
64. Kerr, N. L. (1998). HARKing: hypothesizing after the results are known. Personality and Social Psychology Review 2, 196–217.
65. Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S. & Nosek, B. A. (2016). Badges to acknowledge open practices: a simple, low cost, effective method for increasing transparency. PLoS Biology 14, e1002456.
66. Koricheva, J., Jennions, M. D. & Lau, J. (2013). Temporal trends in effect sizes: causes, detection, & implications. In Handbook of Meta-analysis in Ecology and Evolution (eds J. Gurevitch, J. Koricheva and K. Mengersen), pp. 237–254. Princeton University Press, Princeton.
67. Lakens, D. & Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability: practical recommendations to increase the informational value of studies. Perspectives on Psychological Science 9, 278–292.
68. Levine, T., Asada, K. J. & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: evidence and implications of a publication bias against nonsignificant findings. Communication Monographs 76, 286–302.
69. MacCoun, R. & Perlmutter, S. (2015). Hide results to seek the truth. Nature 526, 187–189.
70. MacCoun, R. & Perlmutter, S. (0000). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions (eds S. O. Lilienfeld and I. Waldman), pp. 589–612. Wiley, New York.
71. Markowetz, F. (2015). Five selfish reasons to work reproducibly. Genome Biology 16, 1.
72. McNutt, M. (2014). Reproducibility. Science 343, 231–231.
73. Milinski, M. (1997). How to avoid seven deadly sins in the study of behavior. Advances in the Study of Behavior 26, 159–180.
74. Møller, A. P. & Jennions, M. D. (2002). How much variance can be explained by ecologists and evolutionary biologists? Oecologia 132, 492–500.
75. Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S., Morey, C. C., Newman, D. P., Schonbrodt, F. D., Vanpaemel, W., Wagenmakers, E.-J. & Zwaan, R. A. (2016). The peer reviewers' openness initiative: incentivizing open research practices through peer review. Royal Society Open Science 3, 150547.
76. Mundry, R. (2011). Issues in information theory-based statistical inference – a commentary from a frequentist's perspective. Behavioral Ecology and Sociobiology 65, 57–68.
77. Mundry, R. & Nunn, C. L. (2009). Stepwise model fitting and statistical inference: turning noise into signal pollution. The American Naturalist 173, 119–123.
78. Nakagawa, S. (2004). A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral Ecology 15, 1044–1045.
79. Nakagawa, S. & Parker, T. H. (2015). Replicating research in ecology and evolution: feasibility, incentives, and the cost-benefit conundrum. BMC Biology 13, 88.
80. Neuhäuser, M., Forstmeier, W. & Bretz, F. (2001). The distribution of extra-pair young within and among broods – a technique to calculate deviations from randomness. Journal of Avian Biology 32, 358–363.
81. Nickerson, R. S. (1998). Confirmation bias: a ubiquitous phenomenon in many guises. Review of General Psychology 2, 175–220.
82. Nosek, B. A., Spies, J. R. & Motyl, M. (2012). Scientific utopia: II. restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7, 615–631.
83. Nuzzo, R. (2014). Statistical errors. Nature 506, 150–152.
84. Nuzzo, R. (2015). Foolong ourselves. Nature 526, 182–185.
85. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349, aac4716.
86. Palmer, A. R. (2000). Quasireplication and the contract of error: lessons from sex ratios, heritabilities and fluctuating asymmetry. Annual Review of Ecology and Systematics 31, 441–480.
87. Parker, T. H. (2013). What do we really know about the signalling role of plumage colour in blue tits? A case study of impediments to progress in evolutionary biology. Biological Reviews 88, 511–536.
88. Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J. & Nakagawa, S. (2016). Transparency in ecology and evolution: real problems, real solutions. Trends in Ecology & Evolution 31, 711–719.
89. Pereira, T. V. & Ioannidis, J. P. A. (2011). Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology 64, 1060–1069.
90. Pike, N. (2011). Using false discovery rates for multiple comparisons in ecology and evolution. Methods in Ecology and Evolution 2, 278–282.
91. Pillemer, D. & Light, R. (1984). Summing up: the science of reviewing research. Harvard University Press, Cambridge.
92. Price, A. L., Patterson, N. J., Plenge, R. M., Weinblatt, M. E., Shadick, N. A. & Reich, D. (2006). Principal components analysis corrects for stratification in genome-wide association studies. Nature Genetics 38, 904–909.
93. Prinz, F., Schlange, T. & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews. Drug Discovery 10, 712.
94. Prum, R. O. (2010). The Lande-Kirkpatrick mechanism is the null model of evolution by intersexual selection: implications for meaning, honesty, and design in intersexual signals. Evolution 64, 3085–3100.
95. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin 86, 638–641.
96. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D. & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review 16, 225–237.
97. Ruxton, G. & Colegrave, N. (2010). Experimental Design for the Life Sciences. Oxford University Press, Oxford, UK.
98. Scargle, J. D. (1999). Publication bias (the “File-Drawer Problem”) in scientific inference. arXiv preprint physics/9909033.
99. Schielzeth, H. & Forstmeier, W. (2009). Conclusions beyond support: overconfident estimates in mixed models. Behavioral Ecology 20, 416–420.
100. Seguin, A. & Forstmeier, W. (2012). No band color effects on male courtship rate or body mass in the zebra finch: four experiments and a meta-analysis. PLoS One 7, e37785.
101. Sheldon, B. C. (2000). Differential allocation: tests, mechanisms and implications. Trends in Ecology & Evolution 15, 397–402.
102. Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22, 1359–1366.
103. Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2012). A 21 word solution. Available at http://ssrn.com/abstract=2160588 Accessed 4.3.2014.
104. Simonsohn, U., Nelson, L. D. & Simmons, J. P. (2014). P-curve: a key to the file-drawer. Journal of Experimental Psychology: General 143, 534–547.
105. Simonsohn, U., Simmons, J. P. & Nelson, L. D. (2015). Specification curve: descriptive and inferential statistics on all reasonable specifications. Manuscript available at http://ssrn.com/abstract=2694998 Accessed 30.10.2015.
106. Smaldino, P. E. & McElreath, R. (2016). The natural selection of bad science. arXiv preprint arXiv:1605.09511.
107. Smith, D. R., Hardy, I. C. W. & Gammell, M. P. (2011). Power rangers: no improvement in the statistical power of analyses published in Animal Behaviour. Animal Behaviour 81, 347–352.
108. Steegen, S., Tuerlinckx, F., Gelman, A. & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11, 702–712.
109. Trivers, R. (2011). The Folly of Fools. Basic, New York.
110. Valcu, M. & Kempenaers, B. (2010). Spatial autocorrelation: an overlooked concept in behavioral ecology. Behavioral Ecology 21, 902–905.
111. Valcu, M. & Valcu, C. M. (2011). Data transformation practices in biomedical sciences. Nature Methods 8, 104–105.
112. van Wilgenburg, E. & Elgar, M. A. (2013). Confirmation bias in studies of nestmate recognition: a cautionary note for research into the behaviour of animals. PLoS One 8, e53548.
113. Vazquez, A. I., Bates, D. M., Rosa, G. J. M., Gianola, D. & Weigel, K. A. (2010). Technical note: an R package for fitting generalized linear mixed models in animal breeding. Journal of Animal Science 88, 497–504.
114. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J. & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science 7, 632–638.
115. Ware, J. J. & Munafo, M. R. (2015). Significance chasing in research practice: causes, consequences and possible solutions. Addiction 110, 4–8.
116. Wedderburn, R. W. M. (1974). Quasi-likelihood functions, generalized linear models, and the Gauss-Newton method. Biometrika 61, 439–447.
117. Weissgerber, T. L., Garovic, V. D., Milin-Lazovic, J. S., Winham, S. J., Obradovic, Z., Trzeciakowski, J. P. & Milic, N. M. (2016). Reinventing biostatistics education for basic scientists. PLoS Biology 14, e1002430.
118. Whittingham, M. J., Stephens, P. A., Bradbury, R. B. & Freckleton, R. P. (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology 75, 1182–1189.