Power failure: Why small sample size undermines the reliability of neuroscience

Nature Reviews Neuroscience

Volumn 14, Issue 5, 2013, Pages 365-376

  Button, Katherine S ^a   Ioannidis, John P A ^b   Mokrysz, Claire ^a   Nosek, Brian A ^c   Flint, Jonathan ^d   Robinson, Emma S J ^a   Munafò, Marcus R ^a  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c UNIVERSITY OF VIRGINIA   (United States)

^d UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; MATHEMATICAL ANALYSIS; MEDICAL ETHICS; MEDICAL LITERATURE; MEDICAL RESEARCH; NEUROIMAGING; NEUROSCIENCE; NONHUMAN; PRIORITY JOURNAL; REPRODUCIBILITY; STATISTICS;

HUMANS; NEUROSCIENCES; PROBABILITY; REPRODUCIBILITY OF RESULTS; SAMPLE SIZE;

EID: 84876665206     PISSN: 1471003X     EISSN: 14710048     Source Type: Journal    
DOI: 10.1038/nrn3475     Document Type: Article

References (103)

1. Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005).
2. Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891-904 (2012).
3. Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1-20 (1975).
4. Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631 (2012).
5. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366 (2011).
6. Sullivan, P. F. Spurious genetic associations. Biol. Psychiatry 61, 1121-1126 (2007).
7. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531-533 (2012).
8. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discov. 10, 712 (2011).
9. Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855-3859 (2011).
10. Munafo, M. R., Stothart, G. & Flint, J. Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119-120 (2009).
11. Sterne, J. A. & Davey Smith, G. Sifting the evidence - what's wrong with significance tests? BMJ 322, 226-231 (2001).
12. Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450-456 (2011).
13. Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19, 640-648 (2008).
14. Tversky, A. & Kahneman, D. Belief in the law of small numbers. Psychol. Bull. 75, 105-110 (1971).
15. Masicampo, E. J. & Lalande, D. R. A peculiar prevalence of p values just below.05. Q. J. Exp. Psychol. 65, 2271-2279 (2012).
16. Carp, J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage 63, 289-300 (2012).
17. Dwan, K. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081 (2008).
18. Sterne, J. A. et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 343, d4002 (2011).
19. Joy-Gaba, J. A. & Nosek, B. A. The surprisingly limited malleability of implicit racial evaluations. Soc. Psychol. 41, 137-146 (2010).
20. Schmidt, K. & Nosek, B. A. Implicit (and explicit) racial attitudes barely changed during Barack Obama's presidential campaign and early presidency. J. Exp. Soc. Psychol. 46, 308-314 (2010).
21. Evangelou, E., Siontis, K. C., Pfeiffer, T. & Ioannidis, J. P. Perceived information gain from randomized trials correlates with publication in high-impact factor journals. J. Clin. Epidemiol. 65, 1274-1281 (2012).
22. Pereira, T. V. & Ioannidis, J. P. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. J. Clin. Epidemiol. 64, 1060-1069 (2011).
23. Faul, F., Erdfelder, E., Lang, A. G. & Buchner, A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175-191 (2007).
24. Babbage, D. R. et al. Meta-analysis of facial affect recognition difficulties after traumatic brain injury. Neuropsychology 25, 277-285 (2011).
25. Bai, H. Meta-analysis of 5, 10-methylenetetrahydrofolate reductase gene poymorphism as a risk factor for ischemic cerebrovascular disease in a Chinese Han population. Neural Regen. Res. 6, 277-285 (2011).
26. Bjorkhem-Bergman, L., Asplund, A. B. & Lindh, J. D. Metformin for weight reduction in non-diabetic patients on antipsychotic drugs: a systematic review and meta-analysis. J. Psychopharmacol. 25, 299-305 (2011).
27. Bucossi, S. et al. Copper in Alzheimer's disease: a meta-analysis of serum, plasma, and cerebrospinal fluid studies. J. Alzheimers Dis. 24, 175-185 (2011).
28. Chamberlain, S. R. et al. Translational approaches to frontostriatal dysfunction in attention-deficit/hyperactivity disorder using a computerized neuropsychological battery. Biol. Psychiatry 69, 1192-1203 (2011).
29. Chang, W. P., Arfken, C. L., Sangal, M. P. & Boutros, N. N. Probing the relative contribution of the first and second responses to sensory gating indices: a meta-analysis. Psychophysiology 48, 980-992 (2011).
30. Chang, X. L. et al. Functional parkin promoter polymorphism in Parkinson's disease: new data and meta-analysis. J. Neurol. Sci. 302, 68-71 (2011).
31. Chen, C. et al. Allergy and risk of glioma: a meta-analysis. Eur. J. Neurol. 18, 387-395 (2011).
32. Chung, A. K. & Chua, S. E. Effects on prolongation of Bazett's corrected QT interval of seven second-generation antipsychotics in the treatment of schizophrenia: a meta-analysis. J. Psychopharmacol. 25, 646-666 (2011).
33. Domellof, E., Johansson, A. M. & Ronnqvist, L. Handedness in preterm born children: a systematic review and a meta-analysis. Neuropsychologia 49, 2299-2310 (2011).
34. Etminan, N., Vergouwen, M. D., Ilodigwe, D. & Macdonald, R. L. Effect of pharmaceutical treatment on vasospasm, delayed cerebral ischemia, and clinical outcome in patients with aneurysmal subarachnoid hemorrhage: a systematic review and meta-analysis. J. Cereb. Blood Flow Metab. 31, 1443-1451 (2011).
35. Feng, X. L. et al. Association of FK506 binding protein 5 (FKBP5) gene rs4713916 polymorphism with mood disorders: a meta-analysis. Acta Neuropsychiatr. 23, 12-19 (2011).
36. Green, M. J., Matheson, S. L., Shepherd, A., Weickert, C. S. & Carr, V. J. Brain-derived neurotrophic factor levels in schizophrenia: a systematic review with meta-analysis. Mol. Psychiatry 16, 960-972 (2011).
37. Han, X. M., Wang, C. H., Sima, X. & Liu, S. Y. Interleukin-6-74G/C polymorphism and the risk of Alzheimer's disease in Caucasians: a meta-analysis. Neurosci. Lett. 504, 4-8 (2011).
38. Hannestad, J., DellaGioia, N. & Bloch, M. The effect of antidepressant medication treatment on serum levels of inflammatory cytokines: a meta-analysis. Neuropsychopharmacology 36, 2452-2459 (2011).
39. Hua, Y., Zhao, H., Kong, Y. & Ye, M. Association between the MTHFR gene and Alzheimer's disease: a meta-analysis. Int. J. Neurosci. 121, 462-471 (2011).
40. Lindson, N. & Aveyard, P. An updated meta-analysis of nicotine preloading for smoking cessation: investigating mediators of the effect. Psychopharmacology 214, 579-592 (2011).
41. Liu, H. et al. Association of 5-HTT gene polymorphisms with migraine: a systematic review and meta-analysis. J. Neurol. Sci. 305, 57-66 (2011).
42. Liu, J. et al. PITX3 gene polymorphism is associated with Parkinson's disease in Chinese population. Brain Res. 1392, 116-120 (2011).
43. MacKillop, J. et al. Delayed reward discounting and addictive behavior: a meta-analysis. Psychopharmacology 216, 305-321 (2011).
44. Maneeton, N., Maneeton, B., Srisurapanont, M. & Martin, S. D. Bupropion for adults with attention-deficit hyperactivity disorder: meta-analysis of randomized, placebo-controlled trials. Psychiatry Clin. Neurosci. 65, 611-617 (2011).
45. Ohi, K. et al. The SIGMAR1 gene is associated with a risk of schizophrenia and activation of the prefrontal cortex. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 1309-1315 (2011).
46. Olabi, B. et al. Are there progressive brain changes in schizophrenia? A meta-analysis of structural magnetic resonance imaging studies. Biol. Psychiatry 70, 88-96 (2011).
47. Oldershaw, A. et al. The socio-emotional processing stream in Anorexia Nervosa. Neurosci. Biobehav. Rev. 35, 970-988 (2011).
48. Oliver, B. J., Kohli, E. & Kasper, L. H. Interferon therapy in relapsing-remitting multiple sclerosis: a systematic review and meta-analysis of the comparative trials. J. Neurol. Sci. 302, 96-105 (2011).
49. Peerbooms, O. L. et al. Meta-analysis of MTHFR gene variants in schizophrenia, bipolar disorder and unipolar depressive disorder: evidence for a common genetic vulnerability? Brain Behav. Immun. 25, 1530-1543 (2011).
50. Pizzagalli, D. A. Frontocingulate dysfunction in depression: toward biomarkers of treatment response. Neuropsychopharmacology 36, 183-206 (2011).
51. Rist, P. M., Diener, H. C., Kurth, T. & Schurks, M. Migraine, migraine aura, and cervical artery dissection: a systematic review and meta-analysis. Cephalalgia 31, 886-896 (2011).
52. Sexton, C. E., Kalu, U. G., Filippini, N., Mackay, C. E. & Ebmeier, K. P. A meta-analysis of diffusion tensor imaging in mild cognitive impairment and Alzheimer's disease. Neurobiol. Aging 32, 2322.e5-2322.e18 (2011).
53. Shum, D., Levin, H. & Chan, R. C. Prospective memory in patients with closed head injury: a review. Neuropsychologia 49, 2156-2165 (2011).
54. Sim, H. et al. Acupuncture for carpal tunnel syndrome: a systematic review of randomized controlled trials. J. Pain 12, 307-314 (2011).
55. Song, F. et al. Meta-analysis of plasma amyloid-β levels in Alzheimer's disease. J. Alzheimers Dis. 26, 365-375 (2011).
56. Sun, Q. L. et al. Correlation of E-selectin gene polymorphisms with risk of ischemic stroke A meta-analysis. Neural Regen. Res. 6, 1731-1735 (2011).
57. Tian, Y., Kang, L. G., Wang, H. Y. & Liu, Z. Y. Meta-analysis of transcranial magnetic stimulation to treat post-stroke dysfunction. Neural Regen. Res. 6, 1736-1741 (2011).
58. Trzesniak, C. et al. Adhesio interthalamica alterations in schizophrenia spectrum disorders: a systematic review and meta-analysis. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 877-886 (2011).
59. Veehof, M. M., Oskam, M. J., Schreurs, K. M. & Bohlmeijer, E. T. Acceptance-based interventions for the treatment of chronic pain: a systematic review and meta-analysis. Pain 152, 533-542 (2011).
60. Vergouwen, M. D., Etminan, N., Ilodigwe, D. & Macdonald, R. L. Lower incidence of cerebral infarction correlates with improved functional outcome after aneurysmal subarachnoid hemorrhage. J. Cereb. Blood Flow Metab. 31, 1545-1553 (2011).
61. Vieta, E. et al. Effectiveness of psychotropic medications in the maintenance phase of bipolar disorder: a meta-analysis of randomized controlled trials. Int. J. Neuropsychopharmacol. 14, 1029-1049 (2011).
62. Wisdom, N. M., Callahan, J. L. & Hawkins, K. A. The effects of apolipoprotein E on non-impaired cognitive functioning: a meta-analysis. Neurobiol. Aging 32, 63-74 (2011).
63. Witteman, J., van Ijzendoorn, M. H., van de Velde, D., van Heuven, V. J. & Schiller, N. O. The nature of hemispheric specialization for linguistic and emotional prosodic perception: a meta-analysis of the lesion literature. Neuropsychologia 49, 3722-3738 (2011).
64. Woon, F. & Hedges, D. W. Gender does not moderate hippocampal volume deficits in adults with posttraumatic stress disorder: a meta-analysis. Hippocampus 21, 243-252 (2011).
65. Xuan, C. et al. No association between APOE ε 4 allele and multiple sclerosis susceptibility: a meta-analysis from 5472 cases and 4727 controls. J. Neurol. Sci. 308, 110-116 (2011).
66. Yang, W. M., Kong, F. Y., Liu, M. & Hao, Z. L. Systematic review of risk factors for progressive ischemic stroke. Neural Regen. Res. 6, 346-352 (2011).
67. Yang, Z., Li, W. J., Huang, T., Chen, J. M. & Zhang, X. Meta-analysis of Ginkgo biloba extract for the treatment of Alzheimer's disease. Neural Regen. Res. 6, 1125-1129 (2011).
68. Yuan, H. et al. Meta-analysis of tau genetic polymorphism and sporadic progressive supranuclear palsy susceptibility. Neural Regen. Res. 6, 353-359 (2011).
69. Zafar, S. N., Iqbal, A., Farez, M. F., Kamatkar, S. & de Moya, M. A. Intensive insulin therapy in brain injury: a meta-analysis. J. Neurotrauma 28, 1307-1317 (2011).
70. Zhang, Y. G. et al. The-1082G/A polymorphism in IL-10 gene is associated with risk of Alzheimer's disease: a meta-analysis. J. Neurol. Sci. 303, 133-138 (2011).
71. Zhu, Y., He, Z. Y. & Liu, H. N. Meta-analysis of the relationship between homocysteine, vitamin B(12), folate, and multiple sclerosis. J. Clin. Neurosci. 18, 933-938 (2011).
72. Ioannidis, J. P. & Trikalinos, T. A. An exploratory test for an excess of significant findings. Clin. Trials 4, 245-253 (2007).
73. Ioannidis, J. P. Excess significance bias in the literature on brain volume abnormalities. Arch. Gen. Psychiatry 68, 773-780 (2011).
74. Pfeiffer, T., Bertram, L. & Ioannidis, J. P. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 6, e18362 (2011).
75. Tsilidis, K. K., Papatheodorou, S. I., Evangelou, E. & Ioannidis, J. P. Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk. J. Natl Cancer Inst. 104, 1867-1878 (2012).
76. Ioannidis, J. Clarifications on the application and interpretation of the test for excess significance and its extensions. J. Math. Psychol. (in the press).
77. David, S. P. et al. Potential reporting bias in small fMRI studies of the brain. PLoS Biol. (in the press).
78. Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010).
79. Ioannidis, J. P. Extrapolating from animals to humans. Sci. Transl. Med. 4, 151ps15 (2012).
80. Jonasson, Z. Meta-analysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data. Neurosci. Biobehav. Rev. 28, 811-825 (2005).
81. Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824-2829 (2008).
82. Sena, E., van der Worp, H. B., Howells, D. & Macleod, M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 30, 433-439 (2007).
83. Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Methuen, 1958).
84. Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).
85. Bassler, D., Montori, V. M., Briel, M., Glasziou, P. & Guyatt, G. Early stopping of randomized clinical trials for overt efficacy is problematic. J. Clin. Epidemiol. 61, 241-246 (2008).
86. Montori, V. M. et al. Randomized trials stopped early for benefit: a systematic review. JAMA 294, 2203-2209 (2005).
87. Berger, J. O. & Wolpert, R. L. The Likelihood Principle: A Review, Generalizations, and Statistical Implications (ed. Gupta, S. S.) (Institute of Mathematical Sciences, 1998).
88. Vesterinen, H. M. et al. Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the Journal of Cerebral Blood Flow and Metabolism. J. Cereb. Blood Flow Metab. 31, 1064-1072 (2011).
89. Smith, R. A., Levine, T. R., Lachlan, K. A. & Fediuk, T. A. The high cost of complexity in experimental design and data analysis: type I and type II error rates in multiway ANOVA. Hum. Comm. Res. 28, 515-530 (2002).
90. Perel, P. et al. Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 334, 197 (2007).
91. Nosek, B. A. & Bar-Anan, Y. Scientific utopia: I. Opening scientific communication. Psychol. Inquiry 23, 217-243 (2012).
92. Open-Science-Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657-660 (2012).
93. Simera, I. et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 8, 24 (2010).
94. Ioannidis, J. P. The importance of potential studies that have not existed and registration of observational data sets. JAMA 308, 575-576 (2012).
95. Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H. & Ioannidis, J. P. Public availability of published research data in high-impact journals. PLoS ONE 6, e24357 (2011).
96. Ioannidis, J. P. et al. Repeatability of published microarray gene expression analyses. Nature Genet. 41, 149-155 (2009).
97. Ioannidis, J. P. & Khoury, M. J. Improving validation practices in "omics" research. Science 334, 1230-1232 (2011).
98. Chambers, C. D. Registered Reports: A new publishing initiative at Cortex. Cortex 49, 609-610 (2013).
99. Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450-456 (2011).
100. Siontis, K. C., Patsopoulos, N. A. & Ioannidis, J. P. Replication of past candidate loci for common diseases and phenotypes in 100 genome-wide association studies. Eur. J. Hum. Genet. 18, 832-837 (2010).
101. Ioannidis, J. P. & Trikalinos, T. A. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J. Clin. Epidemiol. 58, 543-549 (2005).
102. Ioannidis, J. Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645-654 (2012).
103. Zollner, S. & Pritchard, J. K. Overcoming the winner's curse: estimating penetrance parameters from case-control data. Am. J. Hum. Genet. 80, 605-615 (2007).