Scanning the horizon: Towards transparent and reproducible neuroimaging research

Nature Reviews Neuroscience

Volumn 18, Issue 2, 2017, Pages 115-126

  Poldrack, Russell A ^a   Baker, Chris I ^b   Durnez, Joke ^a,c   Gorgolewski, Krzysztof J ^a   Matthews, Paul M ^d   Munafò, Marcus R ^e   Nichols, Thomas E ^f   Poline, Jean Baptiste ^g   Vul, Edward ^h   Yarkoni, Tal ⁱ  

^a STANFORD UNIVERSITY   (United States)

^b NATIONAL INSTITUTE OF MENTAL HEALTH   (United States)

^c NeuroSpin   (France)

^d HAMMERSMITH HOSPITAL   (United Kingdom)

^e UNIVERSITY OF BRISTOL   (United Kingdom)

^f UNIVERSITY OF WARWICK   (United Kingdom)

^g UNIVERSITY OF CALIFORNIA   (United States)

^h UNIVERSITY OF CALIFORNIA   (United States)

ⁱ UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CLINICAL PRACTICE; DATA ANALYSIS; DIFFUSION WEIGHTED IMAGING; ERROR; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; NEUROSCIENCE; NONHUMAN; POWER ANALYSIS; PRIORITY JOURNAL; SOFTWARE; SOFTWARE ERROR; STATISTICAL ANALYSIS; STATISTICAL POWER; STRUCTURAL NUCLEAR MAGNETIC RESONANCE IMAGING; FUNCTIONAL NEUROIMAGING; NUCLEAR MAGNETIC RESONANCE IMAGING; PRACTICE GUIDELINE; REPRODUCIBILITY; STANDARDS; STATISTICS; STATISTICS AND NUMERICAL DATA; TRENDS;

FUNCTIONAL NEUROIMAGING; HUMANS; MAGNETIC RESONANCE IMAGING; PRACTICE GUIDELINES AS TOPIC; REPRODUCIBILITY OF RESULTS; SOFTWARE; STATISTICS AS TOPIC;

EID: 85008419692     PISSN: 1471003X     EISSN: 14710048     Source Type: Journal    
DOI: 10.1038/nrn.2016.167     Document Type: Article

References (78)

1. Poldrack, R. A. & Farah, M. J. Progress and challenges in probing the human brain. Nature 526, 371-379 (2015).
2. Logothetis, N. K. What we can do and what we cannot do with fMRI. Nature 453, 869-878 (2008).
3. Biswal, B. B. et al. Toward discovery science of human brain function. Proc. Natl Acad. Sci. USA 107, 4734-4739 (2010).
4. Kriegeskorte, N. et al. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60, 1126-1141 (2008).
5. Norman, K. A., Polyn, S. M., Detre, G. J. & Haxby, J. V. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10, 424-430 (2006).
6. Poldrack, R. A. Inferring mental states from neuroimaging data: from reverse inference to large-scale decoding. Neuron 72, 692-697 (2011).
7. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).
8. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366 (2011).
9. Gelman, A. & Loken, E. The statistical crisis in science. American Scientist 102, 40 (2014).
10. Ioannidis, J. P. A., Fanelli, D., Dunne, D. D. & Goodman, S. N. Meta-research: evaluation and improvement of research methods and practices. PLoS Biol. 13, e1002264 (2015).
11. Collins, F. S. & Tabak, L. A. Policy: NIH plans to enhance reproducibility. Nature 505, 612-613 (2014).
12. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365-376 (2013).
13. Yarkoni, T. Big correlations in little studies: inflated fMRI correlations reflect low statistical power-commentary on Vul et al. (2009). Perspect. Psychol. Sci. 4, 294-298 (2009).
14. David, S. P. et al. Potential reporting bias in fMRI studies of the brain. PLoS ONE 8, e70104 (2013).
15. Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C. & Wager, T. D. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8, 665-670 (2011).
16. Friston, K. J., Frith, C. D., Liddle, P. F. & Frackowiak, R. S. Comparing functional (PET) images: The assessment of significant change. J. Cereb. Blood Flow Metab. 11, 690-699 (1991).
17. Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W. & Smith, S. M. FSL. Neuroimage 62, 782-790 (2012).
18. Worsley, K. J. et al. A unified statistical approach for determining significant signals in images of cerebral activation. Hum. Brain Mapp. 4, 58-73 (1996).
19. Cheng, D. & Schwartzman, A. Distribution of the height of local maxima of Gaussian random fields. Extremes 18, 213-240 (2015).
20. Van Essen, D. C. et al. The WU-Minn Human Connectome Project: An overview. Neuroimage 80, 62-79 (2013).
21. Tong, Y. et al. Seeking optimal region-of-interest (ROI) single-value summary measures for fMRI studies in imaging genetics. PLoS ONE 11, e0151391 (2016).
22. Devlin, J. T. & Poldrack, R. A. In praise of tedious anatomy. Neuroimage 37, 1033-1041 (2007).
23. Desikan, R. S. et al. An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage 31, 968-980 (2006).
24. Durnez, J. et al. Power and sample size calculations for fMRI studies based on the prevalence of active peaks. Preprint at bioRxiv http://dx.doi.org/10.1101/049429 (2016).
25. Mumford, J. A. & Nichols, T. E. Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation. Neuroimage 39, 261-268 (2008).
26. Mennes, M., Biswal, B. B., Castellanos, F. X. & Milham, M. P. Making data sharing work: The FCP/INDI experience. Neuroimage 82, 683-691 (2013).
27. Thompson, P. M. et al. The ENIGMA Consortium: large-scale collaborative analyses of neuroimaging and genetic data. Brain Imaging Behav. 8, 153-182 (2014).
28. Rohlfing, T. & Poline, J.-B. Why shared data should not be acknowledged on the author byline. Neuroimage 59, 4189-4195 (2012).
29. Austin, M. A., Hair, M. S. & Fullerton, S. M. Research guidelines in the era of large-scale collaborations: An analysis of Genome-wide Association Study Consortia. Am. J. Epidemiol. 175, 962-969 (2012).
30. Savoy, R. L. Using small numbers of subjects in fMRI-based research. IEEE Eng. Med. Biol. Mag. 25, 52-59 (2006).
31. Poldrack, R. A. et al. Long-Term neural and physiological phenotyping of a single human. Nat. Commun. 6, 8885 (2015).
32. Kerr, N. L. HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196-217 (1998).
33. Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422-1425 (2015).
34. Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P. & Willmes, K. Registered reports: realigning incentives in scientific publishing. Cortex 66, A1-A2 (2015).
35. Sidén, P., Eklund, A., Bolin, D. & Villani, M. Fast Bayesian whole-brain fMRI analysis with spatial 3D priors. Neuroimage 146, 211-225 (2016).
36. Carp, J. On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Front. Neurosci. 6, 149 (2012).
37. Penny, W. D., Friston, K. J., Ashburner, J. T., Kiebel, S. J. & Nichols, T. E. Statistical Parametric Mapping: The Analysis of Functional Brain Images (Elsevier Science, 2011).
38. Cox, R. W. AFNI: what a long strange trip it's been. Neuroimage 62, 743-747 (2012).
39. Heininga, V. E., Oldehinkel, A. J., Veenstra, R. & Nederhof, E. I just ran a thousand analyses: benefits of multiple testing in understanding equivocal evidence on gene-environment interactions. PLoS ONE 10, e0125383 (2015).
40. Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D. & Etchells, J. P. Instead of 'playing the game' it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neurosci. 1, 4-17 (2014).
41. Muthukumaraswamy, S. D., Routley, B., Droog, W., Singh, K. D. & Hamandi, K. The effects of AMPA blockade on the spectral profile of human early visual cortex recordings studied with non-invasive MEG. Cortex 81, 266-275 (2016).
42. Hobson, H. M. & Bishop, D. V. M. Mu suppression-A good measure of the human mirror neuron system? Cortex 82, 290-310 (2016).
43. Churchill, N. W. et al. Optimizing preprocessing and analysis pipelines for single-subject fMRI: 2. Interactions with ICA, PCA, task contrast and inter-subject heterogeneity. PLoS ONE 7, e31147 (2012).
44. Bennett, C. M., Miller, M. B. & Wolford, G. L. Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction. Neuroimage 47, S125 (2009).
45. Eklund, A., Nichols, T. E. & Knutsson, H. Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates. Proc. Natl Acad. Sci. USA 113, 7900-7905 (2016).
46. Nichols, T. & Hayasaka, S. Controlling the familywise error rate in functional neuroimaging: A comparative review. Stat. Methods Med. Res. 12, 419-446 (2003).
47. Wager, T. D., Lindquist, M. & Kaplan, L. Meta-Analysis of functional neuroimaging data: current and future directions. Soc. Cogn. Affect. Neurosci. 2, 150-158 (2007).
48. Lieberman, M. D. & Cunningham, W. A. Type I and Type II error concerns in fMRI research: re-balancing the scale. Soc. Cogn. Affect. Neurosci. 4, 423-428 (2009).
49. Bennett, C. M., Wolford, G. L. & Miller, M. B. The principled control of false positives in neuroimaging. Soc. Cogn. Affect. Neurosci. 4, 417-422 (2009).
50. Hayasaka, S. & Nichols, T. E. Validating cluster size inference: random field and permutation methods. Neuroimage 20, 2343-2356 (2003).
51. Gorgolewski, K. J. et al. NeuroVault.org: A web-based repository for collecting and sharing unthresholded statistical maps of the human brain. Front. Neuroinform. 9, 8 (2015).
52. Hunt, L. T., Dolan, R. J. & Behrens, T. E. J. Hierarchical competitions subserving multi-Attribute choice. Nat. Neurosci. 17, 1613-1622 (2014).
53. Shehzad, Z. et al. A multivariate distance-based analytic framework for connectome-wide association studies. Neuroimage 93 (Pt. 1), 74-94 (2014).
54. Rubinov, M. & Sporns, O. Complex network measures of brain connectivity: uses and interpretations. Neuroimage 52, 1059-1069 (2010).
55. Craddock, R. C., Milham, M. P. & LaConte, S. M. Predicting intrinsic brain activity. Neuroimage 82, 127-136 (2013).
56. Butler, R. W. & Finelli, G. B. The infeasibility of quantifying the reliability of life-critical real-Time software. IEEE Trans. Software Eng. 19, 3-12 (1993).
57. Cox, R. W., Reynolds, R. C. & Taylor, P. A. AFNI and clustering: false positive rates redux. Preprint at bioRxiv http://dx.doi.org/10.1101/065862 (2016).
58. Waskom, M. L., Kumaran, D., Gordon, A. M., Rissman, J. & Wagner, A. D. Frontoparietal representations of task context support the flexible control of goal-directed cognition. J. Neurosci. 34, 10743-10755 (2014).
59. Poldrack, R. A. et al. Guidelines for reporting an fMRI study. Neuroimage 40, 409-414 (2008).
60. Carp, J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage 63, 289-300 (2012).
61. Guo, Q. et al. The reporting of observational clinical functional magnetic resonance imaging studies: A systematic review. PLoS ONE 9, e94412 (2014).
62. Nichols, T. E. et al. Best practices in data analysis and sharing in neuroimaging using MRI. Preprint at bioRxiv http://dx.doi.org/10.1101/054262 (2016).
63. Poldrack, R. A. Can cognitive processes be inferred from neuroimaging data? Trends Cogn. Sci. 10, 59-63 (2006).
64. Gelman, A. & Stern, H. The difference between 'significant' and 'not significant' is not itself statistically significant. Am. Stat. 60, 328-331 (2006).
65. Nieuwenhuis, S., Forstmann, B. U. & Wagenmakers, E.-J. Erroneous analyses of interactions in neuroscience: A problem of significance. Nat. Neurosci. 14, 1105-1107 (2011).
66. Boekel, W. et al. A purely confirmatory replication study of structural brain-behavior correlations. Cortex 66, 115-133 (2015).
67. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531-533 (2012).
68. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
69. Zuo, X.-N. et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1, 140049 (2014).
70. Poldrack, R. A. et al. Toward open sharing of task-based fMRI data: The OpenfMRI project. Front. Neuroinform. 7, 1-12 (2013).
71. Gil, Y. et al. Toward the geoscience paper of the future: best practices for documenting and sharing research from data to software to provenance. Earth Space Sci. 3, 388-415 (2016).
72. Boulesteix, A.-L. Ten simple rules for reducing overoptimistic reporting in methodological computational research. PLoS Comput. Biol. 11, e1004191 (2015).
73. Gorgolewski, K. J. et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci. Data 3, 160044 (2016).
74. Flint, J. & Munafò, M. R. Candidate and non-candidate genes in behavior genetics. Curr. Opin. Neurobiol. 23, 57-61 (2013).
75. Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450 (2011).
76. Burgess, S. et al. Using published data in Mendelian randomization: A blueprint for efficient identification of causal risk factors. Eur. J. Epidemiol. 30, 543-552 (2015).
77. Stein, J. L. et al. Identification of common variants associated with human hippocampal and intracranial volumes. Nat. Genet. 44, 552-561 (2012).
78. Barch, D. M. et al. Function in the human connectome: Task-fMRI and individual differences in behavior. Neuroimage 80, 169-189 (2013).