Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines

NeuroImage

Volumn 145, Issue , 2017, Pages 166-179

  Varoquaux, Gaël ^a,b   Raamana, Pradeep Reddy ^c,d   Engemann, Denis A ^b,e,f   Hoyos Idrobo, Andrés ^a,b   Schwartz, Yannick ^a,b   Thirion, Bertrand ^a,b  

^a INRIA SACLAY   (France)

^b DIF   (France)

^c ROTMAN RESEARCH INSTITUTE   (Canada)

^d UNIVERSITY OF TORONTO   (Canada)

^e Université Paris Sud and Université Paris Saclay ^*  (France)

^f INSTITUT DU CERVEAU ET DE LA MOELLE ÉPINIÈRE   (France)

Author keywords

Bagging; Cross validation; Decoding; FMRI; Model selection; MVPA; Sparse

Indexed keywords

ACCURACY; ARTICLE; BRAIN DECODER; FUNCTIONAL MAGNETIC RESONANCE IMAGING; MAGNETOENCEPHALOGRAPHY; NEUROIMAGING; PRACTICE GUIDELINE; PREDICTION; PRIORITY JOURNAL; SIMULATION; VALIDATION PROCESS; WORKING MEMORY; BRAIN DISEASE; DIAGNOSTIC IMAGING; HUMAN; PROCEDURES; STANDARDS;

BRAIN DISEASES; HUMANS; NEUROIMAGING;

EID: 85006839843     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2016.10.038     Document Type: Article

References (61)

1. Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., Gramfort, A., Thirion, B., Varoquaux, G., Machine learning for neuroimaging with scikit-learn. Front. Neuroinform., 8, 2014.
2. Arlot, S., Celisse, A., A survey of cross-validation procedures for model selection. Stat. Surv., 4, 2010, 40.
3. Ashburner, J., Friston, K.J., Diffeomorphic registration using geodesic shooting and gauss-newton optimisation. NeuroImage 55 (2011), 954–967.
4. Breiman, L., Bagging predictors. Mach. Learn., 24, 1996, 123.
5. Breiman, L., Spector, P., Submodel selection and evaluation in regression. the x-random case. Int. Stat. Rev./Rev. Int. Stat., 1992, 291–319.
6. Carroll, M.K., Cecchi, G.A., Rish, I., Garg, R., Rao, A.R., Prediction and interpretation of distributed neural activity with sparse models. NeuroImage, 44, 2009, 112.
7. Chen, X., Pereira, F., Lee, W., et al. Exploring predictive and reproducible modeling with the single-subject FIAC dataset. Hum. Brain Mapp., 27, 2006, 452.
8. Churchill, N.W., Yourganov, G., Strother, S.C., Comparing within-subject classification and regularization methods in fMRI for large and small sample sizes. Hum. Brain Mapp., 35, 2014, 4499.
9. Demirci, O., Clark, V.P., Magnotta, V.A., Andreasen, N.C., Lauriello, J., Kiehl, K.A., Pearlson, G.D., Calhoun, V.D., A review of challenges in the use of fMRI for disease classification/characterization and a projection pursuit application from a multi-site fMRI schizophrenia study. Brain Imaging Behav. 2 (2008), 207–226.
10. Duncan, K.J., Pattamadilok, C., Knierim, I., Devlin, J.T., Consistency and variability in functional localisers. Neuroimage, 46, 2009, 1018.
11. Fu, C.H., Mourao-Miranda, J., Costafreda, S.G., Khanna, A., Marquand, A.F., Williams, S.C., Brammer, M.J., Pattern classification of sad facial processing toward the development of neurobiological markers in depression. Biol. Psychiatry 63 (2008), 656–662.
12. Gorgolewski, K., Burns, C.D., Madison, C., Clark, D., Halchenko, Y.O., Waskom, M.L., Ghosh, S.S., Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python a flexible, lightweight and extensible neuroimaging data processing framework in python. Front. Neuroinform., 5, 2011, 13.
13. Gramfort, A., Thirion, B., Varoquaux, G., Identifying predictive regions from fMRI with TV-L1 prior. PRNI, 2013, 17.
14. Gramfort, A., Luessi, M., Larson, E., Engemann, D.A., Strohmeier, D., Brodbeck, C., Goj, R., Jas, M., Brooks, T., Parkkonen, L., et al. MEG and EEG data analysis with MNEPython. Front. Neurosci., 2013.
15. Gramfort, A., Luessi, M., Larson, E., Engemann, D.A., Strohmeier, D., Brodbeck, C., Parkkonen, L., Hämäläinen, M.S., MNE software for processing MEG and EEG data. Neuroimage 86 (2014), 446–460.
16. Grosenick, L., Klingenberg, B., Katovich, K., Knutson, B., Taylor, J.E., Interpretable whole-brain prediction analysis with graphnet. NeuroImage, 72, 2013, 304.
17. Hanke, M., Halchenko, Y.O., Sederberg, P.B., Hanson, S.J., Haxby, J.V., Pollmann, S., PyMVPA: a python toolbox for multivariate pattern analysis of fmri data. Neuroinformatics, 7, 2009, 37.
18. Hastie, T., Tibshirani, R., Friedman, J., The Elements of Statistical Learning. 2009, Springer.
19. Haxby, J.V., Gobbini, I.M., Furey, M.L., et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293, 2001, 2425.
20. Haynes, J.D., A primer on pattern-based approaches to fMRI: principles, pitfalls, and perspectives. Neuron, 87, 2015, 257.
21. Haynes, J.D., Rees, G., Decoding mental states from brain activity in humans. Nat. Rev. Neurosci., 7, 2006, 523.
22. Henson, R., Shallice, T., Gorno-Tempini, M., Dolan, R., Face repetition effects in implicit and explicit memory tests as measured by fMRI. Cereb. Cortex, 12, 2002, 178.
23. Hoyos-Idrobo, A., Schwartz, Y., Varoquaux, G., Thirion, B., Improving sparse recovery on structured images with bagged clustering. PRNI, 2015.
24. Kamitani, Y., Tong, F., Decoding the visual and subjective contents of the human brain. Nat. Neurosci., 8, 2005, 679.
25. Karahanoğlu, F.I., Caballero-Gaudes, C., Lazeyras, F., Van De Ville, D., Total activation: fMRI deconvolution through spatio-temporal regularization. Neuroimage, 73, 2013, 121.
26. Knops, A., Thirion, B., Hubbard, E.M., Michel, V., Dehaene, S., Recruitment of an area involved in eye movements during mental arithmetic. Science, 324, 2009, 1583.
27. Kohavi, R., 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection, in: IJCAI, 14, pp. 1137.
28. Kriegeskorte, N., Goebel, R., Bandettini, P., Information-based functional brain mapping. Proc. Natl. Acad. Sci. USA, 103, 2006, 3862.
29. Kuncheva, L.I., Rodríguez, J.J., Classifier ensembles for fMRI data analysis an experiment. Magn. Reson. Imaging, 28, 2010, 583.
30. LaConte, S., Anderson, J., Muley, S., Ashe, J., Frutiger, S., Rehm, K., Hansen, L., Yacoub, E., Hu, X., Rottenberg, D., The evaluation of preprocessing choices in single-subject bold fMRI using npairs performance metrics. NeuroImage, 18, 2003, 10.
31. LaConte, S., Strother, S., Cherkassky, V., Anderson, J., Hu, X., Support vector machines for temporal classification of block design fMRI data. NeuroImage 26 (2005), 317–329.
32. Langs, G., Menze, B.H., Lashkari, D., Golland, P., Detecting stable distributed patterns of brain activation using gini contrast. NeuroImage, 56, 2011, 497.
33. Larson-Prior, L.J., Oostenveld, R., Della Penna, S., Michalareas, G., Prior, F., Babajani-Feremi, A., Schoffelen, J.M., Marzetti, L., de Pasquale, F., Pompeo, F.Di., et al. Adding dynamics to the human connectome project with MEG. Neuroimage 80 (2013), 190–201.
34. Marcus, D.S., Wang, T.H., Parker, J., et al. Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J Cogn. Neurosci., 19, 2007, 1498.
35. Meinshausen, N., Bühlmann, P., Stability selection. J. R. Stat. Soc. B, 72, 2010, 417.
36. Michel, V., Gramfort, A., Varoquaux, G., Eger, E., Thirion, B., Total variation regularization for fMRI-based prediction of behavior. IEEE Trans. Med. Imaging, 30, 2011, 1328.
37. Millman, K.J., Brett, M., Analysis of functional magnetic resonance imaging in python. Comput. Sci. Eng. 9 (2007), 52–55.
38. Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A., Predicting human brain activity associated with the meanings of nouns. Science, 320, 2008, 1191.
39. Moran, J.M., Jolly, E., Mitchell, J.P., Social-cognitive deficits in normal aging. J. Neurosci., 32, 2012, 5553.
40. Mouro-Miranda, J., Bokde, A.L., Born, C., Hampel, H., Stetter, M., Classifying brain states and determining the discriminating activation patterns: support vector machine on functional MRI data. NeuroImage, 28, 2005, 980.
41. Naselaris, T., Kay, K.N., Nishimoto, S., Gallant, J.L., Encoding and decoding in fMRI. Neuroimage, 56, 2011, 400.
42. Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V., Beyond mind-reading multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci., 10, 2006, 424.
43. Pedregosa, F., Varoquaux, G., Gramfort, A., et al. Scikit-learn machine learning in python. J. Mach. Learn. Res., 12, 2011, 2825.
44. Penny, W.D., Friston, K.J., Ashburner, J.T., Kiebel, S.J., Nichols, T.E., Statistical parametric mapping: the analysis of functional brain images. 2007, Academic Press, London.
45. Pereira, F., Mitchell, T., Botvinick, M., Machine learning classifiers and fMRI: a tutorial overview. Neuroimage, 45, 2009, S199.
46. Platt, J., Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. large margin Classif., 10, 1999, 61.
47. Poldrack, R.A., Halchenko, Y.O., Hanson, S.J., Decoding the large-scale structure of brain function by classifying mental states across individuals. Psychol. Sci., 20, 2009, 1364.
48. Poldrack, R.A., Barch, D.M., Mitchell, J.P., et al. Toward open sharing of task-based fMRI data: the OpenfMRI project. Front. Neuroinform., 7, 2013.
49. Raamana, P.R., Weiner, M.W., Wang, L., Beg, M.F., Thickness network features for prognostic applications in dementia. Neurobiol. Aging 36:Suppl. 1 (2015), S91–S102.
50. Rasmussen, P.M., Hansen, L.K., Madsen, K.H., Churchill, N.W., Strother, S.C., Model sparsity and brain pattern interpretation of classification models in neuroimaging. Pattern Recognit. 45:Suppl. 1 (2012), 2085–2100.
51. Rondina, J.M., Shawe-Taylor, J., Mourão-Miranda, J., Stability-based multivariate mapping using scors. PRNI, 2013, 198.
52. Ryali, S., Supekar, K., Abrams, D., Menon, V., Sparse logistic regression for whole-brain classification of fMRI data. NeuroImage, 51, 2010, 752.
53. Schwartz, Y., Thirion, B., Varoquaux, G., Mapping cognitive ontologies to and from the brain. NIPS, 2013.
54. Sitt, J.D., King, J.R., El Karoui, I., Rohaut, B., Faugeras, F., Gramfort, A., Cohen, L., Sigman, M., Dehaene, S., Naccache, L., Large scale screening of neural signatures of consciousness in patients in a vegetative or minimally conscious state. Brain 137 (2014), 2258–2270.
55. Strother, S.C., Anderson, J., Hansen, L.K., Kjems, U., Kustra, R., Sidtis, J., Frutiger, S., Muley, S., LaConte, S., Rottenberg, D., The quantitative evaluation of functional neuroimaging experiments: the npairs data analysis framework. NeuroImage, 15, 2002, 747.
56. Strother, S.C., Rasmussen, P.M., Churchill, N.W., Hansen, L.K., Stability and reproducibility in fMRI analysis. Pract. Appl. Sparse Model., 2014, 99.
57. Varoquaux, G., Thirion, B., How machine learning is shaping cognitive neuroimaging. GigaScience, 3, 2014, 28.
58. Varoquaux, G., Gramfort, A., Thirion, B., Small-sample brain mapping: sparse recovery on spatially correlated designs with randomization and clustering. ICML, 2012, 1375.
59. Wager, T.D., Davidson, M.L., Hughes, B.L., et al. Neural mechanisms of emotion regulation: evidence for two independent prefrontal-subcortical pathways. Neuron, 59, 2008, 1037.
60. Yamashita, O., aki Sato, M., Yoshioka, T., Tong, F., Kamitani, Y., Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. NeuroImage, 42, 2008, 1414.
61. Yarkoni, T., Westfall, J., 2016. Choosing Prediction over Explanation in Psychology: Lessons from Machine Learning, Figshare Preprint.