Decoding neural representational spaces using multivariate pattern analysis

Annual Review of Neuroscience

Volumn 37, Issue , 2014, Pages 435-456

  Haxby, James V ^a,b   Connolly, Andrew C ^a   Guntupalli, J Swaroop ^a  

^a Department of Psychological and Brain Sciences Center for Cognitive Neuroscience Dartmouth College ^*  (United States)

^b UNIVERSITY OF TRENTO   (Italy)

Author keywords

FMRI; Hyperalignment; MVPA; Neural decoding; Population response; RSA

Indexed keywords

ALGORITHM; ENDONEURIUM; FUNCTIONAL MAGNETIC RESONANCE IMAGING; MACHINE LEARNING; MULTIVARIATE ANALYSIS; NERVE CELL; NERVE CELL NETWORK; PRIORITY JOURNAL; REVIEW; ANIMAL; BIOLOGICAL MODEL; BRAIN MAPPING; HUMAN; IMAGE PROCESSING; PHYSIOLOGY; PROCEDURES;

ANIMALS; BRAIN MAPPING; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MODELS, NEUROLOGICAL; NEURONS;

EID: 84904482224     PISSN: 0147006X     EISSN: 15454126     Source Type: Book Series    
DOI: 10.1146/annurev-neuro-062012-170325     Document Type: Review

References (74)

1. Abdi H, Dunlop JP,Williams LJ. 2009. How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the Bootstrap and 3-waymultidimensional scaling (DISTATIS). NeuroImage 45:89-95
2. Abdi H, Williams LJ, Conolly AC, Gobbini MI, Dunlop JP, Haxby JV. 2012a. Multiple Subject Barycentric Discriminant Analysis (MUSUBADA): how to assign scans to categories without using spatial normalization. Comp. Math. Methods Med. 2012:634165
3. Abdi H, Williams LJ, Valentin D, Bennani-Dosse M. 2012b. STATIS and DISTATIS: optimum multitable principal component analysis and three way metric multidimensional scaling. Wiley Interdiscip. Rev. Comput. Stat. 4:124-67
4. Aflalo TN, Graziano MSA. 2006. Possible origins of the complex topographic organization of motor cortex: reduction of a multidimensional space onto a two-dimensional array. J. Neurosci. 26:6288-97
5. Aflalo TN, Graziano MSA. 2011. Organization of the macaque extrastriate cortex re-examined using the principle of spatial continuity of function. J. Neurophysiol. 105:305-20
6. Brants M, Baeck A,Wagemans J, Op de Beeck H. 2011. Multiple scales of organization for object selectivity in ventral visual cortex. NeuroImage 56:1372-81
7. Carlin JD, Calder AJ, Kriegeskorte N, Nili H, Rowe JB. 2011. A head view-invariant representation of gaze direction in anterior superior temporal cortex. Curr. Biol. 21:1817-21
8. Carlson TA, Hogendoorn H, Kanai R, Mesik J, Turret J. 2011. High temporal resolution decoding of object position and category. J. Vis. 11:1-17
9. Carlson TA, Schrater P, He S. 2003. Patterns of activity in the categorical representations of objects. J. Cogn. Neurosci. 15:704-17
10. CaseyM, Thompson J, Kang O, Raizada R,Wheatley T. 2012. Population codes representing musical timbre for high-level fMRI categorization of music genres. In Machine Learning and Interpretation in Neuroimaging, Ser. Vol. 7263, ed. G Langs, I Rish, M Grosse-Wentrup, B Murphy, pp. 36-41. Berlin/Heidelberg: Springer-Verlag
11. Chen Y, Namburi P, Elliott LT, Heinzle J, Soon CS, et al. 2011. Cortical surface-based searchlight decoding. NeuroImage 56:582-92
12. Connolly AC, Gobbini MI, Haxby JV. 2012a. Three virtues of similarity-based multi-voxel pattern analysis: an example from the human object vision pathway. In Understanding Visual Population Codes (UVPC): Toward A Common Multivariate Framework for Cell Recording and Functional Imaging, ed. N Kriegeskorte, G Kreiman, pp. 335-55. Cambridge, MA: MIT Press
13. Connolly AC, Guntupalli JS, Gors J, Hanke M, Halchenko YO, et al. 2012b. The representation of biological classes in the human brain. J. Neurosci. 32:2608-18
14. Conroy BR, Singer BD, Guntupalli JS, Ramadge PJ, Haxby JV. 2013. Inter-subject alignment of human cortical anatomy using functional connectivity. NeuroImage 81:400-11
15. Cortes C, Vapnik V. 1995. Support-vector networks. Mach. Learn. 20:273-97
16. Cox DD, Savoy RL. 2003. Functional magnetic resonance imaging (fMRI) brain reading: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage 19:261-70
17. DiCarlo JJ, Cox DD. 2007. Untangling invariant object recognition. Trends Cogn. Sci. 11:333-41
18. Durbin R, Mitchison G. 1990. A dimension reduction framework for understanding cortical maps. Nature 343:644-47
19. Edelman S, Grill-Spector K, Kushnir T, Malach R. 1998. Toward direct visualization of the internal shape space by fMRI. Psychobiology 26:309-21
20. Formisano I, De Martino F, Bonte M, Goebel R. 2008. Who is saying what? Brain-based decoding of human voice and speech. Science 322:970-73
21. Freiwald WA, Tsao DY. 2010. Functional compartmentalization and viewpoint generalization within the macaque face-processing system. Science 330:845-51
22. FristonK,Kiebel S. 2009. Predictive coding under the free-energy principle.Phil.Trans. R. Soc. B. 364:1211-21
23. Graziano MSA, Aflalo TN. 2007a. Mapping behavioral repertoire onto the cortex. Neuron 56:239-51
24. GrazianoMSA, Aflalo TN. 2007b. Rethinking cortical organization: moving away from discrete areas arranged in hierarchies. Neuroscientist 13:138-47
25. Hanke M, Halchenko YO, Sederberg PB, Hanson SJ, Haxby JV, Pollman S. 2009. PyMVPA: A Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics. 7:37-53
26. Hanson SJ,ToshihikoM,Haxby JV. 2004. Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a face area? NeuroImage 23:156-67
27. Harrison SA, Tong F. 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature 458:632-35
28. Haxby JV. 2012. Multivariate pattern analysis of fMRI: the early beginnings. NeuroImage 62:852-55
29. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425-30
30. Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, et al. 2011. A common, highdimensional model of the representational space in human ventral temporal cortex. Neuron 72:404-16
31. Haynes JD, Rees G. 2005. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8:686-91
32. Haynes JD, Rees G. 2006. Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7:523-34
33. Haynes JD, Sakai K, Rees G, Gilbert S, Frith C, Passingham RE. 2007. Reading hidden intentions in the human brain. Curr. Biol. 17:323-28
34. Horikawa T, Tamaki M, Miyawaki Y, Kamitani Y. 2013. Neural decoding of visual imagery during sleep. Science 340:639-42
35. Hung CP, Kreiman G, Poggio T, DiCarlo JJ. 2005. Fast readout of object identity from macaque inferior temporal cortex. Science 310:863-66
36. Kamitani Y, Tong F. 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8:679-85
37. Kanwisher N. 2010. Functional specificity in the human brain: a window into the functional architecture of the mind. Proc. Natl. Acad. Sci. USA 107:11163-70
38. Kanwisher N,McDermott J, Chun MM. 1997. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17:4302-11
39. Kay KN, Naselaris T, Prenger RJ, Gallant JL. 2008. Identifying natural images from human brain activity. Nature 452:352-55
40. Kiani R, Esteky H, Mirpour K, Tanaka K. 2007. Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. J. Neurophysiol. 97:4296-309
41. Kohler PJ, Fogelson SV, Reavis EA, Meng M, Guntupalli JS, et al. 2013. Pattern classification precedes region-average hemodynamic response in early visual cortex. NeuroImage 78:249-60
42. Kohonen T. 1982. Self-organizing formation of topologically correct feature maps. Biol. Cybern. 43:59-69
43. Kohonen T. 2001. Self-Organizing Maps. Berlin: Springer
44. Kriegeskorte N, Goebel R, Bandettini P. 2006. Information-based functional brain mapping. Proc. Natl. Acad. Sci. USA 103:3863-68
45. Kriegeskorte N, Kievet RA. 2013. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17:401-12
46. Kriegeskorte N, Mur M, Bandettini P. 2008a. Representational similarity analysis-connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2:4
47. Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J, et al. 2008b. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60:1126-41
48. Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. 2009. Circular analysis is systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12:535-40
49. Mahon BZ, Anzellotti S, Schwarzbach J, Zampini M, Caramazza A. 2009. Category-specific organization in the human brain does not require visual experience. Neuron 63:397-405
50. Martin A. 2007. The representation of object concepts in the brain. Annu. Rev. Psychol 58:25-45
51. Mitchell TM, Shinkareva SV, Carlson A, Chang K-M,Malave VL, et al. 2008. Predicting human brain activity associated with the meanings of nouns. Science 320:1191-95
52. Miyawaki Y, Uchida H, YamashitaO, SatoM,Morito Y, et al. 2008. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60:915-29
53. Naselaris T, Kay KN, Nishimoto S, Gallant JL. 2011. Encoding and decoding in fMRI. NeuroImage 56:400-10
54. Naselaris T, Prenger RJ, Kay KN, Oliver M, Gallant JL. 2009. Bayesian reconstruction of natural images from human brain activity. Neuron 63:902-15
55. Nishimoto S, Vu AT, Naselaris T, Bejamini Y, Yu B, Gallant JL. 2011. Reconstructing visual experience from brain activity evoked by natural movies. Curr. Biol. 21:1641-46
56. Norman KA, Polyn SM, Detre GJ, Haxby JV. 2006. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10:424-30
57. Oosterhof NN, Wiestler T, Downing PE, Diedrichsen J. 2011. A comparison of volume-based and surfacebased multi-voxel pattern analysis. NeuroImage 56:593-600
58. Op de Beeck H, Brants M, Baeck A, Wagemans J. 2010. Distributed subordinate specificity for bodies, faces, and buildings in human ventral visual cortex. NeuroImage 49:3414-25
59. OToole AJ, Jiang F, Abdi H, Haxby JV. 2005. Partially distributed representations of objects and faces in ventral temporal cortex. J. Cogn. Neurosci. 17:580-90
60. OToole AJ, Jiang F, Abdi H, Penard N, Dunlop JP, Parent MA. 2007. Theoretical, statistical, and practical perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data. J. Cogn. Neurosci. 191:735-52
61. Pasley BN, David SV, Mesgarani N, Flinker A, Shamma SA, et al. 2012. Reconstructing speech from human auditory cortex. PLoS Biol. 10:e1001251
62. Pereira F, Mitchell T, Botvinick M. 2009. Machine learning classifiers and fMRI: a tutorial overview. NeuroImage 45(Suppl. 1):S199-209
63. Raizada RDS,Connolly AC. 2012. What makes different peoples representations alike: neural similarity space solves the problem of across-subject fMRI decoding. J. Cogn. Neurosci. 24:868-77
64. Sabuncu M, Singer BD, Conroy B, Bryan RE, Ramadge PJ, Haxby JV. 2010. Function-based intersubject alignment of human cortical anatomy. Cereb. Cortex 20:130-40
65. Schonemann PH. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika 31:1-10
66. Schreiner CE. 1995. Order and disorder in auditory cortical maps. Curr. Opin. Neurobiol. 5:489-96
67. Shinkareva SV,Malave VL, Mason RA, Mitchell TM, JustMA. 2011. Commonality of neural representations of words and pictures. NeuroImage 54:2418-25
68. Shinkareva SV,Mason RA,Malave VL, Wang W, Mitchell TM, JustMA. 2008. Using fMRI brain activation to identify cognitive states associated with perception of tools and dwellings. PLoS ONE 1:e1394
69. Soon CS, Brass M, Heinze HJ, Haynes JD. 2008. Unconscious determinants of free decisions in the human brain. Nat. Neurosci. 5:543-45
70. Staeren N, Renvall H, De Martino F, Goebel R, Formisano E. 2009. Sound categories are represented as distributed patterns in the human auditory cortex. Curr. Biol. 19:498-502
71. Sudre G, Pomerleau D, Palatucci M, Wehbe L, Fyshe A, et al. 2012. Tracking neural coding of perceptual and semantic features of concrete nouns. NeuroImage 62:451-63
72. Tong F, Pratte MS. 2012. Decoding patterns of human brain activity. Annu. Rev. Psychol. 63:483-509
73. Weiner K, Grill-Spector K. 2011. The improbable simplicity of the fusiform face area. Trends Cogn. Sci. 16:251-54
74. Yamashita O, Sato M-A, Yoshioka T, Tong F, Kamitani Y. 2008. Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. NeuroImage 42:1414-29