Multivariate cross-classification: Applying machine learning techniques to characterize abstraction in neural representations

Frontiers in Human Neuroscience

Volumn 9, Issue MAR, 2015, Pages

  Kaplan, Jonas T ^a   Man, Kingson ^a   Greening, Steven G ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

fMRI methods; MPVA; Multivariate pattern analysis; Multivariate pattern analysis techniques; Multivariate pattern classification; Similarity based representation

Indexed keywords

ALGORITHM; ARTICLE; BRAIN REGION; CLASSIFIER; HUMAN; IMAGERY; IMAGINATION; LANGUAGE ABILITY; MACHINE LEARNING; MENTAL FUNCTION; MULTIVARIATE ANALYSIS; MULTIVARIATE CROSS CLASSIFICATION; NEURAL REPRESENTATION ABSTRACTION; RECALL; SEMANTIC MEMORY; TASK PERFORMANCE; VISUAL MEMORY; VISUAL STIMULATION; WORKING MEMORY;

EID: 84933677907     PISSN: None     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2015.00151     Document Type: Article

References (86)

1. Akama, H., Murphy, B., Na, L., Shimizu, Y., and Poesio, M. (2012). Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study. Front. Neuroinform. 6:24. doi: 10.3389/fninf.2012.00024
2. Albers, A. M., Kok, P., Toni, I., Dijkerman, H. C., and de Lange, F. P. (2013). Shared representations for working memory and mental imagery in early visual cortex. Curr. Biol. 23, 1427-1431. doi: 10.1016/j.cub.2013.05.065
3. Amedi, A., von Kriegstein, K., van Atteveldt, N. M., Beauchamp, M. S., and Naumer, M. J. (2005). Functional imaging of human crossmodal identification and object recognition. Exp. Brain Res. 166, 559-571. doi: 10.1007/s00221-005-2396-5
4. Beck, D. M., and Kastner, S. (2009). Top-down and bottom-up mechanisms in biasing competition in the human brain. Vision Res. 49, 1154-1165. doi: 10.1016/j.visres.2008.07.012
5. Buccino, G., Binkofski, F., Fink, G. R., Fadiga, L., Fogassi, L., Gallese, V., et al. (2001). Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur. J. Neurosci. 13, 400-404. doi: 10.1111/j.1460-9568.2001.01385.x
6. Buchweitz, A., Shinkareva, S. V., Mason, R. A., Mitchell, T. M., and Just, M. A. (2012). Identifying bilingual semantic neural representations across languages. Brain Lang. 120, 282-289. doi: 10.1016/j.bandl.2011.09.003
7. Bundesen, C. (1990). A theory of visual attention. Psychol. Rev. 97, 523-547. doi: 10.1037/0033-295X.97.4.523
8. Caggiano, V., Fogassi, L., Rizzolatti, G., Pomper, J. K., Thier, P., Giese, M. A., et al. (2011). View-based encoding of actions in mirror neurons of area f5 in macaque premotor cortex. Curr. Biol. 21, 144-148. doi: 10.1016/j.cub.2010.12.022
9. Cichy, R. M., Heinzle, J., and Haynes, J. D. (2012). Imagery and perception share cortical representations of content and location. Cereb. Cortex 22, 372-380. doi: 10.1093/cercor/bhr106
10. Correia, J., Formisano, E., Valente, G., Hausfeld, L., Jansma, B., and Bonte, M. (2014). Brain-based translation: fMRI decoding of spoken words in bilinguals reveals language-independent semantic representations in anterior temporal lobe. J. Neurosci. 34, 332-338. doi: 10.1523/jneurosci.1302-13.2014
11. Coutanche, M. N., and Thompson-Schill, S. L. (2014). Creating concepts from converging features in human cortex.Cereb. Cortex doi: 10.1093/cercor/bhu057. [Epub ahead of print].
12. Davis, T., LaRocque, K. F., Mumford, J. A., Norman, K. A., Wagner, A. D., and Poldrack, R. A. (2014a). What do differences between multi-voxel and univariate analysis mean? How subject-, voxel- and trial-level variance impact fMRI analysis. Neuroimage 97, 271-283. doi: 10.1016/j.neuroimage.2014.04.037
13. Davis, T., and Poldrack, R. A. (2013). Measuring neural representations with fMRI: practices and pitfalls. Ann. N Y Acad. Sci. 1296, 108-134. doi: 10.1111/nyas.12156
14. Davis, T., Xue, G., Love, B. C., Preston, A. R., and Poldrack, R. A. (2014b). Global neural pattern similarity as a common basis for categorization and recognition memory. J. Neurosci. 34, 7472-7484. doi: 10.1523/jneurosci.3376-13.2014
15. Decety, J., and Grèzes, J. (2006). The power of simulation: imagining one's own and other's behavior. Brain Res.1079, 4-14. doi: 10.1016/j.brainres.2005.12.115
16. Desimone, R. (1998). Visual attention mediated by biased competition in extrastriate visual cortex. Philos. Trans. R. Soc. Lond. B Biol. Sci. 353, 1245-1255. doi: 10.1098/rstb.1998.0280
17. Desimone, R., and Duncan, J. (1995). Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193-222. doi: 10.1146/annurev.neuro.18.1.193
18. Dinstein, I., Gardner, J. L., Jazayeri, M., and Heeger, D. J. (2008). Executed and observed movements have different distributed representations in human aIPS. J. Neurosci. 28, 11231-11239. doi: 10.1523/jneurosci.3585-08.2008
19. Doehrmann, O., and Naumer, M. J. (2008). Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res. 1242, 136-150. doi: 10.1016/j.brainres.2008.03.071
20. Ester, E. F., Serences, J. T., and Awh, E. (2009). Spatially global representations in human primary visual cortex during working memory maintenance. J. Neurosci. 29, 15258-15265. doi: 10.1523/jneurosci.4388-09.2009
21. Etzel, J. A., Gazzola, V., and Keysers, C. (2008). Testing simulation theory with cross-modal multivariate classification of fMRI data. PLoS One 3:e3690. doi: 10.1371/journal.pone.0003690
22. Etzel, J. A., Valchev, N., and Keysers, C. (2011). The impact of certain methodological choices on multivariate analysis of fMRI data with support vector machines. Neuroimage 54, 1159-1167. doi: 10.1016/j.neuroimage.2010.08.050
23. Fadiga, L., Craighero, L., and Olivier, E. (2005). Human motor cortex excitability during the perception of others' action. Curr. Opin. Neurobiol. 15, 213-218. doi: 10.1016/j.conb.2005.03.013
24. Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain119(Pt. 2), 593-609. doi: 10.1093/brain/119.2.593
25. Gallese, V., and Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends Cogn. Sci.2, 493-501. doi: 10.1016/s1364-6613(98)01262-5
26. Gazzola, V., Aziz-Zadeh, L., and Keysers, C. (2006). Empathy and the somatotopic auditory mirror system in humans. Curr. Biol. 16, 1824-1829. doi: 10.1016/j.cub.2006.07.072
27. Grèzes, J., and Decety, J. (2001). Functional anatomy of execution, mental simulation, observation and verb generation of actions: a meta-analysis. Hum. Brain Mapp. 12, 1-19. doi: 10.1002/1097-0193(200101)12:1<1::aid-hbm10>3.0.co;2-v
28. Harrison, S. A., and Tong, F. (2009). Decoding reveals the contents of visual working memory in early visual areas.Nature 458, 632-635. doi: 10.1038/nature07832
29. Haxby, J. V., Connolly, A. C., and Guntupalli, J. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annu. Rev. Neurosci. 37, 435-456. doi: 10.1146/annurev-neuro-062012-170325
30. Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., and Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425-2430. doi: 10.1126/science.1063736
31. Haynes, J.-D., and Rees, G. (2006). Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7, 523-534. doi: 10.1038/nrn1931
32. Horikawa, T., Tamaki, M., Miyawaki, Y., and Kamitani, Y. (2013). Neural decoding of visual imagery during sleep.Science 340, 639-642. doi: 10.1126/science.1234330
33. Iacoboni, M. (2009). Imitation, empathy and mirror neurons. Annu. Rev. Psychol. 60, 653-670. doi: 10.1146/annurev.psych.60.110707.163604
34. Iacoboni, M., Woods, R. P., Brass, M., Bekkering, H., Mazziotta, J. C., and Rizzolatti, G. (1999). Cortical mechanisms of human imitation. Science 286, 2526-2528. doi: 10.1126/science.286.5449.2526
35. Ishai, A., Ungerleider, L. G., and Haxby, J. V. (2000). Distributed neural systems for the generation of visual images. Neuron 28, 979-990. doi: 10.1016/s0896-6273(00)00168-9
36. Jehee, J. F., Brady, D. K., and Tong, F. (2011). Attention improves encoding of task-relevant features in the human visual cortex. J. Neurosci. 31, 8210-8219. doi: 10.1523/jneurosci.6153-09.2011
37. Johnson, M. R., and Johnson, M. K. (2014). Decoding individual natural scene representations during perception and imagery. Front. Hum. Neurosci. 8:59. doi: 10.3389/fnhum.2014.00059
38. Johnson, J. D., McDuff, S. G., Rugg, M. D., and Norman, K. A. (2009). Recollection, familiarity and cortical reinstatement: a multivoxel pattern analysis. Neuron 63, 697-708. doi: 10.1016/j.neuron.2009.08.011
39. Johnson-Frey, S. H., Maloof, F. R., Newman-Norlund, R., Farrer, C., Inati, S., and Grafton, S. T. (2003). Actions or hand-object interactions? Human inferior frontal cortex and action observation. Neuron 39, 1053-1058. doi: 10.1016/s0896-6273(03)00524-5
40. Kamitani, Y., and Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nat. Neurosci.8, 679-685. doi: 10.1038/nn1444
41. Kamitani, Y., and Tong, F. (2006). Decoding seen and attended motion directions from activity in the human visual cortex. Curr. Biol. 16, 1096-1102. doi: 10.1016/j.cub.2006.04.003
42. Kassuba, T., Menz, M. M., Röder, B., and Siebner, H. R. (2013). Multisensory interactions between auditory and haptic object recognition. Cereb. Cortex 23, 1097-1107. doi: 10.1093/cercor/bhs076
43. Kastner, S., and Ungerleider, L. G. (2001). The neural basis of biased competition in human visual cortex.Neuropsychologia 39, 1263-1276. doi: 10.1016/s0028-3932(01)00116-6
44. Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., and Rizzolatti, G. (2002). Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846-848. doi: 10.1126/science.1070311
45. Kosslyn, S. M., Thompson, W. L., Kim, I. J., and Alpert, N. M. (1995). Topographical representations of mental images in primary visual cortex. Nature 378, 496-498. doi: 10.1038/378496a0
46. Kriegeskorte, N., and Bandettini, P. (2007). Analyzing for information, not activation, to exploit high-resolution fMRI. Neuroimage 38, 649-662. doi: 10.1016/j.neuroimage.2007.02.022
47. Kriegeskorte, N., Goebel, R., and Bandettini, P. (2006). Information-based functional brain mapping. Proc. Natl. Acad. Sci. U S A 103, 3863-3868. doi: 10.1073/pnas.0600244103
48. Kriegeskorte, N., and Kievit, R. A. (2013). Representational geometry: integrating cognition, computation and the brain. Trends Cogn. Sci. 17, 401-412. doi: 10.1016/j.tics.2013.06.007
49. Kriegeskorte, N., Mur, M., and Bandettini, P. (2008a). Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2:4. doi: 10.3389/neuro.06.004.2008
50. Kriegeskorte, N., Mur, M., Ruff, D. A., Kiani, R., Bodurka, J., Esteky, H., et al. (2008b). Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60, 1126-1141. doi: 10.1016/j.neuron.2008.10.043
51. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S., and Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535-540. doi: 10.1038/nn.2303
52. Man, K., Kaplan, J. T., Damasio, A., and Meyer, K. (2012). Sight and sound converge to form modality-invariant representations in temporoparietal cortex. J. Neurosci. 32, 16629-16636. doi: 10.1523/jneurosci.2342-12.2012
53. Mechelli, A., Price, C. J., Friston, K. J., and Ishai, A. (2004). Where bottom-up meets top-down: neuronal interactions during perception and imagery. Cereb. Cortex 14, 1256-1265. doi: 10.1093/cercor/bhh087
54. Misaki, M., Kim, Y., Bandettini, P. A., and Kriegeskorte, N. (2010). Comparison of multivariate classifiers and response normalizations for pattern-information fMRI. Neuroimage 53, 103-118. doi: 10.1016/j.neuroimage.2010.05.051
55. Mur, M., Bandettini, P. A., and Kriegeskorte, N. (2009). Revealing representational content with pattern-information fMRI-an introductory guide. Soc. Cogn. Affect. Neurosci. 4, 101-109. doi: 10.1093/scan/nsn044
56. Norman, K. A., Polyn, S. M., Detre, G. J., and Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10, 424-430. doi: 10.1016/j.tics.2006.07.005
57. Oosterhof, N. N., Tipper, S. P., and Downing, P. E. (2012a). Viewpoint (in)dependence of action representations: an MVPA study. J. Cogn. Neurosci. 24, 975-989. doi: 10.1162/jocn_a_00195
58. Oosterhof, N. N., Tipper, S. P., and Downing, P. E. (2012b). Visuo-motor imagery of specific manual actions: a multi-variate pattern analysis fMRI study. Neuroimage 63, 262-271. doi: 10.1016/j.neuroimage.2012.06.045
59. Oosterhof, N. N., Tipper, S. P., and Downing, P. E. (2013). Crossmodal and action-specific: neuroimaging the human mirror neuron system. Trends Cogn. Sci. 17, 311-318. doi: 10.1016/j.tics.2013.04.012
60. Oosterhof, N. N., Wiggett, A. J., Diedrichsen, J., Tipper, S. P., and Downing, P. E. (2010). Surface-based information mapping reveals crossmodal vision-action representations in human parietal and occipitotemporal cortex. J. Neurophysiol. 104, 1077-1089. doi: 10.1152/jn.00326.2010
61. Peelen, M. V., Atkinson, A. P., and Vuilleumier, P. (2010). Supramodal representations of perceived emotions in the human brain. J. Neurosci. 30, 10127-10134. doi: 10.1523/jneurosci.2161-10.2010
62. Pereira, F., and Botvinick, M. (2011). Information mapping with pattern classifiers: a comparative study.Neuroimage 56, 476-496. doi: 10.1016/j.neuroimage.2010.05.026
63. Pietrini, P., Furey, M. L., Ricciardi, E., Gobbini, M. I., Wu, W. H., Cohen, L., et al. (2004). Beyond sensory images: object-based representation in the human ventral pathway. Proc. Natl. Acad. Sci. U S A 101, 5658-5663. doi: 10.1073/pnas.0400707101
64. Polyn, S. M., Natu, V. S., Cohen, J. D., and Norman, K. A. (2005). Category-specific cortical activity precedes retrieval during memory search. Science 310, 1963-1966. doi: 10.1126/science.1117645
65. Posner, M. I., Petersen, S. E., Fox, P. T., and Raichle, M. E. (1988). Localization of cognitive operations in the human brain. Science 240, 1627-1631. doi: 10.1126/science.3289116
66. Quadflieg, S., Etzel, J. A., Gazzola, V., Keysers, C., Schubert, T. W., Waiter, G. D., et al. (2011). Puddles, parties and professors: linking word categorization to neural patterns of visuospatial coding. J. Cogn. Neurosci. 23, 2636-2649. doi: 10.1162/jocn.2011.21628
67. Reddy, L., Tsuchiya, N., and Serre, T. (2010). Reading the mind's eye: decoding category information during mental imagery. Neuroimage 50, 818-825. doi: 10.1016/j.neuroimage.2009.11.084
68. Ricciardi, E., Handjaras, G., Bonino, D., Vecchi, T., Fadiga, L., and Pietrini, P. (2013). Beyond motor scheme: a supramodal distributed representation in the action-observation network. PLoS One 8:e58632. doi: 10.1371/journal.pone.0058632
69. Rissman, J., Greely, H. T., and Wagner, A. D. (2010). Detecting individual memories through the neural decoding of memory states and past experience. Proc. Natl. Acad. Sci. U S A 107, 9849-9854. doi: 10.1073/pnas.1001028107
70. Ritchey, M., Wing, E. A., LaBar, K. S., and Cabeza, R. (2013). Neural similarity between encoding and retrieval is related to memory via hippocampal interactions. Cereb. Cortex 23, 2818-2828. doi: 10.1093/cercor/bhs258
71. Rizzolatti, G., and Craighero, L. (2004). The mirror-neuron system. Annu. Rev. Neurosci. 27, 169-192. doi: 10.1146/annurev.neuro.27.070203.144230
72. Saenz, M., Buracas, G. T., and Boynton, G. M. (2002). Global effects of feature-based attention in human visual cortex. Nat. Neurosci. 5, 631-632. doi: 10.1038/nn876
73. Saxe, R., Brett, M., and Kanwisher, N. (2006). Divide and conquer: a defense of functional localizers. Neuroimage30, 1088-1096; discussion 1097-1089. doi: 10.1016/j.neuroimage.2005.12.062
74. Schoenfeld, M. A., Hopf, J. M., Martinez, A., Mai, H. M., Sattler, C., Gasde, A., et al. (2007). Spatio-temporal analysis of feature-based attention. Cereb. Cortex 17, 2468-2477. doi: 10.1093/cercor/bhl154
75. Serences, J. T., and Boynton, G. M. (2007). Feature-based attentional modulations in the absence of direct visual stimulation. Neuron 55, 301-312. doi: 10.1016/j.neuron.2007.06.015
76. Serences, J. T., Ester, E. F., Vogel, E. K., and Awh, E. (2009). Stimulus-specific delay activity in human primary visual cortex. Psychol. Sci. 20, 207-214. doi: 10.1111/j.1467-9280.2009.02276.x
77. Shinkareva, S. V., Malave, V. L., Mason, R. A., Mitchell, T. M., and Just, M. A. (2011). Commonality of neural representations of words and pictures. Neuroimage 54, 2418-2425. doi: 10.1016/j.neuroimage.2010.10.042
78. Simanova, I., Hagoort, P., Oostenveld, R., and van Gerven, M. A. (2014). Modality-independent decoding of semantic information from the human brain. Cereb. Cortex 24, 426-434. doi: 10.1093/cercor/bhs324
79. Spiridon, M., and Kanwisher, N. (2002). How distributed is visual category information in human occipito-temporal cortex? An fMRI study. Neuron 35, 1157-1165. doi: 10.1016/s0896-6273(02)00877-2
80. Stokes, M., Saraiva, A., Rohenkohl, G., and Nobre, A. C. (2011). Imagery for shapes activates position-invariant representations in human visual cortex. Neuroimage 56, 1540-1545. doi: 10.1016/j.neuroimage.2011.02.071
81. Stokes, M., Thompson, R., Cusack, R., and Duncan, J. (2009). Top-down activation of shape-specific population codes in visual cortex during mental imagery. J. Neurosci. 29, 1565-1572. doi: 10.1523/JNEUROSCI.4657-08.2009
82. Todd, M. T., Nystrom, L. E., and Cohen, J. D. (2013). Confounds in multivariate pattern analysis: theory and rule representation case study. Neuroimage 77, 157-165. doi: 10.1016/j.neuroimage.2013.03.039
83. Tong, F., and Pratte, M. S. (2012). Decoding patterns of human brain activity. Annu. Rev. Psychol. 63, 483-509. doi: 10.1146/annurev-psych-120710-100412
84. Uddin, L. Q., Iacoboni, M., Lange, C., and Keenan, J. P. (2007). The self and social cognition: the role of cortical midline structures and mirror neurons. Trends Cogn. Sci. 11, 153-157. doi: 10.1016/j.tics.2007.01.001
85. Vandenberghe, R., Price, C., Wise, R., Josephs, O., and Frackowiak, R. S. (1996). Functional anatomy of a common semantic system for words and pictures. Nature 383, 254-256. doi: 10.1038/383254a0
86. Woo, C. W., Koban, L., Kross, E., Lindquist, M. A., Banich, M. T., Ruzic, L., et al. (2014). Separate neural representations for physical pain and social rejection. Nat. Commun. 5:5380. doi: 10.1038/ncomms6380