Modality-independent decoding of semantic information from the human brain

Cerebral Cortex

Volumn 24, Issue 2, 2014, Pages 426-434

  Simanova, Irina ^a   Hagoort, Peter ^a,b   Oostenveld, Robert ^a   Van Gerven, Marcel A J ^c  

^a RADBOUD UNIVERSITY   (Netherlands)

^b MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^c RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Amodal; FMRI; Multivoxel pattern analysis; Searchlight; Semantics

Indexed keywords

ADULT; ARTICLE; BRAIN REGION; CLASSIFICATION; FEMALE; FRONTAL CORTEX; HUMAN; HUMAN EXPERIMENT; INFERIOR TEMPORAL CORTEX; MALE; MEDICAL INFORMATION; MEDICAL PHOTOGRAPHY; NOMENCLATURE; NORMAL HUMAN; OBJECT RELATION; PRIORITY JOURNAL; RECALL; SEMANTIC MEMORY; SOUND; STIMULUS RESPONSE; SUPPORT VECTOR MACHINE; TASK PERFORMANCE; TRAINING;

AMODAL; FMRI; MULTIVOXEL PATTERN ANALYSIS; SEARCHLIGHT; SEMANTICS;

ADOLESCENT; ADULT; AUDITORY PERCEPTION; BRAIN; BRAIN MAPPING; FEMALE; HUMANS; IMAGINATION; JUDGMENT; MAGNETIC RESONANCE IMAGING; MALE; MENTAL RECALL; NAMES; READING; SEMANTICS; SIGNAL PROCESSING, COMPUTER-ASSISTED; SPEECH PERCEPTION; SUPPORT VECTOR MACHINES; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84896365152     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhs324     Document Type: Article

References (66)

1. Adams RB, Janata P. 2002. A comparison of neural circuits underlying auditory and visual object categorization. Neuroimage. 16:361-377.
2. Alexander MP. 1997. Aphasia: clinical and anatomic aspects. In: Feinberg TE, Farah ME, editors. Behavioral neurology and neuropsychology. New York: McGraw-Hill. p. 133-149.
3. Beauchamp MS, Lee KE, Argall BD, Martin A. 2004. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron. 41:809-823.
4. Binder JR, Desai RH. 2011. The neurobiology of semantic memory. Trends Cogn Sci. 15:527-536.
5. Binder JR, Desai RH, Graves WW, Conant LL. 2009. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb Cortex. 19:2767-2796.
6. Birn RM, Kenworthy L, Case L, Caravella R, Jones TB, Bandettini PA, Martin A. 2010. Neural systems supporting lexical search guided by letter and semantic category cues: a self-paced overt response fMRI study of verbal fluency. Neuroimage. 49:1099-1107.
7. Bright P, Moss H, Tyler LK. 2004. Unitary vs multiple semantics: PET studies of word and picture processing. Brain Lang. 89:417-432.
8. Brumberg JS, Nieto-Castanon A, Kennedy PR, Guenther FH. 2010. Brain-computer interfaces for speech communication. Speech Commun. 52:367-379.
9. Buckner RL, Koutstaal W, Schacter DL, Rosen BR. 2000. Functional MRI evidence for a role of frontal and inferior temporal cortex in amodal components of priming. Brain. 123:620-640.
10. Chao LL, Haxby JV, Martin A. 1999. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat Neurosci. 2:913-919.
11. Chao LL, Weisberg J, Martin A. 2002. Experience-dependent modulation of category-related cortical activity. Cereb Cortex. 12:545-551.
12. Cox DD, Savoy RL. 2003. Functional magnetic resonance imaging (fMRI) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage. 19:261-270.
13. Dasalla CS, Kambara H, Sato M, Koike Y. 2009. Single-trial classification of vowel speech imagery using common spatial patterns. Neural Netw. 22:1334-1339.
14. Devlin JT, Rushworth MF, Matthews PM. 2005. Category-related activation for written words in the posterior fusiform is task specific. Neuropsychologia. 43:69-74.
15. Doehrmann O, Naumer MJ. 2008. Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res. 1242:136-150.
16. Doehrmann O, Naumer MJ, Volz S, Kaiser J, Altmann CF. 2008. Probing category selectivity for environmental sounds in the human auditory brain. Neuropsychologia. 46:2776-2786.
17. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K. 2005. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage. 25:1325-1335.
18. Formisano E, De Martino F, Valente G. 2008. Multivariate analysis of fMRI time series: classification and regression of brain responses using machine learning. Magn Reson Imaging. 26:921-934.
19. Freedman M, Alexander MP, Naeser MA. 1984. Anatomic basis of transcortical motor aphasia. Neurology. 34:409-417.
20. Hagoort P, Baggio G, Willems RM. 2009. Semantic unification. In: Gazzaniga MS, editor. The Cognitive Neurosciences. Boston (MA): MIT Press. p. 819-836.
21. Hauk O, Davis MH, Kherif F, Pulvermüller F. 2008. Imagery or meaning? Evidence for a semantic origin of category-specific brain activity in metabolic imaging. Eur J Neurosci. 27:1856-1866.
22. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 293:2425-2430.
23. Haynes JD, Rees G. 2006. Decoding mental states from brain activity in humans. Nat Rev Neurosci. 7:523-534.
24. Hickok G, Poeppel D. 2007. The cortical organization of speech processing. Nat Rev Neurosci. 8:393-402.
25. Just MA, Cherkassky VL, Aryal S, Mitchell TM. 2010. A neurosemantic theory of concrete noun representation based on the underlying brain codes. PLoS One. 5:e8622.
26. Kamitani Y, Tong F. 2005. Decoding the visual and subjective contents of the human brain. Nat Neurosci. 8:679-685.
27. Kircher T, Sass K, Sachs O, Krach S. 2009. Priming words with pictures: neural correlates of semantic associations in a cross-modal priming task using fMRI. Hum Brain Mapp. 30:4116-4128.
28. Kriegeskorte N, Goebel R, Bandettini P. 2006. Information-based functional brain mapping. Proc Natl Acad Sci USA. 103: 3863-3868.
29. Lewis JW, Brefczynski JA, Phinney RE, Janik JJ, DeYoe EA. 2005. Distinct cortical pathways for processing tool versus animal sounds. J Neurosci. 25:5148-5158.
30. Luria AR, Tsvetkova LS. 1968. The mechanism of "dymanic aphasia". Foundations of Language. 4:296-307.
31. Mahon BZ, Caramazza A. 2009. Concepts and categories: a cognitive neuropsychological perspective. Annu Rev Psychol. 60:27-51.
32. Mandler JM. 2004. Thought before language. Trends Cogn Sci. 8:508-513.
33. Marinkovic K, Dhond RP, Dale AM, Glessner M, Carr V, Halgren E. 2003. Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron. 38:487-497.
34. Martin A. 2007. The representation of object concepts in the brain. Annu Rev Psychol. 58:25-45.
35. Martin A, Chao LL. 2001. Semantic memory and the brain: structure and processes. Curr Opin Neurobiol. 11:194-201.
36. Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA. 2008. Predicting human brain activity associated with the meanings of nouns. Science. 320(5880):1191-1195.
37. Moore CJ, Price CJ. 1999. A functional neuroimaging study of the variables that generate category-specific object processing differences. Brain. 122:943-962.
38. Naumer MJ, Doehrmann O, Müller NG, Muckli L, Kaiser J, Hein G. 2009. Cortical plasticity of audio-visual object representations. Cereb Cortex. 19:1641-1653.
39. Noppeney U, Price CJ, Penny WD, Friston KJ. 2006. Two distinct neural mechanisms for category-selective responses. Cereb Cortex. 16:437-445.
40. Oostenveld R, Fries P, Maris E, Schoffelen JM. 2011. FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci. 2011:156869.
41. Op de Beeck HP, Haushofer J, Kanwisher NG. 2008. Interpreting fMRI data: maps, modules and dimensions. Nat Rev Neurosci. 9:123-135.
42. Patterson K, Nestor PJ, Rogers TT. 2007. Where do you know what you know? The representation of semantic knowledge in the human brain. Nat Rev Neurosci. 8:976-987.
43. Pereira F, Mitchell T, Botvinick M. 2009. Machine learning classifiers and fMRI: a tutorial overview. Neuroimage. 45:S199-S209.
44. Pobric G, Jefferies E, Lambon Ralph MA. 2010. Category-Specific versus Category-General Semantic Impairment Induced by Transcranial Magnetic Stimulation. Curr Biology. 20:964-968.
45. Poser BA, Versluis MJ, Hoogduin JM, Norris DG. 2006. BOLD contrast sensitivity enhancement and artifact reduction with multiecho EPI: parallel-acquired inhomogeneity-desensitized fMRI. Magn Reson Med. 55:1227-1235.
46. Price CJ. 2010. The anatomy of language: a review of 100 fMRI studies published in 2009. Ann N Y Acad Sci. 1191:62-88.
47. Price CJ. 2004. An Overview of Speech Comprehension and Production. In: Frackowiak RSJ, Friston KJ, editors. Human brain function. Amsterdam, Boston: Elsevier Academic Press. p. 517-533.
48. Price CJ, Winterburn D, Giraud AL, Moore CJ, Noppeney U. 2003. Cortical localisation of the visual and auditory word form areas: a reconsideration of the evidence. Brain Lang. 86:272-286.
49. Reddy L, Kanwisher N. 2007. Category selectivity in the ventral visual pathway confers robustness to clutter and diverted attention. Curr Biol. 17:2067-2072.
50. Riddoch MJ, Humphreys GW, Coltheart M, Funnell E. 1988. Semantic systems or system? Neuropsychological evidence re-examined. Cogn Neuropsychol. 5:3-25.
51. Robinson G, Blair J, Cipolotti L. 1998. Dynamic aphasia: an inability to select between competing verbal responses? Brain. 121(Pt 1):77-89.
52. Schneider TR, Engel AK, Debener S. 2008. Multisensory identification of natural objects in a two-way crossmodal priming paradigm. Exp Psychol. 55:121-132.
53. Shinkareva SV, Malave VL, Mason RA, Mitchell TM, Just MA. 2011. Commonality of neural representations of words and pictures. Neuroimage. 54:2418-2425.
54. Shinkareva SV, Mason RA, Malave VL, Wang W, Mitchell TM, Just MA. 2008. Using FMRI brain activation to identify cognitive states associated with perception of tools and dwellings. PLoS One. 3: e1394.
55. Snijders TM, Vosse T, Kempen G, Van Berkum JJ, Petersson KM, Hagoort P. 2009. Retrieval and unification of syntactic structure in sentence comprehension: an FMRI study using word-category ambiguity. Cereb Cortex. 19:1493-1503.
56. Tranel D, Damasio H, Eichhorn GR, Grabowski T, Ponto LL, Hichwa RD. 2003. Neural correlates of naming animals from their characteristic sounds. Neuropsychologia. 41:847-854.
57. Tranel D, Grabowski TJ, Lyon J, Damasio H. 2005. Naming the same entities from visual or from auditory stimulation engages similar regions of left inferotemporal cortices. J Cogn Neurosci. 17:1293-1305.
58. Van Doren L, Dupont P, De Grauwe S, Peeters R, Vandenberghe R. 2010. The amodal system for conscious word and picture identification in the absence of a semantic task. Neuroimage. 49:3295-3307.
59. Vandenberghe R, Price C, Wise R, Josephs O, Frackowiak RS. 1996. Functional anatomy of a common semantic system for words and pictures. Nature. 383:254-256.
60. Vandenbulcke M, Peeters R, Dupont P, Van Hecke P, Vandenberghe R. 2007. Word reading and posterior temporal dysfunction in amnestic mild cognitive impairment. Cereb Cortex. 17:542-551.
61. van Gerven M, Farquhar J, Schaefer R, Vlek R, Geuze J, Nijholt A, Ramsey N, Haselager P, Vuurpijl L, Gielen S et al. 2009. The braincomputer interface cycle. J Neural Eng. 6:041001.
62. Vapnik VN. 2000. The nature of statistical learning theory. Statistics for Engineering and Information Science. New York: Springer.
63. Vindiola M, Wolmetz M. 2011. Mental encoding and neural decoding of abstract cognitive categories: a commentary and simulation. Neuroimage. 54:2822-2827.
64. Wei T, Liang X, He Y, Zang Y, Han Z, Caramazza A, Bi Y. 2012. Predicting conceptual processing capacity from spontaneous neuronal activity of the left middle temporal gyrus. J Neurosci. 32:481-489.
65. Wheatley T, Weisberg J, Beauchamp MS, Martin A. 2005. Automatic priming of semantically related words reduces activity in the fusiform gyrus. J Cogn Neurosci. 17:1871-1885.
66. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. 2002. Brain-computer interfaces for communication and control. Clin Neurophysiol. 113:767-91.