Beyond the word and image: Characteristics of a common meaning system for language and vision revealed by functional and structural imaging

NeuroImage

Volumn 106, Issue , 2015, Pages 72-85

  Jouen, A L ^a   Ellmore, T M ^b   Madden, C J ^a   Pallier, C ^c   Dominey, P F ^a   Ventre Dominey, J ^a  

^a STEM CELL AND BRAIN RESEARCH INSTITUTE   (France)

^b NEW YORK UNIVERSITY   (United States)

^c DIF   (France)

Author keywords

DTI; FMRI; Human; Language; Semantic; Vision

Indexed keywords

ADULT; ANTERIOR TEMPORAL GYRUS; ARTICLE; BOLD SIGNAL; BRAIN FUNCTION; BRAIN REGION; COGNITION; COMPREHENSION; CONNECTOME; CONTROLLED STUDY; DEFAULT MODE NETWORK; DIFFUSION TENSOR IMAGING; ELECTROENCEPHALOGRAM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; IMAGE PROCESSING; INFERIOR FRONTAL GYRUS; INFERIOR PARIETAL LOBULE; LANGUAGE; MALE; MIDDLE FRONTAL GYRUS; MIDDLE TEMPORAL GYRUS; NORMAL HUMAN; NUCLEAR MAGNETIC RESONANCE SCANNER; PARAHIPPOCAMPAL GYRUS; PARIETAL CORTEX; PREMOTOR CORTEX; RETROSPLENIAL CORTEX; SEMANTIC MEMORY; SEMANTICS; SENSORIMOTOR FUNCTION; TEMPORAL CORTEX; VENTRAL FRONTAL POLE; VISUAL SYSTEM FUNCTION; YOUNG ADULT; ANATOMY AND HISTOLOGY; BRAIN CORTEX; COMPUTER ASSISTED DIAGNOSIS; NERVE CELL NETWORK; PHYSIOLOGY; PROCEDURES; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; VISION;

CEREBRAL CORTEX; COMPREHENSION; CONNECTOME; FEMALE; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; LANGUAGE; MALE; NERVE NET; REPRODUCIBILITY OF RESULTS; SEMANTICS; SENSITIVITY AND SPECIFICITY; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84913533937     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2014.11.024     Document Type: Article

References (112)

1. Alexander G.E., DeLong M.R., Strick P.L. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 1986, 9:357-381.
2. Andrews-Hanna J.R., Reidler J.S., Sepulcre J., Poulin R., Buckner R.L. Functional-anatomic fractionation of the brain's default network. Neuron 2010, 65:550-562.
3. Anwander A., Tittgemeyer M., von Cramon D.Y., Friederici A.D., Knosche T.R. Connectivity-based parcellation of Broca's area. Cereb. Cortex 2007, 17:816-825.
4. Bajo M.T. Semantic facilitation with pictures and words. J. Exp. Psychol. Learn. Mem. Cogn. 1988, 14:579-589.
5. Bar M. The proactive brain: memory for predictions. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 2009, 364:1235-1243.
6. Barsalou L.W. Perceptual symbol systems. Behav. Brain Sci. 1999, 22:577-609. (discussion 10-60).
7. Barsalou L.W. Grounded cognition. Annu. Rev. Psychol. 2008, 59:617-645.
8. Barsalou L.W. Simulation, situated conceptualization, and prediction. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 2009, 364:1281-1289.
9. Barsalou L.W., Kyle Simmons W., Barbey A.K., Wilson C.D. Grounding conceptual knowledge in modality-specific systems. Trends Cogn. Sci. 2003, 7:84-91.
10. Basser P.J., Pajevic S., Pierpaoli C., Duda J., Aldroubi A. In vivo fiber tractography using DT-MRI data. Magn. Reson. Med. 2000, 44:625-632.
11. Binder J.R., Desai R.H. The neurobiology of semantic memory. Trends Cogn. Sci. 2011, 15:527-536.
12. Binder J.R., Desai R.H., Graves W.W., Conant L.L. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb. Cortex 2009, 19:2767-2796.
13. Boulenger V., Hauk O., Pulvermuller F. Grasping ideas with the motor system: semantic somatotopy in idiom comprehension. Cereb. Cortex 2009, 19:1905-1914.
14. Bright P., Moss H., Tyler L.K. Unitary vs multiple semantics: PET studies of word and picture processing. Brain Lang. 2004, 89:417-432.
15. Burin D.I., Acion L., Kurczek J., Duff M.C., Tranel D., Jorge R.E. The role of ventromedial prefrontal cortex in text comprehension inferences: Semantic coherence or socio-emotional perspective?. Brain Lang. 2014, 129:58-64.
16. Catani M., Jones D.K., ffytche D.H. Perisylvian language networks of the human brain. Ann. Neurol. 2005, 57:8-16.
17. Corbetta M., Patel G., Shulman G.L. The reorienting system of the human brain: from environment to theory of mind. Neuron 2008, 58:306-324.
18. Cox R.W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 1996, 29:162-173.
19. Crosson B., Benjamin M., Levy I. Role of the basal ganglia in language and semantics: Supporting cast 2007, Cambridge University Press, Cambridge (UK).
20. Damasio A.R. Time-locked multiregional retroactivation: a systems-level proposal for the neural substrates of recall and recognition. Cognition 1989, 33:25-62.
21. Desai R.H., Binder J.R., Conant L.L., Seidenberg M.S. Activation of sensory-motor areas in sentence comprehension. Cereb. Cortex 2010, 20:468-478.
22. Dufour N., Redcay E., Young L., Mavros P.L., Moran J.M., et al. Similar brain activation during false belief tasks in a large sample of adults with and without autism. PLoS ONE 2013, 8:e75468.
23. FeldmanHall O., Dalgleish T., Thompson R., Evans D., Schweizer S., Mobbs D. Differential neural circuitry and self-interest in real vs hypothetical moral decisions. Soc. Cogn. Affect. Neurosci. 2012, 7:743-751.
24. Feldmanhall O., Mobbs D., Dalgleish T. Deconstructing the brain's moral network: dissociable functionality between the temporoparietal junction and ventro-medial prefrontal cortex. Soc. Cogn. Affect. Neurosci. 2014, 9:297-306.
25. Fodor J.A. The modularity of mind: an essay on faculty psychology 1983, MIT Press, Cambridge (MA).
26. Frey S., Campbell J.S., Pike G.B., Petrides M. Dissociating the human language pathways with high angular resolution diffusion fiber tractography. J. Neurosci. 2008, 28:11435-11444.
27. Friederici A.D. Pathways to language: fiber tracts in the human brain. Trends Cogn. Sci. 2009, 13:175-181.
28. Friederici A.D., Bahlmann J., Heim S., Schubotz R.I., Anwander A. The brain differentiates human and non-human grammars: functional localization and structural connectivity. Proc. Natl. Acad. Sci. U. S. A. 2006, 103:2458-2463.
29. Friston K. A theory of cortical responses. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2005, 360:815-836.
30. Friston K.J., Worsley K.J., Frackowiak R.S., Mazziotta J.C., Evans A.C. Assessing the significance of focal activations using their spatial extent. Hum. Brain Mapp. 1994, 1:210-220.
31. Gabrieli J.D., Poldrack R.A., Desmond J.E. The role of left prefrontal cortex in language and memory. Proc. Natl. Acad. Sci. U. S. A. 1998, 95:906-913.
32. Gainotti G. Why are the right and left hemisphere conceptual representations different?. Behav. Neurol. 2014, 2014:603134.
33. Gallese V., Fadiga L., Fogassi L., Rizzolatti G. Action recognition in the premotor cortex. Brain 1996, 119(Pt 2):593-609.
34. Glasser M.F., Rilling J.K. DTI tractography of the human brain's language pathways. Cereb. Cortex 2008, 18:2471-2482.
35. Graves W.W., Binder J.R., Desai R.H., Conant L.L., Seidenberg M.S. Neural correlates of implicit and explicit combinatorial semantic processing. NeuroImage 2010, 53:638-646.
36. Harel A., Kravitz D.J., Baker C.I. Deconstructing visual scenes in cortex: gradients of object and spatial layout information. Cereb. Cortex 2013, 23:947-957.
37. Hart J., Maguire M.J., Motes M., Mudar R.A., Chiang H.S., et al. Semantic memory retrieval circuit: role of pre-SMA, caudate, and thalamus. Brain Lang. 2013, 126:89-98.
38. Hauk O., Pulvermuller F. Neurophysiological distinction of action words in the fronto-central cortex. Hum. Brain Mapp. 2004, 21:191-201.
39. Hauk O., Pulvermuller F. The lateralization of motor cortex activation to action-words. Front. Hum. Neurosci. 2011, 5:149.
40. Hauk O., Johnsrude I., Pulvermuller F. Somatotopic representation of action words in human motor and premotor cortex. Neuron 2004, 41:301-307.
41. Hickok G., Poeppel D. Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 2004, 92:67-99.
42. Hodges J.R., Patterson K., Oxbury S., Funnell E. Semantic dementia. Progressive fluent aphasia with temporal lobe atrophy. Brain 1992, 115(Pt 6):1783-1806.
43. Hodges J.R., Graham N., Patterson K. Charting the progression in semantic dementia: implications for the organisation of semantic memory. Memory 1995, 3:463-495.
44. Huth A.G., Nishimoto S., Vu A.T., Gallant J.L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 2012, 76:1210-1224.
45. Jackendoff R. On beyond zebra: the relation of linguistic and visual information. Cognition 1987, 26:89-114.
46. Jefferies E. The neural basis of semantic cognition: converging evidence from neuropsychology, neuroimaging and TMS. Cortex 2013, 49:611-625.
47. Jefferies E., Lambon Ralph M.A. Semantic impairment in stroke aphasia versus semantic dementia: a case-series comparison. Brain 2006, 129:2132-2147.
48. Johnson M.R., Johnson M.K. Decoding individual natural scene representations during perception and imagery. Front. Hum. Neurosci. 2014, 8:59.
49. Johnson-Laird P.N. Cognition, computers, and mental models. Cognition 1981, 10:139-143.
50. Johnson-Laird P.N. The mental representation of the meaning of words. Cognition 1987, 25:189-211.
51. Johnson-Laird P.N. Levels of representation: consciousness and the computational mind. Science 1988, 239:1546-1547.
52. Johnson-Laird P.N. Mental models and human reasoning. Proc. Natl. Acad. Sci. U. S. A. 2010, 107:18243-18250.
53. Keller S.S., Crow T., Foundas A., Amunts K., Roberts N. Broca's area: nomenclature, anatomy, typology and asymmetry. Brain Lang. 2009, 109:29-48.
54. Kiefer M., Pulvermuller F. Conceptual representations in mind and brain: theoretical developments, current evidence and future directions. Cortex 2012, 48:805-825.
55. Kintsch W. The role of knowledge in discourse comprehension: a construction-integration model. Psychol. Rev. 1988, 95:163-182.
56. Kircher T., Sass K., Sachs O., Krach S. Priming words with pictures: neural correlates of semantic associations in a cross-modal priming task using fMRI. Hum. Brain Mapp. 2009, 30:4116-4128.
57. Kohler E., Keysers C., Umilta M.A., Fogassi L., Gallese V., Rizzolatti G. Hearing sounds, understanding actions: action representation in mirror neurons. Science 2002, 297:846-848.
58. Lambon Ralph M.A., Lowe C., Rogers T.T. Neural basis of category-specific semantic deficits for living things: evidence from semantic dementia, HSVE and a neural network model. Brain 2007, 130:1127-1137.
59. Leemans A., Jones D.K. The B-matrix must be rotated when correcting for subject motion in DTI data. Magn. Reson. Med. 2009, 61:1336-1349.
60. Levine B., Turner G.R., Tisserand D., Hevenor S.J., Graham S.J., McIntosh A.R. The functional neuroanatomy of episodic and semantic autobiographical remembering: a prospective functional MRI study. J. Cogn. Neurosci. 2004, 16:1633-1646.
61. Lewis P.A., Critchley H.D., Smith A.P., Dolan R.J. Brain mechanisms for mood congruent memory facilitation. NeuroImage 2005, 25:1214-1223.
62. Lewis P.A., Rezaie R., Brown R., Roberts N., Dunbar R.I. Ventromedial prefrontal volume predicts understanding of others and social network size. NeuroImage 2011, 57:1624-1629.
63. Lichtheim L. On aphasia. Broca's Region 1885, 318-347. 2006, English translation, Oxford University Press, Oxford. Y. Grodzinsky, K. Amunts (Eds.).
64. Lombardo M.V., Chakrabarti B., Bullmore E.T., Wheelwright S.J., Sadek S.A., et al. Shared neural circuits for mentalizing about the self and others. J. Cogn. Neurosci. 2010, 22:1623-1635.
65. Mahon B.Z., Caramazza A. Concepts and categories: a cognitive neuropsychological perspective. Annu. Rev. Psychol. 2009, 60:27-51.
66. Makris N., Pandya D.N. The extreme capsule in humans and rethinking of the language circuitry. Brain Struct. Funct. 2009, 213:343-358.
67. Makris N., Worth A.J., Sorensen A.G., Papadimitriou G.M., Wu O., et al. Morphometry of in vivo human white matter association pathways with diffusion-weighted magnetic resonance imaging. Ann. Neurol. 1997, 42:951-962.
68. Makris N., Kennedy D.N., McInerney S., Sorensen A.G., Wang R., et al. Segmentation of subcomponents within the superior longitudinal fascicle in humans: a quantitative, in vivo, DT-MRI study. Cereb. Cortex 2005, 15:854-869.
69. Makris N., Papadimitriou G.M., Kaiser J.R., Sorg S., Kennedy D.N., Pandya D.N. Delineation of the middle longitudinal fascicle in humans: a quantitative, in vivo, DT-MRI study. Cereb. Cortex 2009, 19:777-785.
70. Man K., Kaplan J.T., Damasio A., Meyer K. Sight and sound converge to form modality-invariant representations in temporoparietal cortex. J. Neurosci. 2012, 32:16629-16636.
71. Man K., Kaplan J., Damasio H., Damasio A. Neural convergence and divergence in the mammalian cerebral cortex: from experimental neuroanatomy to functional neuroimaging. J. Comp. Neurol. 2013, 521:4097-4111.
72. Menjot de Champfleur N., Lima Maldonado I., Moritz-Gasser S., Machi P., Le Bars E., et al. Middle longitudinal fasciculus delineation within language pathways: a diffusion tensor imaging study in human. Eur. J. Radiol. 2013, 82:151-157.
73. Meyer K., Damasio A. Convergence and divergence in a neural architecture for recognition and memory. Trends Neurosci. 2009, 32:376-382.
74. Meyer K., Kaplan J.T., Essex R., Webber C., Damasio H., Damasio A. Predicting visual stimuli on the basis of activity in auditory cortices. Nat. Neurosci. 2010, 13:667-668.
75. Moll J., Zahn R., de Oliveira-Souza R., Bramati I.E., Krueger F., et al. Impairment of prosocial sentiments is associated with frontopolar and septal damage in frontotemporal dementia. NeuroImage 2011, 54:1735-1742.
76. Molnar-Szakacs I., Uddin L.Q. Self-processing and the default mode network: interactions with the mirror neuron system. Front. Hum. Neurosci. 2013, 7:571.
77. Monchi O., Petrides M., Mejia-Constain B., Strafella A.P. Cortical activity in Parkinson's disease during executive processing depends on striatal involvement. Brain 2007, 130:233-244.
78. Nakano K., Kayahara T., Tsutsumi T., Ushiro H. Neural circuits and functional organization of the striatum. J. Neurol. 2000, 247(Suppl. 5):V1-V15.
79. Nichols T., Brett M., Andersson J., Wager T., Poline J.B. Valid conjunction inference with the minimum statistic. NeuroImage 2005, 25:653-660.
80. Nielsen J.A., Zielinski B.A., Ferguson M.A., Lainhart J.E., Anderson J.S. An evaluation of the left-brain vs. right-brain hypothesis with resting state functional connectivity magnetic resonance imaging. PLoS ONE 2013, 8:e71275.
81. Nixon P., Lazarova J., Hodinott-Hill I., Gough P., Passingham R. The inferior frontal gyrus and phonological processing: an investigation using rTMS. J. Cogn. Neurosci. 2004, 16:289-300.
82. Paivio A. Mental representations: a dual coding approach 1986, Oxford University Press, Oxford, UK.
83. Patterson K., Nestor P.J., Rogers T.T. Where do you know what you know? The representation of semantic knowledge in the human brain. Nat. Rev. Neurosci. 2007, 8:976-987.
84. Price C.J., Wise R.J., Warburton E.A., Moore C.J., Howard D., et al. Hearing and saying. The functional neuro-anatomy of auditory word processing. Brain 1996, 119(Pt 3):919-931.
85. Pulvermuller F. Brain mechanisms linking language and action. Nat. Rev. Neurosci. 2005, 6:576-582.
86. Redgrave P., Rodriguez M., Smith Y., Rodriguez-Oroz M.C., Lehericy S., et al. Goal-directed and habitual control in the basal ganglia: implications for Parkinson's disease. Nat. Rev. Neurosci. 2010, 11:760-772.
87. Regev M., Honey C.J., Simony E., Hasson U. Selective and invariant neural responses to spoken and written narratives. J. Neurosci. 2013, 33:15978-15988.
88. Rizzolatti G., Craighero L. The mirror-neuron system. Annu. Rev. Neurosci. 2004, 27:169-192.
89. Rizzolatti G., Fadiga L., Gallese V., Fogassi L. Premotor cortex and the recognition of motor actions. Brain Res. Cogn. Brain Res. 1996, 3:131-141.
90. Rizzolatti G., Fogassi L., Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat. Rev. Neurosci. 2001, 2:661-670.
91. Sarubbo S., De Benedictis A., Maldonado I.L., Basso G., Duffau H. Frontal terminations for the inferior fronto-occipital fascicle: anatomical dissection, DTI study and functional considerations on a multi-component bundle. Brain Struct. Funct. 2013, 218:21-37.
92. Saur D., Kreher B.W., Schnell S., Kümmerer D., Kellmeyer P., et al. Ventral and dorsal pathways for language. Proc. Natl. Acad. Sci. 2008, 105:18035-18040.
93. Saur D., Schelter B., Schnell S., Kratochvil D., Küpper H., et al. Combining functional and anatomical connectivity reveals brain networks for auditory language comprehension. NeuroImage 2010, 49:3187-3197.
94. Scholz J., Triantafyllou C., Whitfield-Gabrieli S., Brown E.N., Saxe R. Distinct regions of right temporo-parietal junction are selective for theory of mind and exogenous attention. PLoS ONE 2009, 4:e4869.
95. Sherbondy A., Akers D., Mackenzie R., Dougherty R., Wandell B. Exploring connectivity of the brain's white matter with dynamic queries. IEEE Trans. Vis. Comput. Graph. 2005, 11:419-430.
96. Spreng R.N., Mar R.A., Kim A.S. The common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: a quantitative meta-analysis. J. Cogn. Neurosci. 2009, 21:489-510.
97. Stephan K.E., Marshall J.C., Friston K.J., Rowe J.B., Ritzl A., et al. Lateralized cognitive processes and lateralized task control in the human brain. Science 2003, 301:384-386.
98. Talairach J., Tournoux P. Co-planar stereotaxic atlas of the human brain 1988, Thieme Medical, New York.
99. Tettamanti M., Buccino G., Saccuman M.C., Gallese V., Danna M., et al. Listening to action-related sentences activates fronto-parietal motor circuits. J. Cogn. Neurosci. 2005, 17:273-281.
100. Thompson-Schill S.L. Neuroimaging studies of semantic memory: inferring "how" from "where". Neuropsychologia 2003, 41:280-292.
101. Turken U., Dronkers N.F. The neural architecture of the language comprehension network: converging evidence from lesion and connectivity analyses. Front. Syst. Neurosci. 2011, 5.
102. van Dam W.O., Rueschemeyer S.A., Bekkering H. How specifically are action verbs represented in the neural motor system: an fMRI study. NeuroImage 2010, 53:1318-1325.
103. Van Doren L., Dupont P., De Grauwe S., Peeters R., Vandenberghe R. The amodal system for conscious word and picture identification in the absence of a semantic task. NeuroImage 2010, 49:3295-3307.
104. Van Doren L., Schrooten M., Adamczuk K., Dupont P., Vandenberghe R. Chronometry of word and picture identification: common and modality-specific effects. NeuroImage 2012, 59:3701-3712.
105. Vandenberghe R., Price C., Wise R., Josephs O., Frackowiak R.S. Functional anatomy of a common semantic system for words and pictures. Nature 1996, 383:254-256.
106. Vandenberghe R., Wang Y., Nelissen N., Vandenbulcke M., Dhollander T., et al. The associative-semantic network for words and pictures: effective connectivity and graph analysis. Brain Lang. 2013, 127:264-272.
107. Vann S.D., Aggleton J.P., Maguire E.A. What does the retrosplenial cortex do?. Nat. Rev. Neurosci. 2009, 10:792-802.
108. Vargha-Khadem F., Gadian D.G., Watkins K.E., Connelly A., Van Paesschen W., Mishkin M. Differential effects of early hippocampal pathology on episodic and semantic memory. Science 1997, 277:376-380.
109. Visser M., Jefferies E., Lambon Ralph M.A. Semantic processing in the anterior temporal lobes: a meta-analysis of the functional neuroimaging literature. J. Cogn. Neurosci. 2010, 22:1083-1094.
110. Whitney C., Kirk M., O'Sullivan J., Lambon Ralph M.A., Jefferies E. The neural organization of semantic control: TMS evidence for a distributed network in left inferior frontal and posterior middle temporal gyrus. Cereb. Cortex 2011, 21:1066-1075.
111. Whitney C., Kirk M., O'Sullivan J., Lambon Ralph M.A., Jefferies E. Executive semantic processing is underpinned by a large-scale neural network: revealing the contribution of left prefrontal, posterior temporal, and parietal cortex to controlled retrieval and selection using TMS. J. Cogn. Neurosci. 2012, 24:133-147.
112. Xu J., Kemeny S., Park G., Frattali C., Braun A. Language in context: emergent features of word, sentence, and narrative comprehension. NeuroImage 2005, 25:1002-1015.