Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures

Neuropsychologia

Volumn 47, Issue 1, 2009, Pages 169-179

  Kircher, Tilo ^a   Straube, Benjamin ^a   Leube, Dirk ^a   Weis, Susanne ^b   Sachs, Olga ^a   Willmes, Klaus ^b   Konrad, Kerstin ^a   Green, Antonia ^a  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b UNIVERSITY HOSPITAL RWTH AACHEN   (Germany)

Author keywords

Communication; Cross modal processing; fMRI; Gestures; Language; Multisensory integration; Speech

Indexed keywords

ADULT; ARTICLE; ELECTROENCEPHALOGRAM; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GESTURE; HUMAN; MALE; NORMAL HUMAN; PREMOTOR CORTEX; SPEECH PERCEPTION; STIMULUS RESPONSE; VERBAL BEHAVIOR;

ACOUSTIC STIMULATION; ADULT; BRAIN; BRAIN MAPPING; COMPREHENSION; FUNCTIONAL LATERALITY; GESTURES; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; OXYGEN; PHOTIC STIMULATION; REACTION TIME; SPEECH; YOUNG ADULT;

EID: 57249097177     PISSN: 00283932     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2008.08.009     Document Type: Article

References (103)

1. Ahrens K., Liu H.L., Lee C.Y., Gong S.P., Fang S.Y., and Hsu Y.Y. Functional MRI of conventional and anomalous metaphors in Mandarin Chinese. Brain Language 100 (2007) 163-171
2. Allison T., Puce A., and McCarthy G. Social perception from visual cues: Role of the STS region. Trends Cognitive Sciences 4 (2000) 267-278
3. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., and Martin A. Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neurosciences 7 (2004) 1190-1192
4. Beauchamp M.S., Lee K.E., Argall B.D., and Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41 (2004) 809-823
5. Beattie G., and Shovelton H. Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology 184 (1999) 438-462
6. Beattie G., and Shovelton H. What properties of talk are associated with the generation of spontaneous iconic hand gestures?. British Journal of Social Psychology 41 (2002) 403-417
7. Beeman M. Coarse semantic coding and discourse comprehension. In: Beeman M., and Chiarello C. (Eds). Right hemisphere language comprehension: Perspectives from cognitive neuroscience (1998), Lawrence Erlbaum Associates, Mahwah, NJ 255-284
8. Beeman M.J. Bilateral brain processes for comprehending natural language. Trends Cognitive Sciences 911 (2005) 512-518
9. Bonda E., Petrides M., Ostry D., and Evans A. Specific involvement of human parietal systems and the amygdala in the perception of biological motion. Journal of Neuroscience 16 (1996) 3737-3744
10. Buccino G., Binkofski F., Fink G.R., Fadiga L., Fogassi L., Gallese V., Seitz R.J., Zilles K., Rizzolatti G., and Freund H.J. Action observation activates premotor and parietal areas in a somatotopic manner: An fMRI study. European Journal of Neurosciences 13 (2001) 400-404
11. Buckner R.L., Goodman J., Burock M., Rotte M., Koutstaal W., Schacter D.L., Rosen B., and Dale A.M. Function-anatomic correlates of object priming in humans revealed by rapid presentation event-related fMRI. Neuron 20 (1998) 285-296
12. Callan D.E., Callan A.M., Kroos C., and Vatikiotis-Bateson E. Multimodal contribution to speech perception revealed by independent component analysis: A single-sweep EEG case study. Brain Research: Cognitive Brain Research 10 (2001) 349-353
13. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., and Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14 (2003) 2213-2218
14. Callan D.E., Jones J.A., Munhall K., Kroos C., Callan A.M., and Vatikiotis-Bateson E. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. Journal of Cognitive Neurosciences 16 (2004) 805-816
15. Calvert G.A., Brammer M.J., Bullmore E.T., Campbell R., Iversen S.D., and David A.S. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10 (1999) 2619-2623
16. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 10 (2000) 649-657
17. Calvert G.A. Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11 (2001) 1110-1123
18. Carr L., Iacoboni M., Dubeau M.C., Mazziotta J.C., and Lenzi G.L. Neural mechanisms of empathy in humans: a relay from neural systems for imitation to limbic areas. Proceedings of the National Academy of Sciences United States of America 100 (2003) 5497-5502
19. Chaminade T., Meltzoff A.N., and Decety J. An fMRI study of imitation: Action representation and body schema. Neuropsychologia 43 (2005) 115-127
20. Corballis M.C. From mouth to hand: Gesture, speech, and the evolution of right-handedness. Behavioral Brain Sciences 26 (2003) 199-208
21. Costantini M., Galati G., Ferretti A., Caulo M., Tartaro A., Romani G.L., and Aglioti S.M. Neural systems underlying observation of humanly impossible movements: An fMRI study. Cerebral Cortex 15 (2005) 1761-1767
22. Decety J., Grezes J., Costes N., Perani D., Jeannerod M., Procyk E., Grassi F., and Fazio F. Brain activity during observation of actions. Infuence of action content and subject's strategy. Brain 120 (1997) 1763-1777
23. Demonet J.F., Thierry G., and Cardebat D. Renewal of the neurophysiology of language: Functional neuroimaging. Physiological Reviews 85 (2005) 49-95
24. Eviatar Z., and Just M.A. Brain correlates of discourse processing: An fMRI investigation of irony and conventional metaphor comprehension. Neuropsychologia 44 (2006) 2348-2359
25. Frens M.A., Van Opstal A.J., and Van der Willigen R.F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Perceptual Psychophysiology 57 (1995) 802-816
26. Fried I., Katz A., McCarthy G., Sass K.J., Williamson P., Spencer S.S., and Spencer D.D. Functional organization of human supplementary motor cortex studied by electrical stimulation. Journal of Neuroscience 11 (1991) 3656-3666
27. Friederici A.D., Opitz B., and von Cramon D.Y. Segregating semantic and syntactic aspects of processing in the human brain: An fMRI investigation of different word types. Cerebral Cortex 10 (2000) 698-705
28. Friston K.J., Fletcher P., Josephs O., Holmes A., Rugg M.D., and Turner R. Event-related fMRI: Characterizing differential responses. Neuroimage 7 (1998) 30-40
29. Friston K.J., Penny W.D., and Glaser D.E. Conjunction revisited. Neuroimage 25 (2005) 661-667
30. Gauger P.W. The effect of gesture and the presence or absence of the speaker on the listening comprehension of eleventh and twelfth grade high school pupils. Speech Monographs 19 (1952) 116-117
31. Gentilucci M., Bernardis P., Crisi G., and Dalla V.R. Repetitive transcranial magnetic stimulation of Broca's area affects verbal responses to gesture observation. Journal of Cognitive Neurosciences 18 (2006) 1059-1074
32. Goldin Meadow S., and Momeni Sandhofer C. Gestures convey substantive information about a child's thoughts to ordinary listeners. Developmental Sciences 21 (1999) 67-74
33. Goldenberg G., Hermsdörfer J., and Laimgruber K. Imitation of gestures by disconnected hemispheres. Neuropsychologia 39 (2001) 1432-1443
34. Graham J.A., and Argyle M. A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology 10 (1975) 57-67
35. Grant K.W., Walden B.E., and Seitz P.F. Auditory-visual speech recognition by hearing-impaired subjects: Consonant recognition, sentence recognition, and auditory-visual integration. Journal of the Acoustical Society of America 103 (1998) 2677-2690
36. Grezes J., Armony J.L., Rowe J., and Passingham R.E. Activations related to "mirror" and "canonical" neurones in the human brain: An fMRI study. Neuroimage 18 (2003) 928-937
37. Grezes J., Costes N., and Decety J. The effects of learning and intention on the neural network involved in the perception of meaningless actions. Brain 122 Pt 10 (1999) 1875-1887
38. Grezes J., and Decety J. Functional anatomy of execution, mental simulation, observation, and verb generation of actions: A meta-analysis. Human Brain Mapping 12 (2001) 1-19
39. Hanlon R., Brown J., and Gerstman L. Enhancement of naming in nonfluent aphasia through gesture. Brain and Language 38 (1990) 298-314
40. Hari R., Forss N., Avikainen S., Kirveskari E., Salenius S., and Rizzolatti G. Activation of human primary motor cortex during action observation: A neuromagnetic study. Proceedings of the National Academy of Sciences United States of America 95 (1998) 15061-15065
41. Hein G., Doehrmann O., Muller N.G., Kaiser J., Muckli L., and Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. Journal of Neuroscience 27 (2007) 7881-7887
42. Hershenson M. Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology 63 (1962) 289-293
43. Hocking J., and Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cerebral Cortex (2008) 10.1093/cercor/bhn007
44. Holle H., and Gunter T.C. The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neurosciences 19 (2007) 1175-1192
45. Holle H., Gunter T.C., Ruschemeyer S.A., Hennenlotter A., and Iacoboni M. Neural correlates of the processing of co-speech gestures. Neuroimage 39 (2008) 2010-2024
46. Hughes H.C., Reuter L.P., Nozawa G., and Fendrich R. Visual-auditory interactions in sensorimotor processing: Saccades versus manual responses. Journal of Experimental Psychology: Human Perception and Performance 20 (1994) 131-153
47. Jeannerod M. Neural simulation of action: A unifying mechanism for motor cognition. Neuroimage 14 (2001) 103-109
48. Kelly S.D., Barr D.J., Church R.B., and Lynch K. Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language 40 (1999) 577-592
49. Kelly S.D., Kravitz C., and Hopkins M. Neural correlates of bimodal speech and gesture comprehension. Brain and Language 89 (2004) 253-260
50. Kircher T.T., Leube D.T., Erb M., Grodd W., and Rapp A.M. Neural correlates of metaphor processing in schizophrenia. Neuroimage 34 (2007) 281-289
51. Laurienti P.J., Burdette J.H., Wallace M.T., Yen Y.F., Field A.S., and Stein B.E. Deactivation of sensory-specific cortex by crossmodal stimuli. Journal of Cognitive Neurosciences 14 (2002) 420-429
52. Lee S.S., and Dapretto M. Metaphorical vs. literal word meanings: fMRI evidence against a selective role of the right hemisphere. NeuroImage 29 (2006) 536-544
53. Levanen S., Uutela K., Salenius S., and Hari R. Cortical representation of sign language: Comparison of deaf signers and hearing non-signers. Cerebral Cortex 11 (2001) 506-512
54. Levelt W.J.M., Richardson G., and la Heij W. Pointing and voicing in deictic expressions. Journal of Memory and Language 24 (1985) 133-164
55. Lewis J.W., Brefczynski J.A., Phinney R.E., Janik J.J., and DeYoe E.A. Distinct cortical pathways for processing tool versus animal sounds. Journal of Neurosciences 25 (2005) 5148-5158
56. Lewis J.W. Cortical networks related to human use of tools. The Neuroscientist 12 (2006) 211-231
57. Lotze M., Heymans U., Birbaumer N., Veit R., Erb M., Flor H., and Halsband U. Differential cerebral activation during observation of expressive gestures and motor acts. Neuropsychologia 44 (2006) 1787-1795
58. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
59. McNeill D. Hand and mind: What gestures reveal about thought (1992), University of Chicago Press, Chicago, IL
60. Meister I.G., Boroojerdi B., Foltys H., Sparing R., Huber W., and Topper R. Motor cortex hand area and speech: Implications for the development of language. Neuropsychologia 41 (2003) 401-406
61. Moro A., Tettamanti M., Perani D., Donati C., Cappa S.F., and Fazio F. Syntax and the brain: Disentangling grammar by selective anomalies. NeuroImage 13 (2001) 110-118
62. Morrell L.K. Temporal characteristics of sensory interaction in choice reaction times. Journal of Experimental Psychology 77 (1968) 14-18
63. Mummery C.J., Shallice T., and Price C.J. Dual-process model in semantic priming: A functional imaging perspective. NeuroImage 9 (1999) 516-525
64. Nichols T., Brett M., Andersson J., Wager T., and Poline J.B. Valid conjunction inference with the minimum statistic. NeuroImage 25 (2005) 653-660
65. Nishitani N., and Hari R. Temporal dynamics of cortical representation for action. Proceedings of the National Academy of Sciences United States of America 97 (2000) 913-918
66. Nishitani N., Schurmann M., Amunts K., and Hari R. Broca's region: From action to language. Physiology 20 (2005) 60-69
67. Nobe S. Where do most spontaneous representational gestures actually occur with respect to speech?. In: McNeill D. (Ed). Language and Gesture (2000), Cambridge University Press, Cambridge 186-198
68. Noppeney U., Josephs O., Hocking J., Price C.J., and Friston K.J. The effect of prior visual information on recognition of speech and sounds. Cerebral Cortex 18 (2008) 598-609
69. Oldfield R.C. The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia 9 (1971) 97-113
70. Olson I.R., Gatenby J.C., and Gore J.C. A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Brain Research: Cognitive Brain Research 14 (2002) 129-138
71. Özyürek A., Willems R.M., Kita S., and Hagoort P. On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience 19 (2007) 605-616
72. Pekkola J., Ojanen V., Autti T., Jaaskelainen I.P., Mottonen R., Tarkiainen A., and Sams M. Primary auditory cortex activation by visual speech: An fMRI study at 3 T. NeuroReport 16 (2005) 125-128
73. Phan K.L., Wager T., Tayler S.F., and Liberzon I. Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI. NeuroImage 16 (2002) 331-348
74. Poldrack R.A., Wagner A.D., Prull M.W., Desmond J.E., Glover G.H., and Gabrieli J.D. Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. Neuroimage 10 (1999) 15-35
75. Pulvermüller F. Brain mechanisms linking language and action. Nature Review Neurosciences 6 (2005) 576-582
76. Price C.J. The anatomy of language: Contributions from functional neuroimaging. Journal of Anatomy 197 (2000) 335-359
77. Rapp A.M., Leube D.T., Erb M., Grodd W., and Kircher T.T.J. Neural correlates of metaphor processing. Cognitive Brain Reseach 20 (2004) 395-402
78. Rissman J., Eliassen J.C., and Blumstein S.E. An event-related fMRI investigation of implicit semantic priming. Journal of Cognitive Neuroscience 15 (2003) 1160-1175
79. Rizzolatti G., Fogassi L., and Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Reviews Neuroscience 2 (2001) 661-670
80. Rose M., and Douglas J. The differential facilitatory effects of gesture and visualisation processes on object naming in aphasia. Aphasiology 15 (2001) 977-990
81. Roskies A.L., Fiez J.A., Balota D.A., Raichle M.E., and Petersen S.E. Task-dependent modulation of regions in the left inferior frontal cortex during semantic processing. Journal of Cognitive Neuroscience 13 (2001) 829-843
82. Sachs O., Weis S., Krings T., Huber W., and Kircher T. Categorical and thematic knowledge representation in the brain: Neural correlates of taxonomic and thematic conceptual relations. Neuropsychologia 46 (2008) 409-418
83. Sachs O., Weis S., Zellagui N., Huber W., Zvyagintsev M., Mathiak K., and Kircher T. Automatic processing of semantic relations in fMRI: Neural activation during semantic priming of taxonomic and thematic categories. Brain Research 1218 (2008) 194-205
84. Schacter D.L., and Buckner R.L. Priming and the brain. Neuron 20 (1998) 185-195
85. Sekiyama K., Kanno I., Miura S., and Sugita Y. Auditory-visual speech perception examined by fMRI and PET. Neuroscience Research 47 (2003) 277-287
86. Skipper J.I., Goldin-Meadow S., Nusbaum H.C., and Small S.L. Speech-associated gestures, Broca's area, and the human mirror system. Brain and Language 101 (2007) 260-277
87. Slotnick S.D., Moo L.R., Segal J.B., and Hart Jr. J. Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Brain Research: Cognitive Brain Research 17 (2003) 75-82
88. Stein B.E., Meredith M.A., Huneycutt W.S., and McDade L. Behavioural indices of multisensory integration: Orientation to visual cues is affected by auditory stimuli. Journal of Cognitive Neuroscience 1 (1989) 12-24
89. Straube B., Green A., Weis S., Chatterjee A., and Kircher T. Memory effects of speech and gesture binding: Cortical and hippocampal activation in relation to subsequent memory performance. Journal of Cognitive Neuroscience (2008) 10.1162/jocn.2009.21053
90. Stringaris A.K., Medford N.C., Giampietro V., Brammer M.J., and David A.S. Deriving meaning: Distinct neural mechanisms for metaphoric, literal, and non-meaningful sentences. Brain and Language 100 (2007) 150-162
91. Sumby W., and Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26 (1954) 212-215
92. Talairach J., and Tournoux P. Co-planar stereotaxic atlas of the human brain (1988), Thieme Medical, New York
93. Valenzeno L., Alibali M.W., and Klatzkya R. Teacher's gestures facilitate students learning: A lesson in symmetry. Contemporary Educational Psychology 28 (2003) 187-204
94. Wheatley T., Weisberg J., Beauchamp M.S., and Martin A. Automatic priming of semantically related words reduces activity in the fusiform gyrus. Journal of Cognitive Neuroscience 17 (2005) 1871-1885
95. Wiggs C.L., and Martin A. Properties and mechanisms of perceptual priming. Current Opinion in Neurobiology 8 (1998) 227-233
96. Willems R.M., Ozyurek A., and Hagoort P. When language meets action: The neural integration of gesture and speech. Cerebral Cortex 17 (2007) 2322-2333
97. Willems R.M., and Hagoort P. Neural evidence for the interplay between language, gesture, and action: A review. Brain and Language 101 (2007) 278-289
98. Wilson S.M., Saygin A.P., Sereno M.I., and Iacoboni M. Listening to speech activates motor areas involved in speech production. Nature Neuroscience 7 (2004) 701-702
99. Wicker B., Keysers C., Plailly J., Royet J.P., Gallese V., and Rizzolatti G. Both of us disgusted in my insula: The common neural basis of seeing and feeling disgust. Neuron 40 (2003) 655-664
100. Wise R.J., Greene J., Buchel C., and Scott S.K. Brain regions involved in articulation. Lancet 353 (1999) 1057-1061
101. Wright T.M., Pelphrey K.A., Allison T., McKeown M.J., and McCarthy G. Polysensory interactions along temporal regions evoked by audiovisual speech. Cerebral Cortex 13 (2003) 1034-1043
102. Wu Y.C., and Coulson S. Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology 42 (2005) 654-667
103. Wu Y.C., and Coulson S. Iconic gestures prime related concepts: An ERP study. Psychonomic Bulletin & Review 14 (2007) 57-63