When language meets action: The neural integration of gesture and speech

Cerebral Cortex

Volumn 17, Issue 10, 2007, Pages 2322-2333

  Willems, Roel M ^a   Özyürek, Asli ^a,b   Hagoort, Peter ^a,c  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^b MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^c RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Broca's area; fMRI; Gestures; Inferior frontal cortex; Semantic processing

Indexed keywords

ADULT; ARTICLE; BRAIN REGION; CONTROLLED STUDY; DIAGONAL BAND OF BROCA; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GESTURE; HUMAN; HUMAN EXPERIMENT; LANGUAGE ABILITY; LANGUAGE PROCESSING; MALE; NERVE CELL NETWORK; NEUROIMAGING; NORMAL HUMAN; PREMOTOR CORTEX; PRIORITY JOURNAL; SEMANTICS; SPEECH; VERBALIZATION;

ADULT; BRAIN MAPPING; CEREBRAL CORTEX; FEMALE; GESTURES; HUMANS; LANGUAGE; MAGNETIC RESONANCE IMAGING; MALE; MOTOR ACTIVITY; SEMANTICS; SPEECH;

EID: 34948883257     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhl141     Document Type: Article

References (100)

1. Aboitiz F, Garcia R. 1997. The evolutionary origin of the language areas in the human brain: a neuroanatomical perspective. Brain Res Rev. 25(3):381-396.
2. Aboitiz F, Garcia RR, Bosman C, Brunetti E. 2006. Cortical memory mechanisms and language origins. Brain Lang. 98(1):40-56.
3. Amunts K, Schleicher A, Burgel U, Mohlberg H, Uylings HB, Zilles K. 1999. Broca's region revisited: cytoarchitecture and intersubject variability. J Comp Neurol. 412(2):319-341.
4. Arbib MA. 2005. From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics. Behav Brain Sci. 28(2):105-124.
5. Badre D, Poldrack RA, Pare-Blagoev EJ, Insler RZ, Wagner AD. 2005. Dissociable controlled retrieval and generalized selection mechanisms in ventrolateral prefrontal cortex. Neuron. 47(6):907-918.
6. Beattie G, Shovelton H. 1999. Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. J Lang Soc Psychol. 18(4):438-462.
7. Bernardis P, Gentilucci M. 2006. Speech and gesture share the same communication system. Neuropsychologia. 44(2):178-190.
8. Bookheimer S. 2002. Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. Annu Rev Neurosci. 25:151-188.
9. Bosman C, Lopez V, Aboitiz F. 2005. Sharpening Occam's razor: is there need for a hand-signing stage prior to vocal communication? Behav Brain Sci. 28(2):128-129.
10. Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ. 2001. Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci. 13(2):400-404.
11. Buccino G, Vogt S, Ritzl A, Fink GR, Zilles K, Freund HJ, Rizzolatti G. 2004. Neural circuits underlying imitation learning of hand actions: an event-related fMRI study. Neuron. 42(2):323-334.
12. Butterworth G, Shovelton H. 1978. Gesture and silence as indicators of planning in speech. In: Campbell N, Smith P, editors. Recent advances in the psychology of language. New York: Plenum. p. 347-360.
13. Calvert GA. 2001. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex. 11(12):1110-1123.
14. Cassell J, McNeill D, McCullough KE. 1999. Speech-gesture mismatches: evidence for one underlying representation of linguistic and non-linguistic information. Pragmatics Cogn. 7(1):1-34.
15. Church RB, Goldin Meadow S. 1986. The mismatch between gesture and speech as an index of transitional knowledge. Cognition. 23(1):43-71.
16. Clark HH. 1996. Using language. New York, NY: Cambridge University Press.
17. Corbetta M, Shulman GL. 2002. Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci. 3(3):201-215.
18. Corina DP, Knapp H. 2006. Sign language processing and the mirror neuron system. Cortex. 42(4):529-539.
19. Costantini M, Galati G, Ferretti A, Caulo M, Tartaro A, Romani GL, Aglioti SM. 2005. Neural systems underlying observation of humanly impossible movements: an fMRI study. Cereb Cortex. 15(11):1761-1767.
20. Dale AM. 1999. Optimal experimental design for event-related fMRI. Hum Brain Mapp. 8(2-3):109-114.
21. de Araujo IE, Rolls ET, Velazco MI, Margot C, Cayeux I. 2005. Cognitive modulation of olfactory processing. Neuron. 46(4):671-679.
22. de Lange FP, Hagoort P, Toni I. 2005. Neural topography and content of movement representations. J Cogn Neurosci. 17(1):97-112.
23. Decety J, Grezes J, Costes N, Perani D, Jeannerod M, Procyk E, Grassi F, Fazio F. 1997. Brain activity during observation of actions. Influence of action content and subject's strategy. Brain. 120(Pt 10):1763-1777.
24. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K. 2005. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage. 25(4):1325-1335.
25. Emmorey K. 2006. The role of Broca's area in sign language. In: Grodzinsky Y, Amunts K, editors. Broca's region. New York: Oxford University Press. p. 169-184.
26. Fadiga L, Fogassi L, Pavesi G, Rizzolatti G. 1995. Motor facilitation during action observation: a magnetic stimulation study. J Neurophysiol. 73(6):2608-2611.
27. Forman SD, Cohen JD, Fitzgerald M, Eddy WF, Mintun MA, Noll DC. 1995. Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn Reson Med. 33(5):636-647.
28. Friederici AD, Rüschemeyer S, Hahne Aü, Fiebach CJ. 2003. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cereb Cortex. 13:170-177.
29. Friston KJ, Fletcher P, Josephs O, Holmes A, Rugg MD, Turner R. 1998. Event-related fMRI: characterizing differential responses. Neuroimage. 7(1):30-40.
30. Fuster JM. 2003. Cortex and mind. New York: Oxford University Press. Gallagher HL, Frith CD. 2004. Dissociable neural pathways for the perception and recognition of expressive and instrumental gestures. Neuropsychologia. 42(13):1725-1736.
31. Goldin Meadow S. 2003. Hearing gesture: how our hands help us think. Cambridge, MA: Belknap Press of Harvard University Press.
32. Goldin Meadow S, Kim S, Singer M. 1999. What the teacher's hands tell the student's mind about math. J Educ Psychol. 91(4):720-730.
33. Goldin Meadow S, Momeni Sandhofer C. 1999. Gestures convey substantive information about a child's thoughts to ordinary listeners. Dev Sci. 2(1):67-74.
34. Grezes J, Armony JL, Rowe J, Passingham RE. 2003. Activations related to "mirror" and "canonical" neurones in the human brain: an fMRI study. Neuroimage. 18(4):928-937.
35. Grezes J, Costes N, Decety J. 1999. The effects of learning and intention on the neural network involved in the perception of meaningless actions. Brain. 122(Pt 10):1875-1887.
36. Hagoort P. 2003. How the brain solves the binding problem for language: a neurocomputational model of syntactic processing. Neuroimage. 20(Suppl 1):S18-S29.
37. Hagoort P. 2005a. Broca's complex as the unification space for language. In: Cutler A, editor. Twenty first century psycholinguistics: four cornerstones. Mahwah, NJ: Lawrence Erlbaum Associates Publishers. p. 157-172.
38. Hagoort P. 2005b. On Broca, brain, and binding: a new framework. Trends Cogn Sci. 9(9):416-423.
39. Hagoort P, Brown C. 1994. Brain responses to lexical ambiguity resolution and parsing. In: Frazier L, Clifton Charles J, Rayner K, editors. Perspectives in sentence processing. Hillsdale, NJ, UK: Lawrence Erlbaum Associates. p. 45-80.
40. Hagoort P, Hald L, Bastiaansen M, Petersson KM. 2004. Integration of word meaning and world knowledge in language comprehension. Science. 304(5669):438-441.
41. Hamzei F, Rijntjes M, Dettmers C, Glauche V, Weiller C, Buchel C. 2003. The human action recognition system and its relationship to Broca's area: an fMRI study. Neuroimage. 19(3):637-644.
42. Hari R, Forss N, Avikainen S, Kirveskari E, Salenius S, Rizzolatti G. 1998. Activation of human primary motor cortex during action observation: a neuromagnetic study. Proc Natl Acad Sci USA. 95(25):15061-15065.
43. Holle H, Gunter TC. 2006. Dynamic co-speech gestures may be used as disambiguation cues: ERP evidence. J Cogn Neurosci (Suppl). 18:221.
44. Iacoboni M, Woods RP, Brass M, Bekkering H, Mazziotta JC, Rizzolatti G. 1999. Cortical mechanisms of human imitation. Science. 286(5449):2526-2528.
45. Jeannerod M. 2001. Neural simulation of action: a unifying mechanism for motor cognition. Neuroimage. 14(1 Pt 2):S103-S109.
46. Josephs O, Turner R, Friston K. 1997. Event-related fMRI. Hum Brain Mapp. 5:243-248.
47. Kelly SD, Barr DJ, Church RB, Lynch K. 1999. Offering a hand to pragmatic understanding: the role of speech and gesture in comprehension and memory. J Mem Lang. 40(4):577-592.
48. Kelly SD, Church RB. 1998. A comparison between children's and adults' ability to detect conceptual information conveyed through representational gestures. Child Dev. 69(1):85-93.
49. Kelly SD, Kravitz C, Hopkins M. 2004. Neural correlates of bimodal speech and gesture comprehension. Brain Lang. 89(1):253-260.
50. Kendon A. 2004. Gesture: visible action as utterance. Cambridge, UK: Cambridge University Press.
51. Kita S, Özyürek A. 2003. What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: evidence for an interface representation of spatial thinking and speaking. J Mem Lang. 48(1):16-32.
52. Koski L, Wohlschlager A, Bekkering H, Woods RP, Dubeau MC, Mazziotta JC, Iacoboni M. 2002. Modulation of motor and premotor activity during imitation of target-directed actions. Cereb Cortex. 12(8):847-855.
53. Krauss RM, Morrel Samuels P, Colasante C. 1991. Do conversational hand gestures communicate? J Pers Soc Psychol. 61(5):743-754.
54. Kuperberg GR, Holcomb PJ, Sitnikova T, Greve D, Dale AM, Caplan D. 2003. Distinct patterns of neural modulation during the processing of conceptual and syntactic anomalies. J Cogn Neurosci. 15(2):272-293.
55. Kuperberg GR, McGuire PK, Bullmore ET, Brammer MJ, Rabe-Hesketh S, Wright IC, Lythgoe DJ, Williams SC, David AS. 2000. Common and distinct neural substrates for pragmatic, semantic, and syntactic processing of spoken sentences: an fMRI study. J Cogn Neurosci. 12(2):321-341.
56. Kutas M, Hillyard SA. 1980. Reading senseless sentences: brain potentials reflect semantic incongruity. Science. 207(4427):203-205.
57. Kutas M, Hillyard SA. 1984. Brain potentials during reading reflect word expectancy and semantic association. Nature. 307(5947):161-163.
58. Kutas M, Van Petten CK. 1994. Psycholinguistics electrified: eventrelated brain potential investigations. In: Gernsbacher MA, editor. Handbook of psycholinguistics. San Diego, CA: Academic Press. p. 83-143.
59. MacSweeney M, Campbell R, Woll B, Brammer MJ, Giampietro V, David AS, Calvert GA, McGuire PK. 2006. Lexical and sentential processing in British Sign Language. Hum Brain Mapp. 27(1):63-76.
60. MacSweeney M, Campbell R, Woll B, Giampietro V, David AS, McGuire PK, Calvert GA, Brammer MJ. 2004. Dissociating linguistic and nonlinguistic gestural communication in the brain. Neuroimage. 22(4):1605-1618.
61. MacSweeney M, Woll B, Campbell R, McGuire PK, David AS, Williams SC, Suckling J, Calvert GA, Brammer MJ. 2002. Neural systems underlying British Sign Language and audio-visual English processing in native users. Brain. 125(Pt 7):1583-1593.
62. McNeill D. 1992. Hand and mind: what gestures reveal about thought. Chicago, IL: University of Chicago Press.
63. McNeill D. 2000. Language and gesture. Cambridge, UK: Cambridge University Press.
64. McNeill D, Cassell J, McCullough KE. 1994. Communicative effects of speech-mismatched gestures. Res Lang Soc Interact. 27(3):223-237.
65. Melinger A, Levelt WJM. 2004. Gesture and the communicative intention of the speaker. Gesture. 4(2):119-141.
66. Mesulam MM. 1990. Large-scale neurocognitive networks and distributed processing for attention, language, and memory. Ann Neurol. 28(5):597-613.
67. Mesulam MM. 1998. From sensation to cognition. Brain. 121(Pt 6):1013-1052.
68. Miezin FM, Maccotta L, Ollinger JM, Petersen SE, Buckner RL. 2000. Characterizing the hemodynamic response: effects of presentation rate, sampling procedure, and the possibility of ordering brain activity based on relative timing. Neuroimage. 11(6 Pt 1):735-759.
69. Molnar-Szakacs I, Iacoboni M, Koski L, Mazziotta JC. 2005. Functional segregation within pars opercularis of the inferior frontal gyrus: evidence from fMRI studies of imitation and action observation. Cereb Cortex. 15(7):986-994.
70. Morrel Samuels P, Krauss RM. 1992. Word familiarity predicts temporal asynchrony of hand gestures and speech. J Exp Psychol Learn Mem Cogn. 18(3):615-622.
71. Nakamura A, Maess B, Knosche TR, Gunter TC, Bach P, Friederici AD. 2004. Cooperation of different neuronal systems during hand sign recognition. Neuroimage. 23(1):25-34.
72. Neville HJ, Bavelier D, Corina D, Rauschecker J, Karni A, Lalwani A, Braun A, Clark V, Jezzard P, Turner R. 1998. Cerebral organization for language in deaf and hearing subjects: biological constraints and effects of experience. Proc Natl Acad Sci USA. 95(3):922-929.
73. Newman AJ, Bavelier D, Corina D, Jezzard P, Neville HJ. 2002. A critical period for right hemisphere recruitment in American Sign Language processing. Nat Neurosci. 5(1):76-80.
74. Ni W, Constable RT, Mencl WE, Pugh KR, Fulbright RK, Shaywitz SE, Shaywitz BA, Gore JC, Shankweiler D. 2000. An event-related neuroimaging study distinguishing form and content in sentence processing. J Cogn Neurosci. 12(1):120-133.
75. Nichols T, Brett M, Andersson J, Wager T, Poline JB. 2005. Valid conjunction inference with the minimum statistic. Neuroimage. 25(3):653-660.
76. Nishitani N, Hari R. 2000. Temporal dynamics of cortical representation for action. Proc Natl Acad Sci USA. 97(2):913-918.
77. Nishitani N, Schurmann M, Amunts K, Hari R. 2005. Broca's region: from action to language. Physiology. 20:60-69.
78. Oldfield RC. 1971. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia. 9(1):97-113.
79. Özyürek A. 2002. Do speakers design their cospeech gestures for their addressees?: the effects of addressee location on representational gestures. J Mem Lang. 46(4):688-704.
80. Özyürek A, Willems RM, Kita S, Hagoort P. Forthcoming. On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. J Cogn Neurosci.
81. Petersson KM, Forkstam C, Ingvar M. 2004. Artificial syntactic violations activate Broca's region. Cogn Sci. 28:383-407.
82. Poldrack RA, Wagner AD, Prull MW, Desmond JE, Glover GH, Gabrieli JD. 1999. Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. Neuroimage. 10(1):15-35.
83. Pulvermuller F. 2005. Brain mechanisms linking language and action. Nat Rev Neurosci. 6(7):576-582.
84. Rizzolatti G, Arbib MA. 1998. Language within our grasp. Trends Neurosci. 21(5):188-194.
85. Rizzolatti G, Fogassi L, Gallese V. 2001. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat Rev Neurosci. 2(9):661-670.
86. Rodd JM, Davis MH, Johnsrude IS. 2005. The neural mechanisms of speech comprehension: fMRI studies of semantic ambiguity. Cereb Cortex. 15(8):1261-1269.
87. Ruschemeyer SA, Zysset S, Friederici AD. 2006. Native and non-native reading of sentences: an fMRI experiment. Neuroimage. 31(1):354-365.
88. Saygin AP, Wilson SM, Hagler DJ, Jr, Bates E, Sereno MI. 2004. Point-light biological motion perception activates human premotor cortex. J Neurosci. 24(27):6181-6188.
89. Schubotz RI, von Cramon DY. 2004. Sequences of abstract nonbiological stimuli share ventral premotor cortex with action observation and imagery. J Neurosci. 24(24):5467-5474.
90. Singer MA, Goldin Meadow S. 2005. Children learn when their teacher's gestures and speech differ. Psychol Sci. 16(2):85-89.
91. Talairach J, Tournoux P. 1988. Co-planar stereotaxic atlas of the human brain. New York: Thieme Medical.
92. Thompson LA, Massaro DW. 1986. Evaluation and integration of speech and pointing gestures during referential understanding. J Exp Child Psychol. 42(1):144-168.
93. Thompson LA, Massaro DW. 1994. Children's integration of speech and pointing gestures in comprehension. J Exp Child Psychol. 57(3):327-354.
94. Thompson-Schill SL, Bedny M, Goldberg RF. 2005. The frontal lobes and the regulation of mental activity. Curr Opin Neurobiol. 15(2):219-224.
95. Thompson-Schill SL, D'Esposito M, Aguirre GK, Farah MJ. 1997. Role of left inferior prefrontal cortex in retrieval of semantic knowledge: a reevaluation. Proc Natl Acad Sci USA. 94(26):14792-14797.
96. Vigneau M, Beaucousin V, Herve PY, Duffau H, Crivello F, Houde O, Mazoyer B, Tzourio-Mazoyer N. 2006. Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. Neuroimage. 30(4):1414-1432.
97. Vosse T, Kempen G. 2000. Syntactic structure assembly in human parsing: a computational model based on competitive inhibition and lexicalist grammar. Cognition. 75(2):105-143.
98. Wagner AD, Pare-Blagoev EJ, Clark J, Poldrack RA. 2001. Recovering meaning: left prefrontal cortex guides controlled semantic retrieval. Neuron. 31(2):329-338.
99. Wu YC, Coulson S. 2005. Meaningful gestures: electrophysiological indices of iconic gesture comprehension. Psychophysiology. 42(6):654-667.
100. Xiong J, Rao S, Jerabek P, Zamarripa F, Woldorff M, Lancaster J, Fox PT. 2000. Intersubject variability in cortical activations during a complex language task. Neuroimage. 12(3):326-339.