Neural evidence for the interplay between language, gesture, and action: A review

Brain and Language

Volumn 101, Issue 3, 2007, Pages 278-289

  Willems, Roel M ^a   Hagoort, Peter ^a,b,c  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^b MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^c RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Action; Brain; Cognitive neuroscience; Embodied cognition; ERP; Gesture; Language

Indexed keywords

ARTICLE; COGNITION; EVENT RELATED POTENTIAL; GESTURE; HUMAN; LANGUAGE PROCESSING; MOTOR ACTIVITY; NEUROIMAGING; SIGN LANGUAGE; SPEECH PERCEPTION; VOCALIZATION;

BRAIN; COGNITIVE SCIENCE; EVOKED POTENTIALS; GESTURES; HUMANS; LANGUAGE; MODELS, PSYCHOLOGICAL; PSYCHOMOTOR PERFORMANCE; SIGN LANGUAGE; SPEECH; SPEECH PERCEPTION;

EID: 34249081224     PISSN: 0093934X     EISSN: 10902155     Source Type: Journal    
DOI: 10.1016/j.bandl.2007.03.004     Document Type: Article

References (107)

1. Anderson M.L. Embodied cognition: A field guide. Artificial Intelligence 149 (2003) 91-130
2. Arbib M.A. From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences 28 2 (2005) 105-124
3. Aziz-Zadeh L., Wilson S.M., Rizzolatti G., and Iacoboni M. Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Current Biology 16 18 (2006) 1818-1823
4. Bates E., and Dick F. Language, gesture, and the developing brain. Developmental Psychobiology 40 3 (2002) 293-310
5. Bernardis P., and Gentilucci M. Speech and gesture share the same communication system. Neuropsychologia 44 2 (2006) 178-190
6. Boulenger V., Roy A.C., Paulignan Y., Deprez V., Jeannerod M., and Nazir T.A. Cross-talk between language processes and overt motor behavior in the first 200 msec of processing. Journal of Cognitive Neuroscience 18 10 (2006) 1607-1615
7. Brown C.M., Hagoort P., and Kutas M. Postlexical integration processes in language comprehension: Evidence from brain-imaging research. In: Gazzaniga M.S. (Ed). The cognitive neurosciences. Second ed. (2000), MIT Press, Cambridge, Mass 881-895
8. Buccino G., Riggio L., Melli G., Binkofski F., Gallese V., and Rizzolatti G. Listening to action-related sentences modulates the activity of the motor system: A combined TMS and behavioral study. Brain Research Cognitive Brain Research 24 3 (2005) 355-363
9. Chiel H.J., and Beer R.D. The brain has a body: Adaptive behavior emerges from interactions of nervous system, body and environment. Trends in Neurosciences 20 12 (1997) 553-557
10. Clark A. Being there: Putting brain, body and world together again (1997), MIT press, Cambridge, MA
11. Clark A. Visual experience and motor action: Are the bonds too tight?. Philosophical Review 110 4 (2001) 495-519
12. Clark A. Is seeing all it seems?. Journal of Consciousness Studies 9 5-6 (2002) 181-202
13. Clark A., and Chalmers D. The extended mind. Analysis 58 (1998) 7-19
14. Corballis M.C. From hand to mouth: The origins of language (2002), Princeton University Press, Princeton, NJ, US
15. Corina D.P., and Knapp H. Sign language processing and the mirror neuron system. Cortex 42 4 (2006) 529-539
16. Devlin J.T., and Watkins K.E. Stimulating language: Insights from TMS. Brain 130 3 (2006) 610-622
17. di Pellegrino G., Fadiga L., Fogassi L., Gallese V., and Rizzolatti G. Understanding motor events: A neurophysiological study. Experimental Brain Research 91 1 (1992) 176-180
18. Emmorey K., Grabowski T., McCullough S., Damasio H., Ponto L., Hichwa R., et al. Motor-iconicity of sign language does not alter the neural systems underlying tool and action naming. Brain and Language 89 1 (2004) 27-37
19. Emmorey K., Grabowski T., McCullough S., Ponto L.L., Hichwa R.D., and Damasio H. The neural correlates of spatial language in English and American sign language: A PET study with hearing bilinguals. Neuroimage 24 3 (2005) 832-840
20. Fadiga L., Craighero L., Buccino G., and Rizzolatti G. Speech listening specifically modulates the excitability of tongue muscles: A TMS study. European Journal of Neuroscience 15 2 (2002) 399-402
21. Federmeier K.D., and Kutas M. Meaning and modality: Influences of context, semantic memory organization, and perceptual predictability on picture processing. Journal of Experimental Psychology: Learning, Memory, and Cognition 27 1 (2001) 202-224
22. Floel A., Ellger T., Breitenstein C., and Knecht S. Language perception activates the hand motor cortex: Implications for motor theories of speech perception. European Journal of Neuroscience 18 3 (2003) 704-708
23. Fodor J.A. The language of thought (1975), Harvard University Press, Cambridge, MA
24. Fodor J.A. The modularity of mind (1983), MIT press, Cambridge, MA
25. Friederici A.D. Broca's area and the ventral premotor cortex in language: Functional differentiation and specificity. Cortex 42 4 (2006) 472-475
26. Fuster J.M. Cortex and mind (2003), Oxford University Press, New York
27. Galantucci B., Fowler C.A., and Turvey M.T. The motor theory of speech perception reviewed. Psychonomic Bulletin and Review 13 3 (2006) 361-377
28. Gallese V., and Lakoff G. The brain's concepts: The role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology 22 3-4 (2005) 455-479
29. Gentilucci M., Bernardis P., Crisi G., and Dalla Volta R. Repetitive transcranial magnetic stimulation of Broca's area affects verbal responses to gesture observation. Journal of Cognitive Neuroscience 18 7 (2006) 1059-1074
30. Gibson J.J. The senses considered as perceptual systems (1966), Houghton Mifflin, Boston
31. Glenberg, A.M. (in press). Language and action: Creating sensible combinations of ideas. In M.G. Gaskell (Ed.), The Oxford handbook of psycholinguistics. Oxford: Oxford University Press.
32. Glenberg A.M., and Kaschak M.P. Grounding language in action. Psychonomic Bulletin and Review 9 3 (2002) 558-565
33. Glenberg, A.M., Sato, M., Cattaneo, L., Riggio, L., Palumbo, D., & Buccino, G. (in press). Processing abstract language. Quarterly Journal of Experimental Psychology.
34. GoldinMeadow S. Hearing gesture: How our hands help us think (2003), Belknap Press of Harvard University Press, Cambridge, MA, US
35. Goldin Meadow S., Kim S., and Singer M. What the teacher's hands tell the student's mind about math. Journal of Educational Psychology 91 4 (1999) 720-730
36. Goldin Meadow S., and Momeni Sandhofer C. Gestures convey substantive information about a child's thoughts to ordinary listeners. Developmental Science 2 1 (1999) 67-74
37. Gunter T.C., and Bach P. Communicating hands: ERPs elicited by meaningful symbolic hand postures. Neuroscience Letters 372 1-2 (2004) 52-56
38. Hagoort P. On Broca, brain, and binding: A new framework. Trends in Cognitive Sciences 9 9 (2005) 416-423
39. Hagoort, P., & van Berkum, J.J. (in press). Beyond the sentence given. Philosophical Transactions of the Royal Society B.
40. Hagoort P., Hald L., Bastiaansen M., and Petersson K.M. Integration of word meaning and world knowledge in language comprehension. Science 304 5669 (2004) 438-441
41. Hauk O., Johnsrude I., and Pulvermuller F. Somatotopic representation of action words in human motor and premotor cortex. Neuron 41 2 (2004) 301-307
42. Hauk O., and Pulvermuller F. Neurophysiological distinction of action words in the fronto-central cortex. Human Brain Mapping 21 3 (2004) 191-201
43. Holle, H., & Gunter, T.C. (in press). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience.
44. Kable J.W., Kan I.P., Wilson A., Thompson-Schill S.L., and Chatterjee A. Conceptual representations of action in the lateral temporal cortex. Journal of Cognitive Neuroscience 17 12 (2005) 1855-1870
45. Kable J.W., Lease-Spellmeyer J., and Chatterjee A. Neural substrates of action event knowledge. Journal of Cognitive Neuroscience 14 5 (2002) 795-805
46. Kelly S.D., Kravitz C., and Hopkins M. Neural correlates of bimodal speech and gesture comprehension. Brain and Language 89 1 (2004) 253-260
47. Kelly S.D., Ward S., Creigh P., and Bartolotti J. An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language 101 (2006) 222-233
48. Kita S., and Özyürek A. What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48 1 (2003) 16-32
49. Koechlin E., and Jubault T. Broca's area and the hierarchical organization of human behavior. Neuron 50 6 (2006) 963-974
50. Krauss R.M., Morrel Samuels P., and Colasante C. Do conversational hand gestures communicate?. Journal of Personality and Social Psychology 61 5 (1991) 743-754
51. Kutas M., and van Petten C.K. Psycholinguistics electrified: Event-related brain potential investigations. In: Gernsbacher M.A. (Ed). Handbook of psycholinguistics (1994), Academic Press, San Diego, CA 83-143
52. Levanen S., Uutela K., Salenius S., and Hari R. Cortical representation of sign language: Comparison of deaf signers and hearing non-signers. Cerebral Cortex 11 6 (2001) 506-512
53. Liberman A.M., and Mattingly I.G. The motor theory of speech perception revised. Cognition 21 1 (1985) 1-36
54. MacSweeney M., Campbell R., Woll B., Brammer M.J., Giampietro V., David A.S., et al. Lexical and sentential processing in British sign language. Human Brain Mapping 27 1 (2006) 63-76
55. MacSweeney M., Campbell R., Woll B., Giampietro V., David A.S., McGuire P.K., et al. Dissociating linguistic and nonlinguistic gestural communication in the brain. Neuroimage 22 4 (2004) 1605-1618
56. MacSweeney M., Woll B., Campbell R., Calvert G.A., McGuire P.K., David A.S., et al. Neural correlates of British sign language comprehension: Spatial processing demands of topographic language. Journal of Cognitive Neuroscience 14 7 (2002) 1064-1075
57. MacSweeney M., Woll B., Campbell R., McGuire P.K., David A.S., Williams S.C., et al. Neural systems underlying British sign language and audio-visual English processing in native users. Brain 125 7 (2002) 1583-1593
58. McNeill D. Hand and mind: What gestures reveal about thought (1992), University of Chicago Press, Chicago, IL, US
59. McNeill D., Cassell J., and McCullough K.E. Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction 27 3 (1994) 223-237
60. Merleau-Ponty M. Phenomenology of Perception ([1945]1962), Routledge & Kegan, London
61. Mesulam M.M. Large-scale neurocognitive networks and distributed processing for attention, language, and memory. Annals of Neurology 28 5 (1990) 597-613
62. Mesulam M.M. From sensation to cognition. Brain 121 6 (1998) 1013-1052
63. Molnar-Szakacs I., Iacoboni M., Koski L., and Mazziotta J.C. Functional segregation within pars opercularis of the inferior frontal gyrus: Evidence from fMRI studies of imitation and action observation. Cerebral Cortex 15 7 (2005) 986-994
64. Nakamura A., Maess B., Knosche T.R., Gunter T.C., Bach P., and Friederici A.D. Cooperation of different neuronal systems during hand sign recognition. Neuroimage 23 1 (2004) 25-34
65. Neville H.J., Bavelier D., Corina D., Rauschecker J., Karni A., Lalwani A., et al. Cerebral organization for language in deaf and hearing subjects: Biological constraints and effects of experience. Proceedings of the National Academy of Sciences of the United States of America 95 3 (1998) 922-929
66. Newman A.J., Bavelier D., Corina D., Jezzard P., and Neville H.J. A critical period for right hemisphere recruitment in American Sign Language processing. Nature Neuroscience 5 1 (2002) 76-80
67. Nieuwland M.S., and van Berkum J.J.A. When peanuts fall in love: n400 evidence for the power of discourse. Journal of Cognitive Neuroscience 18 7 (2006) 1098-1111
68. Noe A., and Thompson E. Introduction. In: Noe A., and Thompson E. (Eds). Vision and mind, selected readings in the philosophy of perception (2002), MIT press, Cambridge, MA 1-14
69. Noppeney U., Josephs O., Kiebel S., Friston K.J., and Price C.J. Action selectivity in parietal and temporal cortex. Brain Research Cognitive Brain Research 25 3 (2005) 641-649
70. O'Regan J.K., and Noe A. A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences 24 5 (2001) 939-973
71. Özyürek, A., Willems, R.M., Kita, S., & Hagoort, P. (in press). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience.
72. In: Pecher D., and Zwaan R.A. (Eds). Grounding cognition: The role of perception and action in memory, language, and thinking (2005), Cambridge University Press, Cambridge, UK
73. Piaget J., and Inhelder B. The psychology of the child ([1966] 1969), Routledge & Kegan Paul, London
74. Pulvermuller F., Hauk O., Nikulin V.V., and Ilmoniemi R.J. Functional links between motor and language systems. European Journal of Neuroscience 21 3 (2005) 793-797
75. Pulvermuller F., Huss M., Kherif F., Moscoso del Prado Martin F., Hauk O., and Shtyrov Y. Motor cortex maps articulatory features of speech sounds. Proceedings of the National Academy of Sciences of the United States of America 103 20 (2006) 7865-7870
76. Pulvermuller F., Shtyrov Y., and Ilmoniemi R. Brain signatures of meaning access in action word recognition. Journal of Cognitive Neuroscience 17 6 (2005) 884-892
77. Rizzolatti G., and Craighero L. The mirror-neuron system. Annual Review of Neuroscience 27 (2004) 169-192
78. Rizzolatti G., Fogassi L., and Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Reviews Neuroscience 2 9 (2001) 661-670
79. Roy D. Grounding words in perception and action: Computational insights. Trends in Cognitive Sciences 9 8 (2005) 389-396
80. Ruschemeyer, S.-A., & Friederici, A.D. (2006). Processing of motor vs. abstract words: An fMRI experiment. Paper presented at Architectures and Mechanisms for Language Processing (AMLaP), Nijmegen.
81. Saffran E., and Scholl A. Clues to the functional and neural architecture of word meaning. In: Brown C.M., and Hagoort P. (Eds). The neurocognition of language (1999), Oxford University Press, Oxford
82. Saygin A.P., Wilson S.M., Dronkers N.F., and Bates E. Action comprehension in aphasia: Linguistic and non-linguistic deficits and their lesion correlates. Neuropsychologia 42 13 (2004) 1788-1804
83. Singer M.A., and Goldin Meadow S. Children learn when their teacher's gestures and speech differ. Psychological Science 16 2 (2005) 85-89
84. Skipper J.I., Goldin Meadow S., Nusbaum H.C., and Small S.L. Speech associated gestures, Broca's area and the human mirror system. Brain and Language 101 (2007) 260-277
85. Smith L.B., and Thelen E. Development as a dynamic system. Trends in Cognitive Sciences 7 8 (2003) 343-348
86. Steels L. Evolving grounded communication for robots. Trends in Cognitive Sciences 7 7 (2003) 308-312
87. Sundara M., Namasivayam A.K., and Chen R. Observation-execution matching system for speech: A magnetic stimulation study. Neuroreport 12 7 (2001) 1341-1344
88. Tettamanti M., Buccino G., Saccuman M.C., Gallese V., Danna M., Scifo P., et al. Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience 17 2 (2005) 273-281
89. Tettamanti M., and Weniger D. Broca's area: A supramodal hierarchical processor?. Cortex 42 4 (2006) 491-494
90. Thelen E., and Smith L.B. A dynamic systems approach to the development of cognition and action (1994), MIT Press, Cambridge, MA
91. Thompson-Schill S.L., Bedny M., and Goldberg R.F. The frontal lobes and the regulation of mental activity. Current Opinion in Neurobiology 15 2 (2005) 219-224
92. Thompson-Schill S.L., D'Esposito M., Aguirre G.K., and Farah M.J. Role of left inferior prefrontal cortex in retrieval of semantic knowledge: A reevaluation. Proceedings of the National Academy of Sciences of the United States of America 94 26 (1997) 14792-14797
93. Tranel D., Kemmerer D., Adolphs R., Damasio H., and Damasio A.R. Neural correlates of conceptual knowledge for actions. Cognitive Neuropsychology 20 3-6 (2003) 409-432
94. van Berkum J.J.A., Hagoort P., and Brown C.M. Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience 11 6 (1999) 657-671
95. van Berkum J.J.A., Zwitserlood P., Hagoort P., and Brown C.M. When and how do listeners relate a sentence to the wider discourse? Evidence from the N400 effect. Cognitive Brain Research 17 3 (2003) 701-718
96. van Gelder T. The distinction between mind and cognition. In: Houng Y.-H., and Ho J.-C. (Eds). Mind and cognition (1995), Academia Sinica, Taipei, Taiwan
97. van Schie H.T., Toni I., and Bekkering H. Comparable mechanisms for action and language: Neural systems behind intentions, goals, and means. Cortex 42 4 (2006) 495-498
98. Varela F.J., Thompson E., and Rosch E. The embodied mind (1991), MIT Press, Cambridge, MA
99. Vigliocco G., Warren J., Siri S., Arciuli J., Scott S., and Wise R. The role of semantics and grammatical class in the neural representation of words. Cerebral Cortex 16 12 (2006) 1790-1796
100. Watkins K.E., and Paus T. Modulation of motor excitability during speech perception: The role of Broca's area. Journal of Cognitive Neuroscience 16 6 (2004) 978-987
101. Watkins K.E., Strafella A.P., and Paus T. Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 41 8 (2003) 989-994
102. Wheeler M. Reconstructing the cognitive world (2005), MIT press, Cambridge, MA
103. Willems R.M., Özyürek A., and Hagoort P. When language meets action: The neural integration of gesture and speech. Cerebral Cortex (2006)
104. Wilson M. Six views of embodied cognition. Psychonomic Bulletin and Review 9 4 (2002) 625-636
105. Wilson S.M., and Iacoboni M. Neural responses to non-native phonemes varying in producibility: Evidence for the sensorimotor nature of speech perception. Neuroimage 33 1 (2006) 316-325
106. Wilson S.M., Saygin A.P., Sereno M.I., and Iacoboni M. Listening to speech activates motor areas involved in speech production. Nature Neuroscience 7 7 (2004) 701-702
107. Wu Y.C., and Coulson S. Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology 42 6 (2005) 654-667