Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 369, Issue 1651, 2014, Pages

  Özyürek, Asli ^a,b  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^b MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

Author keywords

Brain; Co speech gestures; Iconicity; Multimodal language; Semantics

Indexed keywords

BRAIN; HEARING; LANGUAGE; SIGNALING; VISION;

EID: 84905284251     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2013.0296     Document Type: Review

References (79)

1. Klima E, Bellugi U. 1979 Signs of language. Cambridge, MA: Harvard University Press.
2. Stokoe WC. 1960 Sign language structure: an outline of the visual communication systems of the American deaf. Stud. Linguist. Occas. Pap. 8, 1-78.
3. Poizner H, Klima ES, Bellugi U. 1987 What the hands reveal about the brain. Cambridge, MA: MIT Press.
4. Kendon A. 2014 Semiotic diversity in utterance production and the concept of 'language'. Phil. Trans. R. Soc. B 369, 20130293. (doi: 10. 1098/rstb. 2013. 0293).
5. Goldin-Meadow S. 2003 Hearing hands: how our hands help us think. Cambridge, MA: Harvard University Press.
6. Kendon A. 2004 Gesture: visible action as utterance. Cambridge, UK: Cambridge University Press.
7. McNeill D. 1992 Hand and mind: what gestures reveal about thoughts. Chicago, IL: University of Chicago Press.
8. McNeill D. 2005 Gesture and thought. Chicago, IL: University of Chicago Press.
9. Kendon A. 1980 Gesticulation and speech: two aspects of the process of utterance. In The relationship of verbal and nonverbal communication (ed. M Ritchie Key), pp. 207-227. The Hague, The Netherlands: Mouton.
10. Debreslioska S, Özyürek A, Gullberg M, Perniss PM. 2013 Gestural viewpoint signals referent accessibility. Discourse Process. 50, 431-456. (doi: 10. 1080/0163853X. 2013. 824286).
11. So WC, Kita S, Goldin-Meadow S. 2009 Using the hands to identify who does what to whom: gesture and speech go hand-in-hand. Cogn. Sci. 33, 115-125. (doi: 10. 1111/j. 1551-6709. 2008. 01006. x).
12. Enfield NJ. 2009 The anatomy of meaning: speech, gesture, and composite utterances. Cambridge, UK: Cambridge University Press.
13. Kita S, Özyürek A. 2003 What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. J. Mem. Lang. 48, 16-32. (doi: 10. 1016/S0749-596X(02)00505-3).
14. Church B, Kelly S, Holcomb D. 2013 Temporal synchrony between speech, action and gesture during language production. Lang. Cogn. Process. 29, 345-354.
15. Loehr D. 2007 Aspects of rhythm in gesture and speech. Gesture 7, 179-214. (doi: 10. 1075/gest. 7. 2. 04loe).
16. Clark H. 1996 Using language. Cambridge, UK: Cambridge University Press.
17. Wu Y, Coulson S. 2011 Are depictive gestures like pictures? Commonalities and differences in semantic processing. Brain Lang. 119, 184-195. (doi: 10. 1016/j. bandl. 2011. 07. 002).
18. Campbell R. 2008 The processing of audio-visual speech: empirical and neural bases. Phil. Trans. R. Soc. Lond. B 363, 1001-1010. (doi: 10. 1098/rstb. 2007. 2155).
19. Wagner P, Malisz Z, Kopp S. 2014 Speech and gesture in interaction: an overview. Speech Commun. 57, 209-232. (doi: 10. 1016/j. specom. 2013. 09. 008).
20. Graham JA, Argyle M. 1975 A cross-cultural study of the communication of extra-verbal meaning by gestures. Int. J. Psychol. 10, 57-67. (doi: 10. 1080/00207597508247319).
21. Kelly SD, Barr D, Church RB, Lynch K. 1999 Offering a hand to pragmatic understanding: the role of speech and gesture in comprehension and memory. J. Mem. Lang. 40, 577-592. (doi: 10. 1006/jmla. 1999. 2634).
22. Beattie G, Shovelton H. 1999 Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica 123, 1-30. (doi: 10. 1515/semi. 1999. 123. 1-2. 1).
23. McNeill D, Cassell J, McCullough K-E. 1999 Communicative effects of speech-mismatched gestures. Res. Lang. Soc. Interact. 27, 223-238. (doi: 10. 1207/s15327973rlsi2703_4).
24. Goldin Meadow S, Momeni Sandhofer C. 1999 Gestures convey substantive information about a child's thoughts to ordinary listeners. Dev. Sci. 2, 67-74. (doi: 10. 1111/1467-7687. 00056).
25. Singer MA, Goldin Meadow S. 2005 Children learn when their teacher's gestures and speech differ. Psychol. Sci. 16, 85-89. (doi: 10. 1111/j. 0956-7976. 2005. 00786. x).
26. Yap DF, So WC, Yap M, Tan YQ. 2011 Iconic gestures prime words. Cogn. Sci. 35, 171-183. (doi: 10. 1111/j. 1551-6709. 2010. 01141. x).
27. Kutas M, Hillyard SA. 1980 Reading senseless sentences: brain potentials reflect semantic incongruity. Science 207, 203-205. (doi: 10. 1126/science. 7350657).
28. Hagoort P, Van Berkum JJA. 2007 Beyond the sentence given. Phil. Trans. R. Soc. B 362, 801-811. (doi: 10. 1098/rstb. 2007. 2089).
29. Wu Y, Coulson S. 2005 Meaningful gestures: electrophysiological indices of iconic gesture comprehension. Psychophysiology 42, 654-667. (doi: 10. 1111/j. 1469-8986. 2005. 00356. x).
30. Wu Y, Coulson S. 2007 Iconic gestures prime related concepts: an ERP study. Psych. Bull. Rev. 14, 57-63. (doi: 10. 3758/BF03194028).
31. Willems RM, Özyürek A, Hagoort P. 2008 Seeing and hearing meaning: ERP and fMRI evidence of word versus picture integration into a sentence context. J. Cogn. Neurosci. 20, 1235-1249. (doi: 10. 1162/jocn. 2008. 20085).
32. Holle H, Gunter TC. 2007 The role of iconic gestures in speech disambiguation: ERP evidence. J. Cogn. Neurosci. 19, 1175-1192. (doi: 10. 1162/jocn. 2007.19. 7. 1175).
33. Özyürek A, Willems RM, Kita S, Hagoort P. 2007 On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. J. Cogn. Neurosci. 19, 605-616. (doi: 10. 1162/jocn. 2007. 19. 4. 605).
34. Straube B, Green A, Weis S, Kircher T. 2012 A supramodal neural network for speech and gesture semantics: an fMRI study. PLoS ONE 7, e51207. (doi: 10. 1371/journal. pone. 0051207).
35. Xu J, Gannon P, Emmorey K, Smith JF, Braun AR. 2009 Symbolic gestures and spoken language are processed by a common neural system. Proc. Natl Acad. Sci. 106, 20 664-20 669. (doi: 10. 1073/pnas. 0909197106).
36. Kelly SD, Özyürek A, Maris E. 2010 Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychol. Sci. 21, 260-267. (doi: 10. 1177/0956797609357327).
37. Kelly SD, Kravitz C, Hopkins M. 2004 Neural correlates of bimodal speech and gesture comprehension. Brain Lang. 89, 253-260. (doi: 10. 1016/S0093-934X(03)00335-3).
38. Dick AS, Goldin-Meadow S, Hasson U, Skipper J, Small SL. 2009 Co-speech gestures influence neural responses in brain regions associated with semantic processing. Human Brain Mapping 30(11), 3509-3526.
39. Dick AS, Goldin-Meadow S, Solodkin A, Small SL. 2012 Gesture in the developing brain. Dev. Sci. 15, 165-180. (doi: 10. 1111/j. 1467-7687. 2011. 01100. x).
40. Green A, Straube B, Weis S, Jansen A, Willmes K, Konrad K, Kircher T. 2009 Neural integration of iconic and unrelated coverbal gestures: a functional MRI study. Hum. Brain Map. 30, 3309-3324. (doi: 10. 1002/hbm. 20753).
41. Holle H, Gunter TC, Rüschemeyer SA, Hennenlotter A, Iacoboni M. 2008 Neural correlates of the processing of co-speech gestures. NeuroImage 39, 2010-2024. (doi: 10. 1016/j. neuroimage. 2007. 10. 055).
42. Holle H, Obleser J, Rueschemeyer S-A, Gunter TC. 2010 Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. NeuroImage 49, 875-884. (doi: 10. 1016/j. neuroimage. 2009. 08. 058).
43. Kircher T, Straube B, Leube D, Weis S, Sachs O, Willmes K, Konrad K, Green A. 2009 Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures. Neuropsychologia 47, 169-179. (doi: 10. 1016/j. neuropsychologia. 2008. 08. 009).
44. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL. 2007 Speech-associated gestures, Broca's area, and the human mirror system. Brain Lang. 101, 260-277. (doi: 10. 1016/j. bandl. 2007. 02. 008).
45. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL. 2009 Gestures orchestrate brain networks for language understanding. Curr. Biol. 19, 661-667. (doi: 10. 1016/j. cub. 2009. 02. 051).
46. Straube B, Green A, Weis S, Chatterjee A 2009 Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance. J. Cogn. Neurosci. 21, 821-836. (doi: 10. 1162/jocn. 2009. 21053).
47. Straube B, Green A, Bromberger B, Kircher T 2011 The differentiation of iconic and metaphoric gestures: common and unique integration processes. Hum. Br. Mapp. 32, 520-533. (doi: 10. 1002/hbm. 21041).
48. Willems RM, Özyürek A, Hagoort P. 2007 When language meets action: the neural integration of gesture and speech. Cerebral Cortex 17, 2322-2333. (doi: 10. 1093/cercor/bhl141).
49. Willems RM, Özyürek A, Hagoort P. 2009 Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. NeuroImage 47, 1992-2004. (doi: 10. 1016/j. neuroimage. 2009. 05. 066).
50. Bedny M, McGill M, Thompson-Schill SL. 2008 Semantic adaptation and competition during word comprehension. Cerebr. Cortex 18, 2574-2585. (doi: 10. 1093/cercor/bhn018).
51. Gennari SP, MacDonald MC, Postle BR, Seidenberg MS. 2007 Context-dependent interpretation of words: evidence for interactive neural processes. Neuroimage 35, 1278-1286. (doi: 10. 1016/j. neuroimage. 2007. 01. 015).
52. Hoenig K, Scheef L. 2009 Neural correlates of semantic ambiguity processing during context verification. Neuroimage 45, 1009-1019. (doi: 10. 1016/j. neuroimage. 2008. 12. 044).
53. Rodd JM, Davis MH, Johnsrude IS. 2005. The neural mechanisms of speech comprehension: FMRI studies of semantic ambiguity. Cerebr. Cortex 15, 1261-1269. (doi: 10. 1093/cercor/bhi009).
54. Snijders TM, Vosse T, Kempen G, Van Berkum JJA, Petersson KM, Hagoort P. 2009 Retrieval and unification of syntactic structure in sentence comprehension: an FMRI study using word-category ambiguity. Cerebr. Cortex 19, 1493-1503. (doi: 10. 1093/cercor/bhn187).
55. Whitney C, Kirk M, O'Sullivan J, Lambon Ralph MA, Jefferies E. 2011 The neural organization of semantic control: TMS evidence for a distributed network in left inferior frontal and posterior middle temporal gyrus. Cerebr. Cortex 21, 1066-1075. (doi: 10. 1093/cercor/bhq180).
56. Zempleni MZ, Renken R, Hoeks JC, Hoogduin JM, Stowe LA. 2007 Semantic ambiguity processing in sentence context: evidence from event-related FMRI. Neuroimage 34, 1270-1279. (doi: 10. 1016/j. neuroimage. 2006. 09. 048).
57. Dick AS, Mok E, Raja Beharelle A, Goldin-Meadow S, Small SL. 2014 Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech. Hum. Br. Mapp. 35, 900-917. (doi: 10. 1002/hbm. 22222).
58. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ. 2007 Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J. Neurosci. 27, 7881-7887. (doi: 10. 1523/JNEUROSCI. 1740-07. 2007).
59. Hagoort P, Baggio G, Willems RM. 2009 Semantic unification. In The cognitive neurosciences, 4th edn (ed. MS Gazzaniga), pp. 819-836. Cambridge, MA: MIT Press.
60. Josse G, Joseph S, Bertasi E, Giraud A. 2012 The brain's dorsal route for speech represents word meaning: evidence from gesture. PLoS ONE 7-9, e46108. (doi: 10. 1371/journal. pone. 0046108).
61. Chui K. 2005 Temporal patterning of speech and iconic gestures in conversational discourse. J. Pragmat. 37, 871-887. (doi: 10. 1016/j. pragma. 2004. 10. 016).
62. Holler J, Wilkin K. 2009 Communicating common ground: how mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task. Lang. Cogn. Process. 24, 267-289. (doi: 10. 1080/01690960802095545).
63. Habets B, Kita S, Shao Z, Özyürek A, Hagoort P. 2011 The role of synchrony and ambiguity in speech-gesture integration during comprehension. J. Cogn. Neurosci. 23, 1845-1854. (doi: 10. 1162/jocn. 2010. 21462).
64. Obermeier C, Holler H, Gunther T. 2011 What iconic gesture fragments reveal about gesture-speech integration when synchrony is lost: memory can help. J. Cogn. Neurosci. 23, 1648-1663. (doi: 10. 1162/jocn. 2010. 21498).
65. Obermeier C, Dolk T, Gunther T. 2012 The benefit of gestures during communication: evidence from hearing and hearing-impaired individuals. Cortex 48, 857-870. (doi: 10. 1016/j. cortex. 2011. 02. 007).
66. Kelly SD, Ward S, Creigh P, Bartolotti J. 2007 An intentional stance modulates the integration of gesture and speech during comprehension. Brain Lang. 101, 222-233. (doi: 10. 1016/j. bandl. 2006. 07. 008).
67. Holler J, Kokal I, Toni I, Hagoort P, Kelly S, Özyürek A. 2014 Eye'm talking to you: speakers' gaze direction modulates co-speech gesture processing in the right MTG. Soc. Cogn. Affect. Neurosci. (doi: 10. 1093/scan/nsu047).
68. Holler J, Kelly S, Hagoort P, Özyürek A. 2012 When gestures catch the eye: the influence of gaze direction on co-speech gesture comprehension in triadic communication. In Proceedings of the 34th annual meeting of the cognitive science society (eds N Miyake, D Peebles, RP Cooper), pp. 467-472. Austin, TX: Cognitive Society.
69. Emmorey K, Özyürek A. In press. Language in our hands: neural underpinnings of sign language and cospeech gesture. In Handbook of cognitive neuroscience, 5th edn (ed. MS Gazzaniga). Cambridge, UK: MIT Press.
70. Maricchiolo F, Gnisci A, Bonaiuto M, Ficca G. 2009 Effects of different types of hand gestures in persuasive speech on receivers' evaluations. Lang. Cogn. Process. 24, 239-266. (doi: 10. 1080/01690960802159929).
71. Hagoort P. 2005 On Broca, brain, and binding: a new framework. Trends Cogn. Sci. 9, 416-423. (doi: 10. 1016/j. tics. 2005. 07. 004).
72. Calvert GA. 2001 Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral Cortex 11, 1110-1123. (doi: 10. 1093/cercor/11. 12. 1110).
73. Bernardis P, Gentilucci M. 2006 Speech and gesture share the same communication system. Neuropsychologia 44, 178-190. (doi: 10. 1016/j. neuropsychologia. 2005. 05. 007).
74. Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M. 2006 Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Curr. Biol. 16, 1818-1823. (doi: 10. 1016/j. cub. 2006. 07. 060).
75. Molnar-Szakacs I, Iacoboni M, Koski L, Mazziotta JC. 2005 Functional segregation within pars opercularis of the inferior frontal gyrus: evidence from fMRI studies of imitation and action observation. Cerebral Cortex 15, 986-994. (doi: 10. 1093/cercor/bhh199).
76. Willems RM, Hagoort P. 2007 Neural evidence for the interplay between action, gesture and language: a review. Brain Lang. 101, 278-289. (doi: 10. 1016/j. bandl. 2007. 03. 004).
77. Meyer GF, Harrison NR, Wuerger S. 2013 The time course of auditory-visual processing of speech and body actions: evidence for the simultaneous activation of an extended neural network for semantic processing. Neuropsychologia 51, 1716-1725. (doi: 10. 1016/j. neuropsychologia. 2013.05. 014).
78. Perniss P, Thompson RL, Vigliocco G. 2010 Iconicity as a general property of language: evidence from spoken and sign languages. Front. Psychol. 1, 227. (doi: 10. 3389/fpsyg. 2010. 00227).
79. Perniss P, Vigliocco G. 2014 The brdige of iconicity: from a world of experience to the experience of language. Phil. Trans. R. Soc. B 369, 20130300. (doi: 10. 1098/rstb. 2013. 0300).