On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials

Journal of Cognitive Neuroscience

Volumn 19, Issue 4, 2007, Pages 605-616

  Özyürek, Asli ^a,b   Willems, Roel M ^a   Kita, Sotaro ^c   Hagoort, Peter ^a,b  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^b MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^c UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; CONTROLLED STUDY; ELECTROENCEPHALOGRAPHY; EVENT RELATED POTENTIAL; FEMALE; GESTURE; HUMAN; INTERPERSONAL COMMUNICATION; MALE; NORMAL HUMAN; ONLINE SYSTEM; PRIORITY JOURNAL; SEMANTICS; SPEECH;

ADULT; COMPREHENSION; EVOKED POTENTIALS; FEMALE; GESTURES; HUMANS; LANGUAGE TESTS; MALE; NONVERBAL COMMUNICATION; SEMANTICS; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 33947646439     PISSN: 0898929X     EISSN: 15308898     Source Type: Journal    
DOI: 10.1162/jocn.2007.19.4.605     Document Type: Article

References (47)

1. Barret, S. E., & Rugg, M. D. (1990). Event-related potentials and the semantic matching of pictures. Brain and Cognition, 14, 201-212.
2. Beattie, G., & Shovelton, H. (1999). Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica, 123, 1-30.
3. Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the same communication system. Neuropsychologia, 44, 178-190.
4. Butterworth, B., & Beattie, G. (1978). Gesture and silence as indicators of planning in speech. In N. Campbell & P. T. Smith (Eds.), Recent advances in the psychology of language: Formal and experimental approaches (pp. 347-360). New York: Plenum.
5. Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.
6. Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. (2002). Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short term memory. Clinical Neurophysiology, 113, 495-506.
7. Federmeyer, K. D., & Kutas, M. (2001). Meaning and modality: Influences of context, semantic memory, organization, and perceptual predictability on picture processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 202-204.
8. Feyereisen, P., Van de Wiele, M., & Dubois, F. (1988). The meaning of gestures: What can be understood without speech? Cahiers de Psychologie Cognitive, 8, 3-25.
9. Forster, K. I. (1979). Levels of processing and the structure of the language processor. In W. E. Cooper & C. T. Walker (Eds.), Sentence processing: Psycholinguistic essays presented to Merrill Garrett (pp. 27-85). Hillsdale, NJ: Erlbaum.
10. Ganis, G., & Kutas, M. (2003). An electrophysiological study of scene effects on object identification. Cognitive Brain Research, 16, 123-144.
11. Ganis, G., Kutas, M., & Sereno, M. (1996). The search for common sense: An electrophysiological study of the comprehension of words and pictures for reading. Journal of Cognitive Neuroscience, 8, 89-106.
12. Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge: Harvard University Press.
13. Goldin-Meadow, S., & Sanhofer, C. M. (1999). Gesture conveys substantive information about a child's thoughts to ordinary listeners. Developmental Science, 2, 64-74.
14. Graham, J. A., & Argyle, M. (1975). A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology, 10, 57-67.
15. Hagoort, P. (2003a). Interplay between syntax and semantics during sentence comprehension: ERP effects of combining syntactic and semantic violations. Journal of Cognitive Neuroscience, 15, 883-899.
16. Hagoort, P. (2003b). How the brain solves the binding problem for language: A neurocomputational model of syntactic processing. Neuroimage, 20, S18-S29.
17. Hagoort, P. (2005). On Broca, brain, and binding: A new framework. Trends in Cognitive Sciences, 9, 416-423.
18. Hagoort, P., & Brown, C. (1994). Brain responses to lexical ambiguity resolution and parsing. In L. Frazier, C. J. Clifton, & K. Rayner (Eds.), Perspectives in sentence processing (pp. 45-80). Hillsdale, NJ: Erlbaum.
19. Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304, 438-441.
20. Holcomb, P. J., & McPherson, W. B. (1994). Event related potentials reflect semantic priming in an object decision task. Brain and Cognition, 24, 259-276.
21. Kelly, S. D., & Church, R. B. (1998). A comparison between children's and adults' ability to detect children's representational gestures. Child Development, 69, 85-93.
22. Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253-260.
23. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.
24. Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 16-32.
25. Krauss, R. M., Morrel-Samuels, P., & Colasante, C. (1991). Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 61, 743-754.
26. Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203-205.
27. Kutas, M., & Hillyard, S. A. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161-163.
28. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
29. McNeill, D. (1992). Hand and mind. Chicago: University of Chicago Press.
30. McNeill, D. (2000). Language and gesture. Cambridge: Cambridge University Press.
31. McNeill, D., Cassell, J., & McCullough, K.-E. (1999). Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction, 27, 223-238.
32. McPherson, W. B., & Holcomb, P. J. (1999). An electrophysiological investigation of semantic priming with pictures of real objects. Psychophysiology, 36, 53-65.
33. Melinger, A., & Levelt, W. J. M. (2004). Gesture and the communicative intention of the speaker. Gesture, 4, 119-141.
34. Morrel-Samuels, P., & Krauss, R. M. (1992). Word familiarity predicts the temporal asynchrony of hand gesture and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 615-623.
35. Mottonnen, R., Krause, C. M., Tiippana, K., & Sams, M. (2002). Processing of changes in visual speech in human auditory cortex. Cognitive Brain Research, 13, 417-425.
36. Nigam, A., Hoffman, J. E., & Simons, R. F. (1992). N400 to semantically anomalous pictures and words. Journal of Cognitive Neuroscience, 4, 15-22.
37. Osterhout, L., McLaughlin, J., & Bersick, M. (1997). Event-related brain potentials and human language. Trends in Cognitive Sciences, 1, 203-209.
38. Özyürek, A. (2002). Do speakers design their co-speech gestures for their addressees? The effects of addressee location on representational gestures. Journal of Memory and Language, 46, 688-704.
39. Sams, M., Aulanko, R., Hamalainen, M., Hari, R., Lounasmaa, O. V., Lu, S. T., et al. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127, 141-145.
40. Singer, M. A., & Goldin-Meadow, S. (2005). Children learn when their teachers' speech and gestures differ. Psychological Science, 16, 85-89.
41. Sitnikova, T., Kuperberg, G., & Holcomb, P. J. (2003). Semantic integration in videos of real world events: An electrophysiological investigation. Psychophysiology, 40, 160-164.
42. Tanenhaus, M., Spivey-Knowlton, J. M., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.
43. Van Berkum, J., Hagoort, P., & Brown, C. (1999). Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience, 11, 657-671.
44. Van Berkum, J., Zwitserlood, P., Brown, C., & Hagoort, P. (2003). When and how do listeners relate a sentence to the wider discourse? Evidence from the N400 effect. Cognitive Brain Research, 17, 701-718.
45. West, W. C., & Holcomb, P. J. (2002). Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research, 13, 363-375.
46. Willems, R. M., Özyürek, A., & Hagoort, P. (in press). When language meets action: The neural integration of speech and gesture. Cerebral Cortex.
47. Wu, Y. C., & Coulson, S. (2005). Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology, 42, 654-667.