The role of synchrony and ambiguity in speech-gesture integration during comprehension

Journal of Cognitive Neuroscience

Volumn 23, Issue 8, 2011, Pages 1845-1854

  Habets, Boukje ^a   Kita, Sotaro ^b   Shao, Zeshu ^b   Özyurek, Asli ^c,d   Hagoort, Peter ^c,e  

^a UNIVERSITY OF HAMBURG   (Germany)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^c MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^d RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^e RADBOUD UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; AMBIGUITY; ARTICLE; CLINICAL ARTICLE; COMPREHENSION; FEMALE; GESTURE; HUMAN; MALE; PRIORITY JOURNAL; SEMANTICS; SPEECH; TIME; VIDEORECORDING; VISUAL MEMORY;

ADOLESCENT; ANALYSIS OF VARIANCE; BRAIN MAPPING; COMPREHENSION; ELECTROENCEPHALOGRAPHY; ELECTROENCEPHALOGRAPHY PHASE SYNCHRONIZATION; FEMALE; GESTURES; HUMANS; MALE; PHOTIC STIMULATION; REACTION TIME; SPEECH; YOUNG ADULT;

EID: 79957949882     PISSN: 0898929X     EISSN: 15308898     Source Type: Journal    
DOI: 10.1162/jocn.2010.21462     Document Type: Article

References (38)

1. Barrett, S. E., & Rugg, M. D. (1990). Event-related potentials and the semantic matching of pictures. Brain and Cognition, 14, 201-212.
2. Beattie, G., & Shovelton, H. (1999). Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica, 123, 1-30.
3. Bernardis, P., Salillas, E., & Caramelli, N. (2008). Behavioural and neuropsychological evidence of semantic interaction between iconic gestures and words. Cognitive Neuropsychology, 25, 1114-1128.
4. Butterworth, B. L., & Beattie, G. W. (1978). Gesture and silence as indicators of planning in speech. In R. N. Campbell & P. T. Smith (Eds.), Recent advances in the psychology of language 4: Formal and experimental approaches (pp. 347-360). London: Plenum.
5. Cassell, J., McNeill, D., & McCullough, K. E. (1999). Speech- gesture mismatches: Evidence for one underlying representation of linguistic and non-linguistic information. Pragmatics & Cognition, 7, 1-33.
6. Federmeyer, K. D., & Kutas, M. (2001). Meaning and modality: Influences of context, semantic memory, organization, and perceptual predictability on picture processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 202-204.
7. Feyereisen, P., van de Wiele, M., & Dubois, F. (1988). The meaning of gestures: What can be understood without speech? Cahiers de Psychologie Cognitive, 8, 3-25.
8. Ganis, G., Kutas, M., & Sereno, M. (1996). The search for common sense: An electrophysiological study of the comprehension of words and pictures for reading. Journal of Cognitive Neuroscience, 8, 89-106.
9. Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge, MA: Harvard University Press.
10. Graham, J. A., & Argyle, M. (1975). A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology, 10, 57-67.
11. Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304, 438-441.
12. Holle, H., & Gunter, T. C. (2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19, 1175-1192.
13. Huynh, H., & Feldt, L. S. (1976). Estimation of the box correction for degrees of freedom from sample data in randomized block and split-plot designs. Journal of Educational and Behavioral Statistics, 1, 69-82.
14. Kelly, S., Özyürek, A., & Maris, E. (2010). Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 21, 260-267.
15. Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253-260.
16. Kelly, S. D., Ward, S., Creigh, P., & Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 101, 222-233.
17. Kita, S. (2009). Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes, 24, 145-167.
18. Kita, S., Van Gijn, I., & Van der Hulst, H. (1998). Movement phases in signs and co-speech gestures, and their transcription by human coders. In I. Wachsmuth & M. Fröhlich (Eds.), Gesture and sign language in human-computer interaction (pp. 23-35). Berlin: Springer.
19. Krauss, R. M., Morrel-Samuels, P., & Colasante, C. (1991). Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 61, 743-754.
20. Kutas, M., & Hillyard, S. A. (1980). Reading between the lines: Event-related brain potentials during natural sentence processing. Brain and Language, 11, 354-373.
21. Kutas, M., & Hillyard, S. A. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161-163.
22. Marslen-Wilson, W. D. (1987). Functional parallelism in spoken word-recognition. Cognition, 25, 71-102.
23. McNeill, D. (1992). Hand and mind. Chicago: University of Chicago Press.
24. McNeill, D. (2000). Language and gesture. Cambridge: Cambridge University Press.
25. McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.
26. McNeill, D., Cassell, J., & McCullough, K.-E. (1994). Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction, 27, 223-238.
27. McPherson, W. B., & Holcomb, P. J. (1999). An electrophysiological investigation of semantic priming wit pictures of real objects. Psychophysiology, 36, 53-65.
28. Morrel-Samuels, P., & Krauss, R. M. (1992). Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 615-623.
29. Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58, 351-362.
30. Nigam, A., Hoffman, J. E., & Simons, R. F. (1992). N400 to semantically anomalous pictures and words. Journal of Cognitive Neuroscience, 4, 15-22.
31. Oostenveld, R., & Praamstra, P. (2001). The five percent electrode system for high-resolution EEG and ERP measurements. Clinical Neurophysiology, 112, 713-719.
32. Özyurek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19, 605-616.
33. Van Berkum, J. J., Zwitserlood, P., Brown, C., & Hagoort, P. (2003). When and how do listeners relate a sentence to the wider discourse? Evidence from the N400 effect. Cognitive Brain Research, 17, 701-718.
34. West, W. C., & Holcomb, P. J. (2002). Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research, 13, 363-375.
35. Willems, R., Ozyurek, A., & Hagoort, P. (2008). Seeing and hearing meaning: ERP and fMRI evidence of word versus picture integration into a sentence context. Journal of Cognitive Neuroscience, 20, 1235-1249.
36. Wu, Y. C., & Coulson, S. (2005). Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology, 42, 654-667.
37. Wu, Y. C., & Coulson, S. (2007a). Iconic gestures prime related concepts: An ERP study. Psychonomic Bulletin & Review, 14, 57-63.
38. Wu, Y. C., & Coulson, S. (2007b). How iconic gestures enhance communication: An ERP study. Brain and Language, 101, 234-245.