What iconic gesture fragments reveal about gesture-speech integration: When synchrony is lost, memory can help

Journal of Cognitive Neuroscience

Volumn 23, Issue 7, 2011, Pages 1648-1663

  Obermeier, Christian ^a   Holle, Henning ^b   Gunter, Thomas C ^a  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^b UNIVERSITY OF SUSSEX   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; FEMALE; GESTURE; HUMAN; INFORMATION PROCESSING; MALE; PRIORITY JOURNAL; SPEECH; SPEECH ANALYSIS; SPEECH PERCEPTION; VISUAL MEMORY;

ADULT; EVOKED POTENTIALS, AUDITORY; EVOKED POTENTIALS, VISUAL; FEMALE; GESTURES; HUMANS; MALE; MEMORY; MENTAL PROCESSES; REACTION TIME; SPEECH; VOCABULARY; YOUNG ADULT;

EID: 79955806179     PISSN: 0898929X     EISSN: 15308898     Source Type: Journal    
DOI: 10.1162/jocn.2010.21498     Document Type: Article

References (42)

1. Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335-359.
2. Donchin, E., Ritter, W., & McCallum, W. C. (1978). Cognitive psychophysiology: The endogenous components of the ERPs. In E. Callaway, P. Teuting, & S. Koslow (Eds.), Event-related brain potentials in man (pp. 349-441). New York: Academic Press.
3. Emmorey, K., & Corina, D. (1990). Lexical recognition in sign language: Effects of phonetic structure and morphology. Perceptual and Motor Skills, 71, 1227-1252.
4. Friederici, A., Gunter, T. C., Hahne, A., & Mauth, K. (2004). The relative timing of syntactic and semantic processes in sentence comprehension. NeuroReport: For Rapid Communication of Neuroscience Research, 15, 165-169.
5. Gaskell, M. G., & Marslen-Wilson, W. D. (1997). Integrating form and meaning: A distributed model of speech perception. Language and Cognitive Processes, 12, 613-656.
6. Green, A., Straube, B., Weiss, S., Jansen, A., Willmes, K., Konrad, K., et al. (2009). Neural integration of iconic and unrelated coverbal gestures: A functional MRI study. Human Brain Mapping, 30, 3309-3324.
7. Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24, 95-112.
8. Grosjean, F. (1980). Spoken word recognition processes and the gating paradigm. Perception & Psychophysics, 28, 267-283.
9. Grosjean, F. (1983). How long is the sentence? Prediction and prosody in the on-line processing of language. Linguistics, 21, 501-529.
10. Grosjean, F. (1996). Gating. Language and Cognitive Processes, 11, 597-604.
11. Gunter, T. C., Wagner, S., & Friederici, A. D. (2003). Working memory and lexical ambiguity resolution as revealed by ERPs: A difficult case for activation theories. Journal of Cognitive Neuroscience, 15, 643-657.
12. Hadar, U., & Pinchas-Zamir, L. (2004). The semantic specificity of gesture: Implications for gesture classification and unction. Journal of Language and Social Psychology, 23, 204-214.
13. Hinojosa, J. A., Martin-Loeches, M., & Rubia, F. J. (2001). Event-related potentials and semantics: An overview and an integrative proposal. Brain and Language, 78, 128-139.
14. Holle, H., & Gunter, T. C. (2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19, 1175-1192.
15. Holle, H., Gunter, T. C., Rueschemeyer, S. A., Hennenlotter, A., & Iacoboni, M. (2008). Neural correlates of the processing of co-speech gestures. Neuroimage, 39, 2010-2024.
16. Holle, H., Obleser, J., Rueschemeyer, S. A., & Gunter, T. C. (2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage, 49, 875-884.
17. Jansen, E., & Povel, D. J. (2004). The processing of chords in tonal melodic sequences. Journal of New Music Research, 33, 31-48.
18. Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253-260.
19. Kelly, S. D., Ward, S., Creigh, P., & Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 101, 222-233.
20. Kita, S., van Gijn, I., & van der Hulst, H. (1998). Movement phase in signs and co-speech gestures, and their transcriptions by human coders. Lecture Notes in Computer Science, 1371, 23-35.
21. Krauss, R. M., Morrel-Samuels, P., & Colasante, C. (1991). Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 61, 743-754.
22. Lausberg, H., & Kita, S. (2003). The content of the message influences the hand choice in co-speech gestures and in gesturing without speaking. Brain and Language, 86, 57-69.
23. Lausberg, H., & Slöetjes, H. (2008). Gesture coding with the NGCS-ELAN system. In A. J. Spink, M. R. Ballintijn, N. D. Rogers, F. Grieco, L. W. S. Loijens, L. P. J. J. Noldus, et al. (Eds.), Proceedings of Measuring Behavior 2008, 6th International Conference on Methods and Techniques in Behavioral Research (pp. 176-177). Maastricht: Noldus.
24. Levelt, W. J., Richardson, G., & la Heij, W. (1985). Pointing and voicing in deictic expressions. Journal of Memory and Language, 24, 133-164.
25. Marslen-Wilson, W. (1987). Functional parallelism in spoken word-recognition. Cognition, 25, 71-102.
26. Martin, C., Vu, H., Kellas, G., & Metcalf, K. (1999). Strength of discourse context as a determinant of the subordinate bias effect. Quarterly Journal of Experimental Psychology: Section A, Human Experimental Psychology, 52, 813-839.
27. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.
28. McNeill, D., Cassell, J., & McCullough, K.-E. (1994). Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction, 27, 223-237.
29. Morrel-Samuels, P., & Krauss, R. M. (1992). Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 615-622.
30. Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9, 97-113.
31. Özyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19, 605-616.
32. Posner, M. I., & Snyder, C. R. R. (1975). Attention and cognitive control. In R. L. Solso (Ed.), Information processing and cognition: The Loyola symposium. Hillsdale, NJ: Erlbaum.
33. Salasoo, A., & Pisoni, B. P. (1985). Interaction of knowledge sources in spoken word identification. Journal of Memory and Language, 24, 210-231.
34. Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information-processing: 1. Detection, search, and attention. Psychological Review, 84, 1-66.
35. Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information-processing: 2. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127-190.
36. Simpson, G. B. (1981). Meaning dominance and semantic context in the processing of lexical ambiguity. Journal of Verbal Learning and Verbal Behavior, 20, 120-136.
37. Simpson, G. B., & Krueger, M. (1991). Selective access of homograph meanings in sentence context. Journal of Memory and Language, 30, 627-643.
38. Swinney, D. A. (1979). Lexical access during sentence comprehension: (Re)consideration of context effects. Journal of Verbal Learning and Verbal Behavior, 18, 645-659.
39. Treffner, P., Peter, M., & Kleidon, M. (2008). Gestures and phases: The dynamics of speech-hand communication. Ecological Psychology, 20, 32-64.
40. Van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neuropsychologia, 45, 598-607.
41. Willems, R. M., Özyürek, A., & Hagoort, P. (2007). When language meets action. The neural integration of gesture and speech. Cerebral Cortex, 17, 2322-2333.
42. Wu, Y. C., & Coulson, S. (2005). Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology, 42, 654-667.