Eye'm talking to you: Speakers' gaze direction modulates co-speech gesture processing in the right MTG

Social Cognitive and Affective Neuroscience

Volumn 10, Issue 2, 2015, Pages 255-261

  Holler, Judith ^a   Kokal, Idil ^b   Toni, Ivan ^b   Hagoort, Peter ^a,b   Kelly, Spencer D ^c   Özyürek, Asli ^a,d  

^a MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^b RADBOUD UNIVERSITY   (Netherlands)

^c Assistant Professor in the University Libraries   (United States)

^d RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Co speech gestures; Communicative intent; Eye gaze; Middle temporal gyrus; Speech gesture integration

Indexed keywords

ASSOCIATION; BRAIN MAPPING; CINGULATE GYRUS; COMPREHENSION; EYE FIXATION; FEMALE; GESTURE; HUMAN; INTERPERSONAL COMMUNICATION; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; SPEECH; YOUNG ADULT;

BRAIN MAPPING; COMMUNICATION; COMPREHENSION; CUES; FEMALE; FIXATION, OCULAR; GESTURES; GYRUS CINGULI; HUMANS; MAGNETIC RESONANCE IMAGING; SPEECH; YOUNG ADULT;

EID: 84928693257     PISSN: 17495016     EISSN: 17495024     Source Type: Journal    
DOI: 10.1093/scan/nsu047     Document Type: Article

References (70)

1. Argyle, M., Cook, M. (1976). Gaze and Mutual Gaze. Cambridge: Cambridge University Press.
2. Ashburner, J., Friston, K.J. (2005). Unified segmentation. Neuroimage, 26, 839-51.
3. Beeman, M.J., Chiarello, C. (1998). Conplementary right-and left-hemisphere language comprehension. Current Directions in Psychological Science, 7, 2-7.
4. Bottini, G., Corcoran, R., Sterzi, R., et al. (1994). The role of the right hemisphere in the interpretation of figurative aspects of language: A positron emission tomography activation study. Brain, 117, 1241-53.
5. Brunet, E., Sarfati, Y., Hardy-Baylé, M.-C., Decety, J. (2000). A PET investigation of the attribution of intentions with a nonverbal task. Neuroimage, 11, 157-66.
6. Cleret de Langavant, L., Remy, P., Trinkler, I., et al. (2011). Behavioral and neural correlates of communication via pointing. PLoS One, 6, e17719.
7. Conty, L., Dezecache, G., Hugueville, L., Grèzes, J. (2012). Early binding of gaze, gesture and emotion: neural time course and correlates. Journal of Neuroscience, 32, 4531-9.
8. Coulson, S., Wu, Y.C. (2005). Right hemisphere activation of joke-related information: an event-related brain potential study. Journal of Cognitive Neuroscience, 17, 494-506.
9. Dick, A.S., Goldin-Meadow, S., Hasson, U., Skipper, J., Small, S.L. (2009). Co-speech gestures influence neural responses in brain regions associated with semantic processing. Human Brain Mapping, 30, 3509-26.
10. Dick, A.S., Mok, E.H., Beharelle, A.R., Goldin-Meadow, S., Small, S.L. (2012). Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech. Human Brain Mapping, 35, 900-17.
11. Eickhoff, S.B., Stephan, K.E., Mohlberg, H., et al. (2005). A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage, 25, 1325-35.
12. Emmorey, K., Mehta, S., Grabowski, T.J. (2007). The neural correlates of sign versus word production. Neuroimage, 36, 202-8.
13. Enrici, I., Adenzato, M., Cappa, S., Bara, B.G., Tettamanti, M. (2011). Intention processing in communication: a common brain network for language and gestures. Journal of Cognitive Neuroscience, 23, 2415-31.
14. Farroni, T., Csibra, G., Simion, F., Johnson, M.H. (2002). Eye contact detection in humans from birth. Proceedings of the National Academy of Sciences USA, 99, 9602-5.
15. Fletcher, P.C., Happé, F., Frith, U., et al. (1995). Other minds in the brain: a functional imaging study of "theory of mind" in story comprehension. Cognition, 57, 109-28.
16. Goffman, E. (1981). Forms of Talk. Philadelphia: University of Pennsylvania Press.
17. Goodwin, C. (1981). Conversational Organization: Interaction between Speakers and Hearers. New York: Academic Press.
18. Green, A., Straube, B., Weis, S., et al. (2009). Neural integration of iconic and unrelated coverbal gestures: a functional MRI study. Human Brain Mapping, 30, 3309-24.
19. Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cognitive Sciences, 9, 416-23.
20. Holle, H., Gunter, T.C. (2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19, 1175-92.
21. Holle, H., Gunter, T.C., Rüschemeyer, S.A., Hennenlotter, A., Iacoboni, M. (2008). Neural correlates of the processing of co-speech gestures. Neuroimage, 39, 2010-24.
22. Holle, H., Obermeier, C., Schmidt-Kassow, M., Friederici, A.D., Ward, J., Gunter, T.C. (2012). Gesture facilitates the syntactic analysis of speech. Frontiers in Psychology, 3, 74.
23. Holle, H., Obleser, J., Rüschemeyer, S.A., Gunter, T.C. (2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage, 49, 875-84.
24. Holler, J., Kelly, S., Hagoort, P., Ozyurek, A. (2012). When gestures catch the eye: the influence of gaze direction on co-speech gesture comprehension in triadic communication. In: Miyake, N., Peebles, D., Cooper, R.P., editors. Proceedings of the 34th Annual Meeting of the Cognitive Science Society. Austin, TX: Cognitive Society, pp. 467-72.
25. Kable, J.W., Kan, I.P., Wilson, A., Thompson-Schill, S.L., Chatterjee, A. (2005). Conceptual representations of action in the lateral temporal cortex. Journal of Cognitive Neuroscience, 17, 1855-70.
26. Kampe, K., Frith, C.D., Frith, U. (2003). 'Hey John': signals conveying communicative intention towards the self activate brain regions associated with mentalising regardless of modality. Journal of Neuroscience, 23, 5258-63.
27. Kelly, S.D., Barr, D., Church, R.B., Lynch, K. (1999). Offering a hand to pragmatic understanding: the role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40, 577-92.
28. Kelly, S.D., Creigh, P., Bartolotti, J. (2010a). Integrating speech and iconic gestures in a Stroop-like task: evidence for automatic processing. Journal of Cognitive Neuroscience, 22, 683-94.
29. Kelly, S.D., Kravitz, C., Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253-60.
30. Kelly, S.D., ö zyürek, A., Maris, E. (2010b). Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychological Science, 21, 260-7.
31. Kelly, S.D., Ward, S., Creigh, P., Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 101, 222-33.
32. Kendon, A. (1967). Some functions of gaze direction in social interaction. Acta Psychologica, 26, 22-63.
33. Kendon, A. (2004). Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
34. Kircher, T., Straube, B., Leube, D., et al. (2009). Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures. Neuropsychologia, 47, 169-79.
35. Leech, R., Braga, R., Sharp, D.J. (2012). Echoes of the brain within the posterior cingulate cortex. The Journal of Neuroscience, 32, 215-22.
36. Lund, T.E., Nbrgaard, M.D., Rostrup, E., Rowe, J.B., Paulson, O.B. (2005). Motion or activity: their role in intra-and inter-subject variation in fMRI. Neuroimage, 26, 960-4.
37. MacSweeney, M., Capek, C.M., Campbell, R., Woll, B. (2008). The signing brain: the neurobiology of sign language. Trends in Cognitive Sciences, 12, 432-40.
38. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press.
39. Mosconi, M.W., Mack, P.B., McCarthy, G., Pelphrey, K.A. (2005). Taking an "intentional stance" on eye-gaze shifts: A functional neuroimaging study of social perception in children. Neuroimage, 27, 247-52.
40. Noordzij, M.L., Newman-Norlund, S.E., Ruiter, J.P.A.de, Hagoort, P., Levinson, S.C., Toni, I. (2009). Brain mechanisms underlying human communication. Frontiers in Human Neuroscience, 3, art. 14.
41. özyürek, A., Willems, R.M., Kita, S., Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19, 605-16.
42. Parise, E., Handl, A., Palumbo, L., Friederici, A.D. (2011). Influence of eye gaze on word processing: an ERP study with infants. Child Development, 82, 842-53.
43. Paz Fonseca, R., Scherer, L.C., de Oliveira, C.R., de Mattos Pimenta Parente, M.A. (2009). Hemispheric specialization for communicative processing: neuroimaging data on the role of the right hemisphere. Psychology & Neuroscience, 2, 25-33.
44. Pelphrey, K.A., Perlman, S.B. (2009). Charting brain mechanisms for the development of social cognition. In: Rumsey, J.M., Ernst, M., editors. Neuroimaging in Developmental Clinical Neuroscience. Cambridge: Cambridge University Press, pp. 73-90.
45. Pelphrey, K.A., Singerman, J.D., Allison, T., McCarthy, G. (2003). Brain activation evoked by the perception of gaze shifts: the influence of context. Neuropsychologia, 41, 156-70.
46. Pelphrey, K.A., Viola, R.J., McCarthy, G. (2004). When strangers pass: processing of mutual and averted gaze in the superior temporal sulcus. Psychological Science, 15, 598-603.
47. Pfeiffer, U.J., Schilbach, L., Jording, M., Timmermans, B., Bente, G., Vogeley, K. (2012). Eyes on the mind: investigating the influence of gaze dynamics on the perception of others in real-time social interaction. Frontiers in Cognitive Science, 3, 537.
48. Poser, B.A., Versluis, M.J., Hoogduin, J.M., Norris, D.G. (2006). BOLD contrast sensitivity enhancement and artifact reduction with multiecho EPI: Parallel-acquired inhomogeneity-desensitized fMRI. Magnetic Resonance in Medicine, 55, 1227-35.
49. Ruby, P., Decety, J. (2001). Effect of subjective perspective taking during simulation of action: a PET investigation of agency. Nature Neuroscience, 4, 546-50.
50. Sabbagh, M. (1999). Communicative intentions and language: evidence from righthemisphere damage and autism. Brain and Language, 70, 29-69.
51. Schilbach, L., Wohlschläger, A.M., Newen, A., et al. (2006). Being with others: neural correlates of social interaction. Neuropsychologia, 44, 718-30.
52. Schippers, M.B., Gazzola, V., Goebel, R., Keysers, C. (2009). Playing Charades in the fMRI: Are mirror and/or mentalizing areas involved in gestural communication? PLoS One, 4, e6801.
53. Schippers, M.B., Roebroeck, A., Renken, R.J., Nanetti, L., Keysers, C. (2010). Mapping the information flow from one brain to another during gestural communication. Proceedings of the National Academy of Sciences USA, 107, 9388-93.
54. Senju, A., Johnson, M.H. (2009). The eye contact effect: Mechanisms and development. Trends in Cognitive Sciences, 13, 127-34.
55. Skipper, J.I., Goldin-Meadow, S., Nusbaum, H.C., Small, S.L. (2007). Speech associated gestures, Broca's area, and the human mirror system. Brain and Language, 101, 260-77.
56. Skipper, J.I., Goldin-Meadow, S., Nusbaum, H.C., Small, S.L. (2009). Gestures orchestrate brain networks for language understanding. Current Biology, 19, 661-7.
57. Stoyanova, R.S., Ewbank, M.P., Calder, A.J. (2010). You talkin' to me? Self-relevant auditory signals influence perception of gaze direction. Psychological Science, 21, 1765-9.
58. Straube, B., Green, A., Bromberger, B., Kircher, T. (2011a). The differentiation of iconic and metaphoric gestures: common and unique integration processes. Human Brain Mapping, 32, 520-33.
59. Straube, B., Green, A., Chatterjee, A., Kircher, T. (2011b). Encoding social interactions: the neural correlates of true and false memories. Journal of Cognitive Neuroscience, 23, 306-24.
60. Straube, B., Green, A., Jansen, A., Chatterjee, A., Kircher, T. (2010). Social cues, mentalizing and the neural processing of speech accompanied by gestures. Neuropsychologia, 48, 382-93.
61. Straube, B., Green, A., Weis, S., Kircher, T. (2012). A supramodal neural network for speech and gesture semantics: an fMRI study. PLoS One, 7, e51207.
62. Verhagen, L., Dijkerman, H.C., Grol, M.J., Toni, I. (2008). Perceptuo-motor interactions during prehension movements. The Journal of Neuroscience, 28, 4726-35.
63. Vogeley, K., Bente, G. (2010). 'Artificial humans': psychology and neuroscience perspectives on embodiment and nonverbal communication. Neural Networks, 23, 1077-90.
64. Weed, E. (2011). What's left to learn about right hemisphere damage and pragmatic impairment? Aphasiology, 25, 872-89.
65. Weylman, S.T., Brownell, H.H., Roman, M., Gardner, H. (1989). Appreciation of indirect requests by left-and right-brain-damaged patients: the effects of context and conventionality of wording. Brain and Language, 36, 580-91.
66. Willems, R.M., özyürek, A., Hagoort, P. (2007). When language meets action: The neural integration of gesture and speech. Cerebral Cortex, 17, 2322-33.
67. Willems, R.M., Özyürek, A., Hagoort, P. (2009). Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage, 47, 1992-2004.
68. Wilms, M., Schilbach, L., Pfeiffer, U., Bente, G., Fink, G.R., Vogeley, K. (2010). It's in your eyes_using gaze-contingent stimuli to create truly interactive paradigms for social cognitive and affective neuroscience. Social Cognitive and Affective Neuroscience, 5, 98-107.
69. Wu, Y.C., Coulson, S. (2007). How iconic gestures enhance communication: An ERP study. Brain and Language, 101, 234-45.
70. Xu, J., Gannon, P., Emmorey, K., Smith, J.F., Braun, A.R. (2009). Symbolic gestures and spoken language are processed by a common neural system. Proceedings of the National Academy of Sciences USA, 106, 20664-9.