What speakers do and what addressees look at. Visual attention to gestures in human interaction live and on video

Pragmatics and Cognition

Volumn 14, Issue 1, 2006, Pages 53-82

  Gullberg, Marianne ^a   Holmqvist, Kenneth ^b  

^a MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^b LUND UNIVERSITY COGNITIVE SCIENCE   (Sweden)

Author keywords

Cross modal information processing; Eye gaze; Gesture; Interaction

Indexed keywords

EID: 33748423854     PISSN: 09290907     EISSN: 15699943     Source Type: Journal    
DOI: 10.1075/pc.14.1.05gul     Document Type: Article

References (91)

1. Argyle, M. and Cook, M. 1976. Gaze and Mutual gaze. Cambridge: Cambridge University Press.
2. Argyle, M. and Graham, J.A. 1976. "The Central Europe experiment: Looking at persons and looking at things" Journal of Environmental Psychology and Nonverbal Behavior 1(1): 6-16.
3. Bavelas, J.B., Coates, L., and Johnson, T. 2002. "Listener responses as a collaborative process: The role of gaze" Journal of Communication, September 2002: 566-580.
4. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., and Liu, G. 2001. "Impact of early deafness and early exposure to Sign Language on the cerebral organization for motion processing" Journal of Neuroscience 21(22): 8931-8942.
5. Beattie, G. and Shovelton, H. 1999a. "Do iconic hand gestures really contribute anything to the semantic information conveyed by speech?" Semiotica 123(1/2): 1-30.
6. Beattie, G. and Shovelton, H. 1999b. "Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech". Journal of Language and Social Psychology 18(4): 438-462.
7. Bruce, V. and Green, P. 1985. Visual perception. Physiology, psychology and ecology. Hillsdale, NJ: Erlbaum.
8. Brugman, H. and Kita, S. 1995. "Impact of digital video technology on transcription: A case of spontaneous gesture transcription" KODIKAS/CODE: Ars Semeiotica An international Journal of semiotics 18: 95-112.
9. Buswell, G.T. 1935. How People Look at Pictures. Chicago: The University of Chicago Press.
10. Callan, D.E., Jones, J.A., Munhall, K., Callan, A.M., Kroos, C., and Vatikiotis-Bateson, E. 2003. "Neural processes underlying perceptual enhancement by visual speech gestures". Neuroreport 14(17): 2213-2218.
11. Cassell, J., McNeill, D., and McCullough, K.-E. 1999. "Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information" Pragmatics & Cognition 7(1): 1-33.
12. Chun, M.M. 2000. "Contextual cueing of visual attention" Trends in Cognitive Sciences 4(5): 170-178.
13. Deak, G.O., Flom, R.A., and Pick, A.D. 2000. "Effects of gesture and target on 12-and 18-month-olds' joint visual attention to objects in front of or behind them" Developmental Psychology 36(4): 511-523.
14. Decety, J. and Grèzes, J. 1999. "Neural mechanisms subserving the perception of human actions" Trends in Cognitive Sciences 3(5): 172-178.
15. Doherty, M.J. and Anders, J.R. 1999. "A new look at gaze: Pre-school children's understanding of eye-direction". Cognitive Development 14: 549-571.
16. Driver, J., Davis, G., Ricciardelli, P., Kidd, P., Maxwell, E., and Baron-Cohen, S. 1999. "Gaze perception triggers reflexive visuospatial orienting". Visual Cognition 6(5): 509-540.
17. Farah, M.J. 2000, The Cognitive Neuroscience of Vision. Oxford: Blackwell.
18. Fehr, B.J. and Exline, R.V. 1987. "Social visual interaction: A conceptual and literature review". In A.W Siegman and S. Feldstein (eds), Nonverbal Behavior and Communication. Hillsdale, NJ: Erlbaum, 225-326.
19. Folk, C.L. and Remington, R. 1998. "Selectivity in distraction by irrelevant featural singletons: Evidence for two forms of attentional capture" Journal of Experimental Psychology: Human Perception and Performance 24(3): 847-858.
20. Folk, C.L., Remington, R.W., and Johnston, J.C. 1992. "Involuntary covert orienting is contingent on attentional control settings" Journal of Experimental Psychology: Human Perception and Performance 18(4): 1030-1044.
21. Gibson, J.J. and Pick, A.D. 1963. "Perception of another person's looking behavior". American Journal of Psychology 76(3): 386-394.
22. Goodwin, C. 1981. Conversational Organization: Interaction between Speakers and Hearers. New York: Academic Press.
23. Goodwin, C. 1986. "Gestures as a resource for the organization of mutual orientation" Semiotica 62(1/2): 29-49.
24. Gullberg, M. 1998. Gesture as a Communication Strategy in Second Language Discourse. A Study of Learners of French and Swedish. Lund: Lund University Press.
25. Gullberg, M. and Holmqvist, K. 1999. "Keeping an eye on gestures: Visual perception of gestures in face-to-face communication" Pragmatics & Cognition 7(1): 35-63.
26. Hayhoe, M. 2000. "Vision using routines: A functional account of vision". Visual Cognition 7(1/2/3): 43-64.
27. Hayhoe, M. and Ballard, D.H. 2005. "Eye movements in natural behavior". Trends in Cognitive Sciences 9(4): 188-194.
28. Henderson, J.M. 2003. "Human gaze control during real-world scene perception". Trends in Cognitive Sciences 7(11): 498-504.
29. Henderson, J.M. and Hollingworth, A. 1999. "High-level scene perception". Annual Review of Psychology 50: 243-271.
30. Hermsdörfer, J., Goldenberg, G., Wachsmuth, C., Conrad, B., Ceballos-Baumann, O., Bartenstein, P., Schwaiger, M., and Boecker, H. 2001. "Cortical correlates of gesture processing: Clues to the cerebral mechanisms underlying apraxia during the imitation of meaningless gestures". NeuroImage 14: 149-161.
31. Hoffman, J.E. 1998. "Visual attention and eye movements" In H. Pashler (ed), Attention. Hove: Psychology Press Ltd, 119-153.
32. Kelly, S.D., Barr, D.J., Breckinridge Church, R., and Lynch, K. 1999. "Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory". Journal of Memory and Language 40(4): 577-592.
33. Kendon, A. 1967. "Some functions of gaze direction in two-person conversations". Acta Psychologica 26: 22-63.
34. Kendon, A. 1972. "Some relationships between body motion and speech: An analysis of an example". In A.W. Siegman and B. Pope (eds), Studies in Dyadic Communication. New York: Pergamon, 177-210.
35. Kendon, A. 1973. "The role of visible behaviour in the organization of social interaction". In M. von Cranach and I. Vine (eds), Social Communication and Movement. Studies of Interaction and Expression in Man and Chimpanzee. New York: Academic Press, 29-74.
36. Kendon, A. 1980. "Gesticulation and speech: Two aspects of the process of utterance". In M. R. Key (ed), The Relationship of Verbal and Nonverbal Communication. The Hague: Mouton, 207-227.
37. Kendon, A. 1990. Conducting Interaction. Cambridge: Cambridge University Press.
38. Kendon, A. 1994. "Do gestures communicate?: A review" Research on Language and Social Interaction 27(3): 175-200.
39. Kendon, A. 2004. Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press.
40. Kita, S. (ed). 2003. Pointing: Where Language, Culture, and Cognition Meet. Mahwah, NJ: Erlbaum.
41. Klein, R.M. and Shore, D.I. 2000. "Relations among modes of visual orienting". In S. Monsell and J. Driver (eds), Attention and Performance XVIII. Cambridge, MA: The MIT Press, 195-208.
42. Kleinke, C.L. 1986. "Gaze and eye contact: A research review" Psychological Bulletin 100(1): 78-100.
43. Krauss, R.M., Chen, Y., and Chawla, P. 1996. "Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us?" Advances in Experimental Social Psychology 28: 389-450.
44. Land, M., Mennie, N., and Rusted, J. 1999. "The roles of vision and eye movements in the control of activities of daily living" Perception 28(11): 1311-1328.
45. Land, M.F. and Hayhoe, M. 2001. "In what ways do eye movements contribute to everyday activities" Vision Research 41(25-26): 3559-3565.
46. Langton, S.R.H. 2000. "The mutual influence of gaze and head orientation in the analysis of social attention direction". Quarterly Journal of Experimental Psychology 53(3): 825-845.
47. Langton, S.R.H. and Bruce, V. 1999. "Reflexive visual orienting in response to the social attention of others". Visual Cognition 6(5): 541-567.
48. Langton, S.R.H. and Bruce, V. 2000. "You must see the point: Automatic processing of cues to the direction of social attention". Journal of Experimental Psychology: Human Perception and Performance 26(2): 747-757.
49. Langton, S.R.H., O'Malley, C. and Bruce, V. 1996. "Actions speak no louder than words: Symmetrical cross-modal interference effects in the processing of verbal and gestural information". Journal of Experimental Psychology: Human Perception and Performance 22(6): 1357-1375.
50. Langton, S.R.H., Watt, R.J. and Bruce, V. 2000. "Do the eyes have it? Cues to the direction of social attention". Trends in Cognitive Sciences 4(2): 50-59.
51. Latham, K. and Whitaker, D. 1996. "A comparison of word recognition and reading performance in foveal and peripheral vision" Vision Research 37: 2665-2674.
52. Lombard, M. 1995. "Direct responses to people on the screen: Television and personal space" Communication Research 22(3): 288-324.
53. Ludwig, C.J.H. and Gilchrist, I.D. 2002. "Stimulus-driven and goal-driven control over visual selection". Journal of Experimental Psychology: Human Perception and Performance 28(4): 902-912.
54. McNeill, D. 1992. Hand and Mind. What the Hands Reveal about Thought. Chicago: The University of Chicago Press.
55. McNeill, D. 1998. "Speech and gesture integration" In J.M. Iverson and S. Goldin-Meadow (eds), The Nature and Functions of Gesture in Children's Communication. San Francisco: Jossey-Bass, 11-27.
56. McNeill, D., Cassell, J. and McCullough, K.-E. 1994. "Communicative effects of speech mismatched gestures". Research on Language and Social Interaction 27(3): 223-237.
57. McNeill, D., Levy, E.T., and Pedelty, L.L. 1990. "Speech and gesture" In G.R. Hammond (eds), Cerebral Control of Speech and Limb Movements. Amsterdam: North Holland, 203-256.
58. Melcher, D. and Kowler, F. 2001. "Visual scene memory and the guidance of saccadic eye movements". Vision Research 41: 3597-3611.
59. Melinger, A. and Levelt, W.J.M. 2004. "Gesture and the communicative intention of the speaker" Gesture 4(2): 119-141.
60. Moore, C. and Dunham, P.J. (eds). 1995. Joint Attention. Hillsdale, NJ: Erlbaum.
61. Nobe, S., Hayamizu, S., Hasegawa, O., and Takahashi, H. 1998. "Are listeners paying attention to the hand gestures of an anthropomorphic agent? An evaluation using a gaze tracking method". In I. Wachsmuth and M. Fröhlich (eds), Gesture and Sign Language in Human-Computer Interaction. Berlin: Springer, 49-59.
62. Nobe, S., Hayamizu, S., Hasegawa, O., and Takahashi, H. 2000. "Hand gestures of an anthropomorphic agent: Listeners' eye fixation and comprehensior" Cognitive Studies. Bulletin of the Japanese Cognitive Science Society 7(1): 86-92.
63. O'Regan, J.K. and Noë, A. 2001. "A sensorimotor account of vision and visual consciousness". Behavioral and Brain Sciences 24(5): 939-1031.
64. Peigneux, P., Salmon, E., van der Linden, M., Garraux, G., Aerts, J., Delfiore, G., Degueldre, C., Luxen, A., Orban, G., and Franck, G. 2000. "The role of lateral occipitotemporal junction and area MT/V5 in the visual analysis of upper-limb postures". NeuroImage 11:644-655.
65. Pelz, J., Hayhoe, M. and Loeber, R. 2001. "The coordination of eye, head, and hand movements in a natural task". Experimental Brain Research 139: 266-277.
66. Perani, D., Fazio, F., Borghese, N.A., Tettamanti, M., Ferrari, S., Decety, J., and Gilardi, M. C. 2001. "Different brain correlates for watching real and virtual hand actions". NeuroImage 14: 749-758.
67. Posner, M.I. 1980. "Orienting of attention". Quarterly Journal of Experimental Psychology 32(1): 3-25.
68. Raymond, J. E. 2000. "Attentional modulation of visual motion perception". Trends in Cognitive Sciences 4(2): 42-50.
69. Rettenbach, R., Diller, G. and Sireteanu, R. 1999. "Do deaf people see better? Texture segmentation and visual search compensate in adult but not in juvenile subjects". Journal of Cognitive Neuroscience 11(5): 560-583.
70. Rizzolatti, G., Fogassi, L., and Gallese, V. 2001. "Neurophysiological mechanisms underlying the understanding and imitation of action". Nature Reviews Neuroscience 2: 661-670.
71. Rutter, D.R. 1984. Looking and Seeing. The Role of Visual Communication in Social Interaction. Chichester: Wileys.
72. Schank, R.S. and Abelson, R.P. 1977. Scripts, Plans, Goals and Understanding. An Inquiry into Human Knowledge Structures. Hillsdale, N.J.: Erlbaum.
73. Shinoda, H., Hayhoe, M.M. and Shrivastava, A. 2001. "What controls attention in natural environments?". Vision Research 41(25-26): 3535-3545.
74. Slama-Cazacu, T. 1976. "Nonverbal components in message sequence: "Mixed syntax". In W.C. McCormack and S.A. Wurm (eds), Language and Man: Anthropological Issues. The Hague: Mouton, 217-227.
75. Streeck, J. 1993. "Gesture as communication I: Its coordination with gaze and speech" Communication Monographs 60(4): 275-299.
76. Streeck, J. 1994. "Gesture as communication II: The audience as co-author" Research on Language and Social Interaction 27(3): 239-267.
77. Streeck, J. and Knapp, M.L. 1992. "The interaction of visual and verbal features in human communication". In F. Poyatos (ed), Advances in Nonverbal Communication: Interdisciplinary Approaches through the Social and Clinical Sciences, Literature and the Arts. Amsterdam: John Benjamins, 3-23.
78. Swisher, M.V. 1990. "Developmental effects on the reception of signs in peripheral vision". Sign Language Studies 66: 45-60.
79. Swisher, M.X. 1993. "Perceptual and cognitive aspects of recognition of signs in peripheral vision". In M. Marschark and M.D. Clark (eds), Psychological Perspectives on Deafness. Hillsdale, NJ: Erlbaum, 209-228.
80. Theeuwes, J. 1994. "Endogenous and exogenous control of visual selection". Perception 23(4): 429-440.
81. Theeuwes, J., Atchley, P., and Kramer, A. F. 2000. "On the time course of top-down and bottom-up control of visual attention". In S. Monsell and J. Driver (eds), Attention and Performance XVIII. Cambridge, MA: The MIT Press, 105-124.
82. Thompson, L.A., Malmberg, J., Goodell, N.K., and Boring, R.L. 2004. "The distribution of attention across a talker's face". Discourse Processes 38(1): 145-168.
83. Tomasello, M. 1995. "Joint attention as social cognition". In C. Moore and P.J. Dunham (eds), Joint Attention. Hillsdale, NJ: Erlbaum, 103-130.
84. Turano, K.A., Geruschat, D.R., and Baker, F.H. 2003. "Oculomotor strategies for the direction of gaze tested with a real-world activity". Vision Research 43(3): 333-346.
85. Valenza, E., Simion, F., Macchi Cassia, V., and Umiltà, C. 1996. "Face preference at birth". Journal of Experimental Psychology: Human Perception and Performance 22: 892-903.
86. Vatikiotis-Bateson, E., Eigsti, I.-M., Yano, S. and Munhall, K.G. 1998. "Eye movement of perceivers during audiovisual speech perception". Perception and Psychophysics 60(6): 926-940.
87. Watson, O.M. 1970. Proxemic Behavior: A Cross-Cultural Study. The Hague: Mouton.
88. Wolfe, J.M. 1998. "Visual search". In H. Pashler (ed), Attention. Hove: Psychology Press Ltd, 13-73.
89. Yantis, S. 1998. "Control of visual attention". In H. Pashler (ed), Attention. Hove: Psychology Press, 223-256.
90. Yantis, S. 2000. "Goal-directed and stimulus-driven determinants of attentional control". In S. Monsell and J.Driver (eds), Attention and Performance XVIII. Cambridge, MA: MIT Press, 73-103.
91. Yarbus, A. 1967. Eye Movements and Vision. New York: Plenum Press.