How saliency, faces, and sound influence gaze in dynamic social scenes

Journal of Vision

Volumn 14, Issue 8, 2014, Pages

  Coutrot, Antoine ^a   Guyader, Nathalie ^a  

^a UNIV GRENOBLE ALPES   (France)

Author keywords

Audiovisual integration; Database; Expectation maximization; Faces; Gaze; Saliency; Scanpath; Speech

Indexed keywords

ADULT; ARTICLE; AUDIOVISUAL INTEGRATION; DATA BASE; EXPECTATION-MAXIMIZATION; EYE FIXATION; EYE MOVEMENT; FACE; FACES; FEMALE; GAZE; HUMAN; MALE; PHYSIOLOGY; SALIENCY; SCANPATH; SOCIAL STATUS; SOUND DETECTION; SPEECH; VISION; YOUNG ADULT;

AUDIOVISUAL INTEGRATION; DATABASE; EXPECTATION-MAXIMIZATION; FACES; GAZE; SALIENCY; SCANPATH; SPEECH;

ADULT; EYE MOVEMENTS; FACE; FEMALE; FIXATION, OCULAR; HUMANS; MALE; SOCIAL CONDITIONS; SOUND LOCALIZATION; VISION, OCULAR; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84904258424     PISSN: None     EISSN: 15347362     Source Type: Journal    
DOI: 10.1167/14.8.5     Document Type: Article

References (73)

1. Anguera, X., Bozonnet, S., Evans, N., Fredouille, C., Friedland, G., & Vinyals, O. (2012). Speaker diarization: A review of recent research. IEEE Transaction on Audio, Speech, & Language Processing, 20(2), 356-370.
2. Bahill, T., Adler, D., & Stark, L. (1975, June). Most naturally occurring human saccades have magnitudes of 15 degrees or less. Investigative Ophthalmology & Visual Science, 14(6), 468-469, http:// www.iovs.org/content/14/6/468. [PubMed] [Article]
3. Bailly, G., Perrier, P., & Vatikiotis-Bateson, E. (2012). Audiovisual speech processing. Cambridge, UK: Cambridge University Press.
4. Bailly, G., Raidt, S., & Elisei, F. (2010). Gaze, conversational agents, and face-to-face communication. Speech Communication, 52, 598-612.
5. Berthommier, F. (2004). A phonetically neutral model of the low-level audiovisual interaction. Speech Communication, 44(1-4), 31-41.
6. Bertolino, P. (2012). Sensarea: An authoring tool to create accurate clickable videos. In 10th workshop on content-based multimedia indexing (pp. 1-4). Annecy, France.
7. Bindemann, M., Burton, A. M., Hooge, I. T. C., Jenkins, R., & de Haan, E. H. F. (2005). Faces retain attention. Psychonomic Bulletin & Review, 12(6), 1048-1053.
8. Bindemann, M., Burton, A. M., Langton, S. R. H., Schweinberger, S. R., & Doherty, M. J. (2007). The control of attention to faces. Journal of Vision, 7(10):15, 1-8, http://www.journalofvision.org/ content/7/10/15, doi:10.1167/7.10.15. [PubMed] [Article]
9. Birmingham, E., & Kingstone, A. (2009). Saliency does not account for fixations to eyes within social scenes. Vision Research, 49, 2992-3000.
10. Blauth, D. A., Minotto, V. P., Jung, C. R., Lee, B., & Kalker, T. (2012). Voice activity detection and speaker localization using audiovisual cues. Pattern Recognition Letters, 33(4), 373-380.
11. Boltz, M. G. (2004). The cognitive processing of film and musical soundtracks. Memory & Cognition, 32(7), 1194-1205.
12. Boremanse, A., Norcia, A., & Rossion, B. (2013). An objective signature for visual binding of face parts in the human brain. Journal of Vision, 13(11):6, 1-18, http://www.journalofvision.org/content/13/11/ 6, doi:10.1167/13.11.6. [PubMed] [Article]
13. Borji, A., & Itti, L. (2012). State-of-the-art in visual attention modeling. IEEE Transactions on Patterns Analysis and Machine Intelligence, 35(1), 185-207.
14. Branigan, E. (2010). Soundtrack in mind. Projections, 4(1), 41-67.
15. Buchan, J. N., Paŕe, M., & Munhall, K. G. (2007). Spatial statistics of gaze fixations during dynamic face processing. Social Neuroscience, 2(1), 1-13.
16. Buswell, G. T. (1935). How people look at pictures: A study of the psychology of perception in art. Chicago: University of Chicago Press.
17. Calvert, G., Spence, C., & Stein, B. E. (2004). Handbook of multisensory processes. Cambridge, MA: MIT Press.
18. Carpenter, R. H. S. (1988). Movements of the eyes (2nd rev. & enlarged ed.). London, England: Pion Limited.
19. Cerf, M., Harel, J., Einhäuser, W., & Koch, C. (2008). Predicting human gaze using low-level saliency combined with face detection. In Advances in neural information processing systems (pp. 241-248).
20. Chion, M. (1994). Audio-vision: Sound on screen. New York: Columbia University Press.
21. Cohen, A. J. (2005). How music influences the interpretation of film and video: Approaches from experimental psychology. In R. A. Kendall & R. W. H. Savage (Eds.), Selected reports in ethnomusicology: Perspectives in systematic musicology (pp. 15-36). Los Angeles: Department of Ethnomusicology, University of California.
22. Coutrot, A., & Guyader, N. (2013). Toward the introduction of auditory information in dynamic visual attention models. In IEEE international workshop on image analysis for multimedia interactive services (WIAMIS) (pp. 1-4). Paris, France.
23. Coutrot, A., Guyader, N., Ionescu, G., & Caplier, A. (2012). Influence of soundtrack on eye movements during video exploration. Journal of Eye Movement Research, 5(4), 1-10.
24. Crouzet, S. M., Kirchner, H., & Thorpe, S. J. (2010). Fast saccades toward faces: Face detection in just 100 ms. Journal of Vision, 10(4):16, 1-17, http:// www.journalofvision.org/content/10/4/16, doi:10. 1167/10.4.16. [PubMed] [Article]
25. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1-38.
26. Farah, M. J., Wilson, K. D., Drain, M., & Tanaka, J. N. (1998). What is ''special'' about face perception? Psychological Review, 105(3), 482-498.
27. Foulsham, T., Cheng, J. T., Tracy, J. L., Henrich, J., & Kingstone, A. (2010). Gaze allocation in a dynamic situation: Effects of social status and speaking. Cognition, 117(3), 319-331.
28. Foulsham, T., & Sanderson, L. A. (2013). Look who's talking? Sound changes gaze behaviour in a dynamic social scene. Visual Cognition, 21(7), 922-944.
29. Frank, M. C., Vul, E., & Johnson, S. P. (2009). Development of infants' attention to faces during the first year. Cognition, 110, 160-170.
30. Gautier, J., & Le Meur, O. (2012). A time-dependent saliency model combining center and depth biases for 2D and 3D viewing conditions. Cognitive Computation, 4, 1-16.
31. Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2000). The distributed human neural system for face perception. Trends in Cognitive Sciences, 4(6), 223-233.
32. Hershler, O., & Hochstein, S. (2005). At first sight: A high-level pop out effect for faces. Vision Research, 45, 1707-1724.
33. Hirvenkari, L., Ruusuvori, J., Saarinen, V.-M., Kivioja, M., Peräkylä, A., & Hari, R. (2013). Influence of turn-taking in a two-person conversation on the gaze of a viewer. PLoS ONE, 8(8), 1-6.
34. Ho-Phuoc, T., Guyader, N., & Guerin-Dugue, A. (2010). A functional and statistical bottom-up saliency model to reveal the relative contributions of low-level visual guiding factors. Cognitive Computation, 2(4), 344-359.
35. Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 1489-1506.
36. Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Patterns Analysis and Machine Intelligence, 20(11), 1254-1259.
37. Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11), 4302-4311.
38. Kayser, C., Petkov, C. I., Lippert, M., & Logothetis, N. K. (2005). Mechanisms for allocating auditory attention: An auditory saliency map. Current Biology, 15, 1943-1947.
39. Koch, C., & Ullman, S. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry. Human Neurobiology, 4, 219-227.
40. Lansing, C. R., & McConkie, G. W. (2003). Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception & Psychophysics, 65(4), 536-552.
41. Le Meur, O., & Baccino, T. (2013). Methods for comparing scanpaths and saliency maps: Strengths and weaknesses. Behavior Research Methods, 45(1), 251-266.
42. Le Meur, O., Le Callet, P., & Barba, D. (2007). Predicting visual fixations on video based on lowlevel visual features. Vision Research, 47, 2483-2498.
43. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8), 707-710.
44. Marat, S., Ho-Phuoc, T., Granjon, L., Guyader, N., Pellerin, D., & Gúerin-Dugúe, A. (2009). Modelling spatio-temporal saliency to predict gaze direction for short videos. International Journal of Computer Vision, 82(3), 231-243.
45. Marat, S., Rahman, A., Pellerin, D., Guyader, N., & Houzet, D. (2013). Improving visual saliency by adding 'face feature map' and 'center bias'. Cognitive Computation, 5(1), 63-75.
46. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
47. Mital, P. K., Smith, T. J., Hill, R. L., & Henderson, J. M. (2010). Clustering of gaze during dynamic scene viewing is predicted by motion. Cognitive Computation, 3(1), 5-24.
48. Nahorna, O., Berthommier, F., & Schwartz, J.-L. (2012). Binding and unbinding the auditory and visual streams in the McGurk effect. Journal of the Acoustical Society of America, 132(2), 1061-1077.
49. Noulas, A. K., Englebienne, G., & Kröse, B. J. A. (2012). Multimodal speaker diarization. IEEE Transactions on Pattern Analysis & Machine Intelligence, 34(1), 79-93.
50. Nyström, M., & Holmqvist, K. (2008). Semantic override of low-level features in image viewing- both initially and overall. Journal of Eye Movement Research, 2(2), 1-11.
51. Onat, S., Libertus, K., & König, P. (2007). Integrating audiovisual information for the control of overt attention. Journal of Vision, 7(10):11, 1-16, http:// www.journalofvision.org/content/7/10/11, doi:10. 1167/7.10.11. [PubMed] [Article]
52. Pannasch, S., Helmert, J. R., Herbold, A.-K., Roth, K., & Henrik, W. (2008). Visual fixation durations and saccade amplitudes: Shifting relationship in a variety of conditions. Journal of Eye Movement Research, 2(4), 1-19.
53. Quigley, C., Onat, S., Harding, S., Cooke, M., & König, P. (2008). Audio-visual integration during overt visual attention. Journal of Eye Movement Research, 1(2), 1-17.
54. Richardson, D., Dale, R., & Shockley, K. (2008). Synchrony and swing in conversation: Coordination, temporal dynamics, and communication. In I. Wachsmuth, M. Lenzen, & G. Knoblich (Eds.), Embodied communication (pp. 75-94). New York: Oxford University Press.
55. Rudoy, D., Goldman, D. B., Shechtman, E., & Zelnik- Manor, L. (2013). Learning video saliency from human gaze using candidate selection. In Conference on computer vision and pattern recognition (pp. 4321-4328). Portland, OR.
56. Schwartz, J.-L., Robert-Ribes, J., & Escudier, P. (1998). Ten years after Summerfield: A taxonomy of models of audiovisual fusion in speech perception. In R. Campbell, B. Dodd, & D. K. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 85-108). Hove, UK: Psychology Press.
57. Smith, T. J., & Mital, P. K. (2013). Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes. Journal of Vision, 13(8):16, 1-24, http://www. journalofvision.org/content/13/8/16, doi:10.1167/ 13.8.16. [PubMed] [Article]
58. Song, G., Pellerin, D., & Granjon, L. (2013). Different types of sounds influence gaze differently in videos. Journal of Eye Movement Research, 6(4), 1-13.
59. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lipreading (pp. 3-51). New York: Lawrence Erlbaum.
60. Tatler, B. W. (2007). The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image featuredistributions. Journal of Vision, 7(14):4, 1-17, http://www.journalofvision.org/content/7/14/4, doi:10.1167/7.14.4. [PubMed] [Article]
61. Tatler, B. W., Baddeley, R. J., & Vincent, B. T. (2006). The long and the short of it: Spatial statistics at fixation vary with saccade amplitude and task. Vision Research, 46, 1857-1862.
62. Tatler, B. W., Hayhoe, M. M., Land, M. F., & Ballard, D. H. (2011). Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(5):5, 1-23, http://www.journalofvision.org/content/11/5/ 5, doi:10.1167/11.5.5. [PubMed] [Article]
63. Theeuwes, J., & Van der Stigchel, S. (2006). Faces capture attention: Evidence from inhibition of return. Visual Cognition, 13(6), 657-665.
64. Treisman, A. M., & Gelade, G. (1980). A featureintegration theory of attention. Cognitive Psychology, 12, 97-136.
65. Tseng, P.-H., Carmi, R., Cameron, I. G. M., Munoz, D. P., & Itti, L. (2009). Quantifying center bias of observers in free viewing of dynamic natural scenes. Journal of Vision, 9(7):4, 1-16, http://www. journalofvision.org/content/9/7/4, doi:10.1167/9.7. 4. [PubMed] [Article]
66. Vatakis, A., & Spence, C. (2006). Audiovisual synchrony perception for music, speech, and object actions. Brain Research, 1111, 134-142.
67. Vatikiotis-Bateson, E., Eigsti, I.-M., Yano, S., & Munhall, K. G. (1998). Eye movement of perceivers during audiovisualspeech perception. Perception & Psychophysics, 60(6), 926-940.
68. Vincent, B. T., Baddeley, R. J., Correani, A., Troscianko, T., & Leonards, U. (2009). Do we look at lights? Using mixture modelling to distinguish between low- and high-level factors in natural image viewing. Visual Cognition, 17(6-7), 856-879.
69. Võ, M. L. H., Smith, T. J., Mital, P. K., & Henderson, J. M. (2012). Do the eyes really have it? Dynamic allocation of attention when viewing moving faces. Journal of Vision, 12(13):3, 1-14, http://www. journalofvision.org/content/12/13/3, doi:10.1167/ 12.13.3. [PubMed] [Article]
70. Vroomen, J., & Stekelenburg, J. J. (2011). Perception of intersensory synchrony in audiovisual speech: Not that special. Cognition, 118(1), 75-83.
71. Yang, Q., Bucci, M. P., & Kapoula, Z. (2002). The latency of saccades, vergence, and combined eye movements in children and in adults. Investigative Ophthalmology & Visual Science, 43(9), 2939-2949, http://www.iovs.org/content/43/9/2939. [PubMed] [Article]
72. Yarbus, A. L. (1967). Eye movements and vision. New York: Plenum.
73. Zeppelzauer, M., Mitrovic, D., & Breiteneder, C. (2011). Cross-modal analysis of audio-visual film montage. In International conference on computer communications and networks (pp. 1-6). Maui, Hawaii.