Defending yarbus: Eye movements reveal observers' task

Journal of Vision

Volumn 14, Issue 3, 2014, Pages

  Borji, Ali ^a   Itti, Laurent ^a,b  

^a University of Southern California   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Bottom up saliency; Eye movements; Free viewing; Mind reading; Task decoding; Top down attention; Visual attention; Visual search; Yarbus

Indexed keywords

ARTICLE; ATTENTION; BOTTOM-UP SALIENCY; COGNITION; EYE MOVEMENT; FEMALE; FREE VIEWING; HUMAN; MALE; MIND READING; PATTERN RECOGNITION; PHYSIOLOGY; PSYCHOMOTOR PERFORMANCE; TASK DECODING; TOP-DOWN ATTENTION; VISUAL ATTENTION; VISUAL SEARCH; YARBUS; YOUNG ADULT;

BOTTOM-UP SALIENCY; EYE MOVEMENTS; FREE VIEWING; MIND READING; TASK DECODING; TOP-DOWN ATTENTION; VISUAL ATTENTION; VISUAL SEARCH; YARBUS;

ATTENTION; COGNITION; EYE MOVEMENTS; FEMALE; HUMANS; MALE; PATTERN RECOGNITION, VISUAL; PSYCHOMOTOR PERFORMANCE; YOUNG ADULT;

EID: 84897043953     PISSN: None     EISSN: 15347362     Source Type: Journal    
DOI: 10.1167/14.3.29     Document Type: Article

References (116)

1. Albert, M., Toledo, S., Shapiro, M., & Kording, K. (2012). Using mobile phones for activity recognirecognition in Parkinson's patients. Frontiers in Neurology, 3, 158.
2. Ballard, D., Hayhoe, M., & Pelz, J. (1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7, 66-80.
3. Bertram, R., Helle, L., Kaakinen, J. K., & Svedstr̈om, E. (2013). The effect of expertise on eye movement behaviour in medical image perception. Plos One, 8.
4. Betz, T., Kietzmann, T., Wilming, N., & K̈onig, P. (2010). Investigating task-dependent top-down effects on overt visual attention. Journal of Vision, 10(3):15, 1-14, http://www.journalofvision.org/ content/10/3/15, doi:10.1167/10.3.15. [PubMed] [Article]
5. Borji, A. (2012). Boosting bottom-up and top-down visual features for saliency estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 438-445).
6. Borji, A., & Itti, L. (2013). State-of-the-art in modeling visual attention. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), 35, 185-207.
7. Borji, A., Sihite, D. N., & Itti, L. (2012a). Probabilistic learning of task-specific visual attention. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 470-477).
8. Borji, A., Sihite, D. N., & Itti, L. (2012b). Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study. IEEE Trans. Image Processing, 22, 55-69.
9. Borji, A., Sihite, D. N., & Itti, L. (2014). What/where to look next? Modeling top-down visual attention in complex interactive environments. IEEE Transactions on Systems, Man, and Cybernetics, Part ASystems and Humans, in press.
10. Borji, A., Tavakoli, H. R., Sihite, D. N., & Itti, L. (2013). Analysis of scores, datasets, and models in visual saliency prediction. International Conference on Computer Vision (ICCV) (pp. 921-928).
11. Brandt, S. A., & Stark, L. W. (1997). Spontaneous eye movements during visual imagery reflect the content of the visual scene. Journal of Cognitive Neuroscience, 9, 27-38.
12. Bruce, N., & Tsotsos, J. (2009). Saliency, attention, and visual search: An information theoretic approach. Journal of Vision, 9(3):5, 1-24, http://www. journalofvision.org/content/9/3/5, doi:10.1167/9.3. 5. [PubMed] [Article]
13. Bulling, A., Ward, J. A., Gellersen, H., & Tröster, G. (2011). Eye movement analysis for activity recognition using electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 741-753.
14. Bundesen, C., Habekost, T., & Kyllingsboek, S. (2005).A neural theory of visual attention: Bridging cognition and neurophysiology. Psychological Review, 112, 291.
15. Buswell, G. (1935). How people look at pictures. Chicago: University of Chicago Press.
16. Coen-Cagli, R., Coraggio, P., Napoletano, P., Schwartz, O., Ferraro, M., & Boccignone, G. (2009). Visuomotor characterization of eye movements in a drawing task. Vision Research, 49.
17. Castelhano, M., Mack, M., & Henderson, J. (2009). Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3):6, 1-15, http://www.journalofvision.org/ content/9/3/6, doi:10.1167/9.3.6. [PubMed] [Article]
18. Cerf, M., Frady, E. P., & Koch, C. (2009). Faces and text attract gaze independent of the task: Experimental data and computer model. Journal of Vision, 9(12):10, 1-15, http://www.journalofvision.org/ content/9/12/10, doi:10.1167/9.12.10. [PubMed] [Article]
19. Chen, L., Meier, K. M., Blair, M. R., Watson, M. R., & Wood, M. J. (2013). Temporal characteristics of overt attentional behaviour during category learning. Attention Perception & Psychophysics, 75(2), 244-256.
20. Chua, H. F., Boland, J. E., & Nisbett, R. E. (2005). Cultural variation in eye movements during scene perception. Proceedings of the National Academy of Sciences, USA, 102, 12629-12633.
21. Clark, J. J., & O'Regan, J. K. (1998). Word ambiguity and the optimal viewing position in reading. Vision Research, 39, 843-857.
22. Crespi, S., Robino, C., Silva, O., & deSperati, C. (2012). Spotting expertise in the eyes: Billiards knowledge as revealed by gaze shifts in a dynamic visual prediction task. Journal of Vision, 12(11):30, 1-19, http://www.journalofvision.org/content/12/ 11/30, doi:10.1167/12.11.30. [PubMed] [Article]
23. Cyganek, B., & Gruszczynski, S. (2014). Hybrid computer vision system for drivers' eye recognition and fatigue monitoring. Neurocomputing, 126, 78- 94.
24. DeAngelus, M., & Pelz, J. B. (2009). Top-down control of eye movements: Yarbus revisited. Visual Cognition, 17, 790-811.
25. Doshi, A., & Trivedi, M. M. (2009). On the roles of eye gaze and head dynamics in predicting driver's intent to change lanes. Intelligent Transportation Systems, IEEE Transactions on, 10, 453-462.
26. Doshi, A., & Trivedi, M. M. (2012). Head and eye gaze dynamics during visual attention shifts in complex environments. Journal of Vision, 12(2):9, 1-16, http://www.journalofvision.org/content/12/2/9, doi:10.1167/12.2.9. [PubMed] [Article]
27. Duncan, J., & Humphreys, G. W. (1989). Visual search and stimulus similarity. Psychological Review, 96, 433.
28. Egeth, H. E., & Yantis, S. (1997). Visual attention: Control, representation, and time course. Annual Review of Psychology, 48, 269-297.
29. Epelboim, J., & Suppes, P. (2001). A model of eye movements and visual working memory during problem solving in geometry. Vision Research, 41, 1561-1574.
30. Fathi, A., Farhadi, A., & Rehg, J. M. (2011). Understanding egocentric activities. International Conference on Computer Vision (ICCV) (pp. 407- 414).
31. Ferguson, H. J., & Breheny, R. (2011). Eye movements reveal the time-course of anticipating behaviour based on complex, conflicting desires. Cognition, 119.
32. Fix, E., & Hodges, J. L. (1951). Discriminatory analysis, nonparametric discrimination: Consistency properties (Technical Report No. 4). Randolph Field, TX: USAF School of Aviation Medicine.
33. Folk, C. L., & Remington, R. (1998). Selectivity in distraction by irrelevant featural singletons: evidence for two forms of attentional capture. Journal of Experimental Psychology: Human Perception and Performance, 24, 847.
34. Folk, C. L., Remington, R. W., & Johnston, J. C. (1992). Involuntary covert orienting is contingent on attentional control settings. Journal of Experimental Psychology: Human Perception and Performance, 18, 1030.
35. Freund, Y., & Schapire, R. E. (1997). A decisiontheoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 1, 119-139.
36. Garcia-Diaz, A., Fdez-Vidal, X. R., Pardo, X. M., & Dosil, R. (2012). Saliency from hierarchical adaptation through decorrelation and variance normalization. Image and Vision Computing, 30, 51-64.
37. Greene, M., Liu, T., & Wolfe, J. (2012). Reconsidering Yarbus: A failure to predict observers' task from eye movement patterns. Vision Research, 62, 1-8.
38. Griffiths, A. N., Marshall, R. W., & Richens, A. (1984). Saccadic eye movement analysis as a measure of drug effects on human psychomotor performance. British Journal of Clinical Pharmacology, 18, 73S-82S.
39. Guo, C., & Zhang, L. (2010). A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans, on Image Processing, 19(1), 185-198.
40. Hagemann, N., Schorer, J., Canal-Bruland, R., Lotz,S., & Strauss, B. (2010). Visual perception in fencing: Do the eye movements of fencers represent their information pickup? Attention, Perception & Psychophysics, 72, 2204-2214.
41. Haji-Abolhassani, A., & Clark, J. J. (2013). A computational model for task inference in visual search. Journal of Vision, 13(3):29, 1-24, http:// www.journalofvision.org/content/13/3/29, doi:10. 1167/13.3.29. [PubMed] [Article]
42. Harel, J., Koch, C., & Perona, P. (2006). Graph-based visual saliency. Advances in Neural Information Processing Systems (NIPS), 19, 545-552.
43. Harel, J., Moran, C., Huth, A., Einhaeuser, W., & Koch, C. (2008). Decoding what people see from where they look: Predicting visual stimuli from scanpaths. In Attention in Cognitive Systems (pp. 15-26). Berlin: Springer.
44. Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9, 188-194.
45. Hayhoe, M. M., Shrivastava, A., Mruczek, R., & Pelz, J. B. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3(1):6, 49-63, http://www.journalofvision.org/content/3/1/6, doi: 10.1167/3.1.6. [PubMed] [Article]
46. Henderson, J., Shinkareva, S., Wang, J., Luke, S., & Olejarczyk, J. (2013). Predicting cognitive state from eye movements. Plos One, 8(5), e64937.
47. Hou, X., & Zhang, L. (2007). Saliency detection: A spectral residual approach. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1-8).
48. Hou, X., & Zhang, L. (2008). Dynamic visual attention: Searching for coding length increments. Advances in Neural Information Processing Systems (NIPS), 681-688.
49. Iqbal, S., & Bailey, B. (2004). Using eye gaze patterns to identify user tasks. Proceedings of the Grace Hopper Celebration of Women in Computing, October 6-9, 2004, Chicago, IL, USA.
50. Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 1489-1506.
51. Itti, L., & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2, 194-203.
52. Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 1254-1259.
53. Jang, Y.-M., Lee, S., Mallipeddi, R., Kwak, H.-W., & Lee, M. (2011). Recognition of human's implicit intention based on an eyeball movement pattern analysis. In B.-L. Lu, L. Zhang, & J. T. Kwok (Eds.), Neural Information Processing (pp. 138- 145). Berlin: Springer.
54. Jarodzka, H., Scheiter, K., Gerjets, P., & Van Gog, T. (2010). In the eyes of the beholder: How experts and novices interpret dynamic stimuli. Learning and Instruction, 20, 146-154.
55. Jones, W., & Klin, A. (2013). Attention to eyes is present but in decline in 2-6-month-old infants later diagnosed with autism. Nature. E-pub ahead of print, http://www.nature.com/nature/journal/vaop/ ncurrent/full/nature12715.html?WT.mc_id= TWT_NatureNeuro, doi:10.1038/nature12715.
56. Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009). Learning to predict where humans look. International Conference on Computer Vision (ICCV) (pp. 2106-2113).
57. Kaakinen, J. K., & Hyönä, J. (2010). Task effects on eye movements during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1561-1566.
58. Kanan, C., Ray, N., Bseiso, D. N. F., Hsiao, J. H., & Cottrell, G. W. (2014). Predicting an observer's task using multi-fixation pattern analysis. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications (ETRA-2014).
59. Kingstone, A. (2009). Taking a real look at social attention. Current Opinion in Neurobiology, 19(1), 52-56.
60. Klin, A., Lin, D., Gorrindo, P., Ramsay, G., & Jones, W. (2009). Two-year-olds with autism orient to non-social contingencies rather than biological motion. Nature, 459(7244), 257-261.
61. Koch, C., & Ullman, S. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry. Human Neurobiology, 4, 219-227.
62. Kosslyn, S. M. (1994). Image and brain: The resolution of the imagery debate. Cambridge, MA: MIT Press
63. Kuhn, G., & Tatler, B. W. (2005). Magic and fixation: Now you don't see it, now you do. Perception, 34, 1155-1161.
64. Kuhn, G., Tatler, B. W., Findlay, J. M., & Cole, G. G. (2008). Misdirection in magic: Implications for the relationship between eye gaze and attention. Visual Cognition, 16(2-3), 391-405.
65. Land, M. (2006). Eye movements and the control of actions in everyday life. Progress in Retinal and Eye Research, 25, 296-324.
66. Land, M. F., & Hayhoe, M. (2001). In what ways do eye movements contribute to everyday activities? Vision Research, 41, 3559-3565.
67. Land, M. F., & Lee, D. N. (1994). Where we look when we steer. Nature, 369, 742-744.
68. Land, M. F., & McLeod, P. (2000). From eye movements to actions: How batsmen hit the ball. Nature Neuroscience, 3, 1340-1345.
69. Land, M. F., & Tatler, B. W. (2001). Steering with the head: The visual strategy of a racing driver. Current Biology, 11, 1215-1220.
70. Land, M., Mennie, N., & Rusted, J. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28, 1311-1328.
71. Lethaus, F., Baumann, M. R. K., K̈oster, F., & Lemmer, K. (2013). A comparison of selected simple supervised learning algorithms to predict driver intent based on gaze data. Neurocomputing, 121, 108-130.
72. Loetscher, T., Bockisch, C., Nicholls, M., & Brugger, P. (2010). Eye position predicts what number you have in mind. Current Biology, 20, 264-265.
73. Macknik, S. L., King, M., Randi, J., Robbins, A., Teller, Thompson, J., & Martinez-Conde, S. (2008). Attention and awareness in stage magic: Turning tricks into research. Nature Reviews Neuroscience, 9, 871-879
74. Mast, F., & Kosslyn, S. (2002). Eye movements during visual mental imagery. Trends in Cognitive Sciences, 6(7), 271-272.
75. Meijering, B., van Rijn, H., Taatgen, N. A., & Verbrugge, R. (2012). What eye movements can tell about theory of mind in a strategic game. Plos One, 7(9), e45961.
76. Mennie, N., Hayhoe, M., & Sullivan, B. (2007). Lookahead fixations: Anticipatory eye movements in natural tasks. Experimental Brain Research, 179, 427-442.
77. Mills, M., Hollingworth, A., Van der Stigchel, S., Hoffman, L., & Dodd, M. D. (2011). Examining the influence of task set on eye movements and fixations. Journal of Vision, 11(8):17, 1-15, http:// www.journalofvision.org/content/11/8/17, doi:10. 1167/11.8.17. [PubMed] [Article]
78. Navalpakkam, V., & Itti, L. (2005). Modeling the influence of task on attention. Vision Research, 45, 205-231.
79. O'Connell, T., & Walther, D. (2012). Fixation patterns predict scene category. Journal of Vision, 12(9): 801, http://www.journalofvision.org/content/12/9/801, doi:10.1167/12.9.801. [Abstract]
80. Parkhurst, D., Law, K., & Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42, 107-123.
81. Peters, R., & Itti, L. (2007). Congruence between model and human attention reveals unique signatures of critical visual events. In Advances in Neural Information Processing Systems (NIPS).
82. Peters, R. J., Iyer, A., Itti, L., & Koch, C. (2005). Components of bottom-up gaze allocation in natural images. Vision Research, 45, 2397-2416.
83. Pirsiavash, H., & Ramanan, D. (2012). Detecting activities of daily living in first-person camera views. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 2847- 2854).
84. Poynter, W., Barber, M., Inman, J., & Wiggins, C. (2013). Individuals exhibit idiosyncratic eye-movement behavior profiles across tasks. Vision Research, 89, 32-38.
85. Rajashekar, J., Bovik, L., & Cormack, A. (2006). Visual search in noise: Revealing the influence of structural cues by gaze-contingent classification image analysis. Journal of Vision, 6(4):7, 379-386, http://www.journalofvision.org/content/6/4/7, doi: 10.1167/6.4.7. [PubMed] [Article]
86. Rayner, K. (1979). Eye guidance in reading: Fixation locations within words. Perception, 8, 21-30.
87. Rehder, B., & Hoffman, A. B. (2005). Eyetracking and selective attention in category learning. Cognitive Psychology, 51(1), 1-41.
88. Reichle, E. D., Rayner, K., & Pollatsek, A. (2003). The e-z reader model of eye movement control in reading: Comparisons to other models. Behavioral and Brain Sciences, 26, 445-476.
89. Renninger, L. W., Coughlan, J. M., Verghese, P., & Malik, J. (2004). An information maximization model of eye movements. In Advances in Neural Information Processing Systems (NIPS).
90. Risko, E. F., Anderson, N. C., Lanthier, S., & Kingstone, A. (2012). Curious eyes: Individual differences in personality predict eye movement behavior in scene-viewing. Cognition, 122, 86-90.
91. Salzberg, S. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1, 317-328.
92. Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 2, 197-227.
93. Scḧutz, A., Braun, D., & Gegenfurtner, K. (2011). Eye movements and perception: A selective review. Journal of Vision, 11(5):9, 1-30, http://www. journalofvision.org/content/11/5/9, doi:10.1167/11. 5.9. [PubMed] [Article]
94. Seiffert, C., Khoshgoftaar, T. M., Van Ḧulse, J., & Napolitano, A. (2010). RUSBoost: A hybrid approach to alleviating class imbalance. IEEE Transactions on Systems, Man, and Cybernetics -Part A: Systems and Humans, 40, 185-197.
95. Seo, H., & Milanfar, P. (2009). Static and space-time visual saliency detection by self-resemblance. Journal of Vision, 9(12):15, 1-27, http://www.journalofvision.org/content/9/12/15, doi:10.1167/9. 12.15. [PubMed] [Article]
96. Shaffer, J. (1995). Multiple hypothesis testing. Annual Review of Psych, 46, 561-584.
97. Sperling, G. (1960). The information available in brief visual presentations. Psychological Monographs: General and Applied, 74, 1.
98. Sperling, G., & Dosher, B. A. (1986). Strategy and optimization in human information processing. University Park, PA: Citeseer.
99. Subramanian, R., Sebe, N., Kankanhalli, M., & Chua, T. (2010). An eye fixation database for saliency detection in images. In S. Ramanathan, H. Katti, N. Sebe, M. Kankanhali, & T.-S. Chua (Eds.), Computer Vision-ECCV 2010 (pp. 30-43). Berlin: Springer.
100. Tatler, B., Hayhoe, M., Land, M., & Ballard, D. (2011). Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(5):5, 1-23, http://www.journalofvision.org/content/11/5/5, doi:10.1167/11.5.5. [PubMed] [Article]
101. Tatler, B., & Vincent, B. (2009). The prominence of behavioural biases in eye guidance. Visual Cognition, 17, 1029-1054.
102. Tatler, B., Wade, N., Kwan, H., Findlay, J., & Velichkovsky, B. (2010). Yarbus, eye movements, and vision. i-Perception, 1, 7-27.
103. Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review, 113, 766-786.
104. Treisman, A., & Gelade, G. (1980). A feature integration theory of attention. Cognitive Psychology, 12, 97-136.
105. Tseng, P., Cameron, I. G. M., Pari, G., Reynolds, J. N., Munoz, D. P., & Itti, L. (2012). High-throughput classification of clinical populations from natural viewing eye movements. Journal of Neurology, 260(1), 275-284.
106. Victor, T. W., Harbluk, J. L., & Engstr̈om, J. A. (2005). Sensitivity of eye-movement measures to in-vehicle task difficulty. Transportation Research Part F, 8, 167-190.
107. Vogt, S., & Magnussen, S. (2007). Expertise in pictorial perception: Eye-movement patterns and visual memory in artists and laymen. Perception, 36, 91- 100.
108. Windau, J., & Itti, L. (2013). Situation awareness via sensor-equipped eyeglasses. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 5674-5679). IEEE.
109. Woods, D. J., Beecher, G. P., & Ris, M. D. (1978). The effects of stressful arousal on conjugate lateral eye movement. Motivation and Emotion, 2, 345-353.
110. Yantis, S. (2000). Goal-directed and stimulus-driven determinants of attentional control. Attention and Performance, 18, 73-103.
111. Yarbus, A. L. (1967). Eye movements and vision. New York: Plenum.
112. Zelinsky, G., Peng, Y., & Samaras, D. (2013). Eye can read your mind: Decoding gaze fixations to reveal categorical search targets. Journal of Vision, 13(14): 10, 1-13, http://www.journalofvision.org/content/ 13/14/10, doi:10.1167/13.14.10. [PubMed] [Article]
113. Zelinsky, G. J. (2008). A theory of eye movements during target acquisition. Psychological Review, 115, 787-835.
114. Zelinsky, G., Zhang, W., & Samaras, D. (2008). Eye can read your mind: Decoding eye movements to reveal the targets of categorical search tasks. Journal of Vision, 8(6):380, http://www. journalofvision.org/content/8/6/380, doi:10.1167/8. 6.380. [Abstract]
115. Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008). Sun: A Bayesian framework for saliency using natural statistics. Journal of Vision, 8(7):32, 1-20, http://www.journalofvision. org/content/8/7/32, doi:10.1167/8.7.32. [PubMed] [Article]
116. Zhao, Q., & Koch, C. (2012). Learning visual saliency by combining feature maps in a nonlinear manner using adaboost. Journal of Vision, 12(6):22, 1-15, http://www.journalofvision.org/content/12/6/22, doi:10.1167/12.6.22. [PubMed] [Article]