What/where to look next? Modeling top-down visual attention in complex interactive environments

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Volumn 44, Issue 5, 2014, Pages 523-538

  Borji, Ali ^a   Sihite, Dicky N ^a   Itti, Laurent ^a,b  

^a University of Southern California   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

bottom up saliency; complex natural scenes; eye movement prediction; gaze prediction; interactive environments; task driven attention; top down attention; Visual attention

Indexed keywords

BAYESIAN NETWORKS; CLASSIFICATION (OF INFORMATION); COMPLEX NETWORKS; EYE MOVEMENTS; INFERENCE ENGINES; MOTION ESTIMATION; OBJECT DETECTION; PROBABILITY DISTRIBUTIONS;

BOTTOM-UP SALIENCIES; INTERACTIVE ENVIRONMENTS; MOVEMENT PREDICTION; NATURAL SCENES; TASK-DRIVEN; TOPDOWN; VISUAL ATTENTION;

BEHAVIORAL RESEARCH;

EID: 84899666829     PISSN: 21682216     EISSN: 21682232     Source Type: Journal    
DOI: 10.1109/TSMC.2013.2279715     Document Type: Article

References (89)

1. A. Borji, D. N. Sihite, and L. Itti, "Computational modeling of top-down visual attention in interactive environments," in Proc. BMVC, 2011, pp. 1-12.
2. B. J. Scholl, "Objects and attention: The state of the art," Cognition, vol. 80, nos. 1-2, pp. 1-46, 2001. (Pubitemid 32178138)
3. M. I. Posner, "Orienting of attention," Q. J. Exp. Psych, vol. 32, pp. 3-25, Feb. 1980.
4. A. M. Treisman and G. Gelade, "A feature-integration theory of attention," Cognitive Psychol., vol. 12, no. 1, pp. 97-136, 1980.
5. L. Itti and C. Koch, "Computational modeling of visual attention," Nat. Rev. Neurosci., vol. 2, no. 3, pp. 194-203, 2001.
6. J. M. Wolfe and T. S. Horowitz, "What attributes guide the deployment of visual attention and how do they do it?" Nat. Rev. Neurosci., vol. 5, no. 6, pp. 495-501, Jun. 2004. (Pubitemid 38715516)
7. A. L. Yarbus, "Eye movements during perception of complex objects," in Eye Movements and Vision, L. Riggs, Ed. New York, NY, USA: Plenum, 1967.
8. M. Land and M. Hayhoe, "In what ways do eye movements contribute to everyday activities?" Vision Res., vol. 41, nos. 25-26, pp. 3559-3565, 2001. (Pubitemid 33096274)
9. M. Hayhoe and D. Ballard, "Eye movements in natural behavior," Trends Cogn. Sci., vol. 9, no. 4, pp. 188-193. 2005.
10. C. Rothkopf, D. Ballard, and M. Hayhoe, "Task and scene context determines where you look," J. Vision, vol. 7, no. 14, pp. 16, 1-20, 2007.
11. W. Kienzle, A. F. Wichmann, B. Scholkopf, and M. O. Franz, "A nonparametric approach to bottom-up visual saliency," in Proc. NIPS, 2006, pp. 689-696.
12. T. Judd, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look," in Proc. ICCV, 2009, pp. 2106-2113.
13. E. Vig, M. Dorr, T. Martinetz, and E. Barth, "A learned saliency predictor for dynamic natural scenes," in Proc. ICANN, LNCS, no. 6354. 2010, pp. 52-61.
14. J. Li, Y. Tian, T. Huang, and W. Gao, "Probabilistic multitask learning for visual saliency estimation in video," Int. J. Comput. Vision, vol. 2, no. 90, pp. 150-165, 2010.
15. D. Pang, A. Kimura, T. Takeuchi, J. Yamato, and K. Kashino, "A stochastic model of selective visual attention with a dynamic Bayesian network," in Proc. ICME, 2008, pp. 1073-1076.
16. M. F. Land and D. N. Lee, "Where we look when we steer," Nature, vol. 369, no. 6483, pp. 742-744, 1994. (Pubitemid 24225362)
17. R. J. Peters and L. Itti, "Beyond bottom-up: Incorporating taskdependent influences into a computational model of spatial attention," in Proc. CVPR, 2007, pp. 1-8.
18. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254-1259, Nov. 1998. (Pubitemid 128742001)
19. R. J. Peters and L. Itti, "Congruence between model and human attention reveals unique signatures of critical visual events," in Proc. NIPS, 2008, pp. 1145-1152.
20. V. Navalpakkam and L. Itti, "Modeling the influence of task on attention," Vision Res., vol. 45, no. 2, pp. 205-231, 2005. (Pubitemid 39575753)
21. N. Sprague, D. H. Ballard, "Eye movements for reward maximization," in Proc. NIPS, 2003.
22. N. Sprague, D. Ballard, and A. Robinson, "Modeling embodied visual behaviors," ACM Trans. Appl. Perception, vol. 4, no. 2, pp. 1-23, 2007.
23. H. Hecht, S. Vogt, and W. Prinz, "Motor learning enhances perceptual judgment: A case for action-perception transfer," Psychol. Res., vol. 65, no. 1, pp. 3-14, 2001. (Pubitemid 33595047)
24. S. Schhtz-Bosbach and W. Prinz, "Perceptual resonance: Actioninduced modulation of perception," Trends Cogn. Sci., vol. 11, no. 8, pp. 349-355, 2007. (Pubitemid 47077912)
25. G. Rizzolatti, L. Riggio, I. Dascola, and C. Umilt, "Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention," Neuropsychologia, vol. 25, no. 1A, pp. 31-40, 1987. (Pubitemid 17032739)
26. L. Craighero, L. Fadiga, G. Rizzolatti, and C. Umilt, "Action for perception: A motor-visual attentional effect," J. Exp. Psychol. Hum. Percept. Perform., vol. 25, no. 6, pp. 1673-1692, 1999.
27. C. Siagian and L. Itti, "Rapid biologically inspired scene classification using features shared with visual attention," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 2, pp. 300-312, Feb. 2007. (Pubitemid 46420889)
28. N. Pugeault and R. Bowden, "Learning pre-attentive driving behavior from holistic visual features," in Proc. ECCV, 2010, pp. 154-167.
29. K. Ehinger, B. Hidalgo-Sotelo, A. Torralba, and A. Oliva, "Modeling search for people in 900 scenes: A combined source model of eye guidance," Visual Cogn., vol. 17, pp. 945-978, Aug. 2009.
30. S. Chikkerur, T. Serre, C. Tan, and T. Poggio, "What and where: A Bayesian inference theory of visual attention," Vision Res., vol. 50, no. 22, pp. 2233-2247, Oct. 2010.
31. D. Walther and C. Koch, "Modeling attention to salient proto-objects," Neural Netw., vol. 19, no. 9, pp. 1395-1407, 2006. (Pubitemid 44708603)
32. S. Frintrop, "VOCUS: A visual attention system for object detection and goal-directed search," Ph. D. thesis, Springer, Berlin, Germany, 2006.
33. C. Guo and L. Zhang, "A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression," IEEE Trans. Image Process., vol. 19, no. 1, pp. 185-198, Jan. 2010.
34. L. Itti, "Automatic foveation for video compression using a neurobiological model of visual attention," IEEE Trans. Image Process., vol. 13, no. 10, pp. 1304-1318, Oct. 2004.
35. M. S. El-Nasr, T. Vasilakos, C. Rao, and J. Zupko, "Dynamic intelligent lighting for directing visual attention in interactive 3-D scenes," IEEE Trans. Comp. Intell. AI Games, vol. 1, no. 2, pp. 145-153, Jun. 2009.
36. N. Parikh, L. Itti, and J. Weiland, "Saliency-based image processing for retinal prostheses," J. Neural Eng., vol. 7, no. 1, pp. 1-10, 2010.
37. C. Siagian and L. Itti, "Biologically inspired mobile robot vision localization," IEEE Trans. Robot., vol. 25, no. 4, pp. 861-873, Aug. 2009.
38. Y. Ma, X. Hua, L. Lu, and H. Zhang, "A generic framework of user attention model and its application in video summarization," IEEE Trans. Multimedia, vol. 7, no. 5, pp. 907-919, Oct. 2005. (Pubitemid 41452519)
39. B. Mertsching, M. Bollmann, R. Hoischen, and S. Schmalz, "The neural active vision system," in Handbook of Computer Vision and Applications, vol. 3, B. Jahne, H. Haussecke, and P. Geissler, Eds. New York, NY, USA: Academic Press, 1999, pp. 543-568.
40. R. Peters, A. Iyer, L. Itti, and C. Koch, "Components of bottom-up gaze allocation in natural images," Vision Res., vol. 45, no. 18, pp. 2397-2416, Aug. 2005. (Pubitemid 40848284)
41. N. D. B. Bruce and J. K. Tsotsos, "Saliency based on information maximization," in Proc. NIPS, 2005, pp. 155-162.
42. B. W. Tatler, "The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor bases and image feature distributions," J. Vision, vol. 14, no. 7, pp. 1-17, 2007.
43. X. Hou, J. Harel, and C. Koch, "Image signature: Highlighting sparse salient regions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp. 194-201, Jan. 2012.
44. D. Gao, V. Mahadevan, and N. Vasconcelos, "The discriminant centersurround hypothesis for bottom-up saliency," in Proc. NIPS, 2007, pp. 481-488.
45. O. Le Meur, P. Le Callet, and D. Barba, "Predicting visual fixations on video based on low-level visual features," Vision Res., vol. 47, no. 19, pp. 2483-2498, Sep. 2007. (Pubitemid 47376844)
46. H. J. Seo and P. Milanfar, "Static and space-time visual saliency detection by self-resemblance," J. Vision, vol. 9, no. 12, pp. 1-27, 2009.
47. A. Garcia-Diaz, X. R. Fdez-Vidal, X. M. Pardo, and R. Dosil, "Decorrelation and distinctiveness provide with human-like saliency," in Proc. ACIVS, vol. 5807. 2009, pp. 343-354.
48. P. Bian and L. Zhang, "Biological plausibility of spectral domain approach for spatiotemporal visual saliency," Lecture Notes Comput. Sci., vol. 5506, pp. 251-258, 2009.
49. C. Guo and L. Zhang, "A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression," IEEE Trans. Image Process., vol. 19, no. 1, pp. 185-198, Jan. 2010.
50. X. Hou and L. Zhang, "Dynamic visual attention: Searching for coding length increments," in Proc. NIPS, 2008, pp. 681-688.
51. M. Mancas, "Computational attention: Modelisation and application to audio and image processing," Ph. D. thesis, Eng. Faculty Mons (FPMs), Univ. Mons (UMONS), 2007.
52. L. Itti and P. Baldi, "Bayesian surprise attracts human attention," in Proc. NIPS, 2006, pp. 547-554.
53. X. Hou and L. Zhang, "Saliency detection: A spectral residual approach," in Proc. CVPR, 2007, pp. 1-8.
54. L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell, "SUN: A Bayesian framework for saliency using natural statistics," J. Vision, vols. 8/7, no. 32, pp. 1-20, 2008.
55. D. Parkhurst, K. Law, and E. Niebur, "Modeling the role of salience in the allocation of overt visual attention," Vision Res., vol. 42, no. 1, pp. 107-123, 2002. (Pubitemid 34135927)
56. J. Harel, C. Koch, and P. Perona, "Graph-based visual saliency," in Proc. NIPS, vol. 19. 2006, pp. 545-552.
57. D. Walther and C. Koch, "Modeling attention to salient proto-objects," Neural Netw., vol. 19, no. 9, pp. 1395-1407, 2006. (Pubitemid 44708603)
58. P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan, "Object detection with discriminatively trained part based models," IEEE Trans. Pattern Aanl. Mach. Intell., vol. 32, no. 9, pp. 1627-1645, Sep. 2010.
59. [Online]. Available: http://people. csail. mit. edu/torralba/shortCourse RLOC/boosting/boosting
60. M. Cerf, J. Harel, W. Einhuser, and C. Koch, "Predicting human gaze using low-level saliency combined with face detection," in Proc. NIPS, 2007, pp. 241-248.
61. E. D. Reichle, K. Rayner, and A. Pollatsek, "The E-Z reader model of eye movement control in reading: Comparisons to other models," Behavioral Brain Sci., vol. 26, no. 4, pp. 445-476, Aug. 2003. (Pubitemid 38372358)
62. T. Erez, J. Tramper, B. Smart, and S. Gielen, "A POMDP model of eye-hand coordination," in Proc. AAAI, 2011.
63. R. D. Rimey and C. M. Brown, "Control of selective perception using Bayes nets and decision theory," Int. J. Comput. Vision, vol. 12, nos. 2-3, pp. 173-207, 1994.
64. R. C. Cagli, P. Coraggio, P. Napoletano, O. Schwartz, M. Ferraro, and G. Boccignone, "Visuomotor characterization of eye movements in a drawing task," Vision Res., vol. 49, no. 8, pp. 810-818, 2009.
65. S. Ullman, "Visual routines," Cognition, vol. 18, nos. 1-3, pp. 97-159, Dec. 1984.
66. W. Yi and D. H. Ballard, "Recognizing behavior in hand-eye coordination patterns," Int. J. Humanoid Robot., vol. 6, no. 3, pp. 337-359, 2009.
67. A. Borji, D. N. Sihite, and L. Itti, "Quantitative analysis of humanmodel agreement in visual saliency modeling: A comparative study," IEEE Trans. Image Process., vol. 22, no. 1, pp. 55-69, Jan. 2013.
68. A. Borji and L. Itti, "State-of-the-art in visual attention modeling," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 185-207, Jan. 2013.
69. C. Koch and S. Ullman, "Shifts in selective visual attention: Towards the underlying neural circuitry," Human Neurobiol., vol. 4, no. 4, pp. 219-227, 1985. (Pubitemid 16176695)
70. [Online]. Available: http://code. google. com/p/bnt/
71. [Online]. Available: http://www.csie. ntu. edu. tw/?cjlin/libsvm/
72. J. Najemnik and W. S. Geisler, "Optimal eye movement strategies in visual search," Nature, vol. 434, no. 7031, pp. 387-391, Mar. 2005. (Pubitemid 40469182)
73. G. Kootstra, A. Nederveen, and B. de Boer, "Paying attention to symmetry," in Proc. BMVC, 2008, pp. 1115-1125.
74. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," Int. J. Comput. Vision, vol. 42, no. 3, pp. 145-175, 2001. (Pubitemid 32680801)
75. M. Fukuchi, N. Tsuchiya, and C. Koch, "The focus of expansion in optical flow fields acts as a strong cue for visual attention," J. Vision, vol. 9, no. 8, p. 137, 2010.
76. T. Avraham and M. Lindenbaum, "Esaliency (extended saliency): Meaningful attention using stochastic image modeling," IEEE Trans. Pattern Aanl. Mach. Intell., vol. 32, no. 4, pp. 693-708, Apr. 2010.
77. J. M. Henderson, "Human gaze control during real-world scene perception," Trends Cogn. Sci., vol. 7, no. 11, pp. 498-504, Nov. 2003. (Pubitemid 37324734)
78. N. J. Butko and J. R. Movellan, "Optimal scanning for faster object detection," in Proc. CVPR, 2009, pp. 2751-2758.
79. A. Torralba, "Modeling global scene factors in attention," J. Opt. Soc. Amer., vol. 20, no. 7, pp. 1407-1418, 2003.
80. C. Siagian and L. Itti, "Comparison of gist models in rapid scene categorization tasks," in Proc. VSS08, p. 734.
81. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting," J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, Aug. 1997. (Pubitemid 127433398)
82. [Online]. Available: http://cocosci. berkeley. edu/tom/
83. [Online]. Available: http://web. mit. edu/cocosci/josh. html
84. J. Wang, C. Xu, and E. Chng, "Automatic sports video genre classification using pseudo-2D-HMM," in Proc. ICPR, 2006, pp. 1051-4651.
85. J. H. Lai and S. Y. Chien, "Baseball and tennis video annotation with temporal structure decomposition," in Proc. MMSP, 2008, pp. 929-936.
86. J. Hays and A. A. Efros, "Im2Gps: Estimating geographic information from a single image," in Proc. CVPR, 2008, pp. 1-8.
87. J. Lim and Y. Liu, "Y. reinforcement learning in eye movements: Modeling the influences of cyclic top-down and bottom-up processes," IEEE Trans. Syst., Man, Cybern., vol. 39, no. 4, pp. 706-714, Jul. 2009.
88. A. Borji, D. N. Sihite, and L. Itti, "An object-based Bayesian framework for top-down visual attention," in Proc. 26th Conf. AAAI, 2012.
89. A. Borji, D. N. Sihite, and L. Itti, "Probabilistic learning of task-specific visual attention," in Proc. IEEE CVPR, 2012, pp. 470-477.