Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study

IEEE Transactions on Image Processing

Volumn 22, Issue 1, 2013, Pages 55-69

  Borji, Ali ^a   Sihite, Dicky N ^a   Itti, Laurent ^a  

^a University of Southern California   (United States)

Author keywords

Bottom up attention; eye movement prediction; model comparison; visual attention; visual saliency

Indexed keywords

BOTTOM-UP ATTENTION; MODEL COMPARISON; MOVEMENT PREDICTION; VISUAL ATTENTION; VISUAL SALIENCY;

COMPUTER VISION; DISCRETE COSINE TRANSFORMS; EYE MOVEMENTS; VISUALIZATION;

OBJECT RECOGNITION;

AREA UNDER THE CURVE; ARTICLE; ATTENTION; BIOLOGY; COMPARATIVE STUDY; EYE MOVEMENT; FACTUAL DATABASE; HUMAN; IMAGE PROCESSING; METHODOLOGY; PHOTOSTIMULATION; PHYSIOLOGY; STATISTICAL MODEL; VIDEORECORDING;

AREA UNDER CURVE; ATTENTION; COMPUTATIONAL BIOLOGY; DATABASES, FACTUAL; EYE MOVEMENTS; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MODELS, STATISTICAL; PHOTIC STIMULATION; VIDEO RECORDING;

EID: 84871656223     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2012.2210727     Document Type: Article

References (101)

1. D. Walther and C. Koch, "Modeling attention to salient proto-objects," Neural Netw., vol. 19, no. 9, pp. 1395-1407, 2006. (Pubitemid 44708603)
2. A. Salah, E. Alpaydin, and L. Akrun, "A selective attention-based method for visual pattern recognition with application to handwritten digit recognition and face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 3, pp. 420-425, Mar. 2002. (Pubitemid 35289357)
3. C. Kanan and G. Cottrell, "Robust classification of objects, faces, and flowers using national image," in Proc. Comput. Vis. Pattern Recognit., 2010, pp. 1-8.
4. S. Frintrop, "VOCUS: A visual attention system for object detection and goal-directed search," Ph.D. thesis, Dept. Comput. Sci., Univ. Bonn, Bonn, Germany, 2006.
5. V. Navalpakkam and L. Itti, "An integrated model of top-down and bottom-up attention for optimizing detection speed," in Proc. Conf. Comput. Vis. Pattern Recognit., 2006, pp. 2049-2056. (Pubitemid 44931567)
6. S. Frintrop, "General object tracking with a component-based target descriptor," in Proc. Int. Conf. Robot. Autom., 2010, pp. 4531-4536.
7. L. Itti, "Automatic foveation for video compression using a neurobio-logical model of visual attention," IEEE Trans. Image Process., vol. 13, no. 10, pp. 1304-1318, Oct. 2004.
8. Y. Ma, X. Hua, L. Lu, and H. Zhang, "A generic framework of user attention model and its application in video summarization," IEEE Trans. Multimedia, vol. 7, no. 5, pp. 907-919, Oct. 2005. (Pubitemid 41452519)
9. D. DeCarlo and A. Santella, "Stylization and abstraction of photographs," ACM Trans. Graph., vol. 21, no. 3, pp. 769-776, 2002.
10. L. Marchesotti, C. Cifarelli, and G. Csurka, "A framework for visual saliency detection with applications to image thumbnailing," in Proc. Int. Conf. Comput. Vis., 2009, pp. 2232-2239.
11. J. Wang, J. Sun, L. Quan, X. Tang, and H. Y. Shum, "Picture collage," in Proc. Comput. Vis. Pattern Recognit., vol. 1. 2006, pp. 347-354. (Pubitemid 44931350)
12. M. S. El-Nasr, T. Vasilakos, C. Rao, and J. Zupko, "Dynamic intelligent lighting for directing visual attention in interactive 3-D scenes," IEEE Trans. Comput. Intell. AI Games, vol. 1, no. 2, pp. 145-153, Jun. 2009.
13. B. Mertsching, M. Bollmann, R. Hoischen, and S. Schmalz, "The neural active vision system," in Handbook of Computer Vision and Applications. New York: Academic, 1999, pp. 543-568.
14. A. Borji, M. N. Ahmadabadi, B. N. Araabi, and M. Hamidi, "Online learning of task-driven object-based visual attention control," Image Vis. Comput., vol. 28, no. 7, pp. 1130-1145, 2010.
15. C. Siagian and L. Itti, "Biologically inspired mobile robot vision localization," IEEE Trans. Robot., vol. 25, no. 4, pp. 861-873, Aug. 2009.
16. C. Breazeal, "A context-dependent attention system for a social robot," in Proc. Int. Joint Conf. Artif. Intell., 1999, pp. 1146-1151.
17. R. Rosenholtz, A. Dorai, and R. Freeman, "Do predictions of visual perception aid design?" ACM Trans. Appl. Percept., vol. 8, no. 2, pp. 1-27, 2011.
18. M. Ajallooeian, A. Borji, B. N. Araabi, M. N. Ahmadabadi, and H. Moradi, "Fast hand gesture recognition based on saliency maps: An application to interactive robotic marionette playing," in Proc. IEEE Conf. Robot Human Interact. Commun., Oct. 2009, pp. 841-847.
19. N. Parikh, L. Itti, and J. Weiland, "Saliency-based image processing for retinal prostheses," J. Neural Eng., vol. 7, no. 1, pp. 016006-1-016006-10, 2010.
20. The PASCAL Visual Object Classes (2005). [Online]. Available: http://pascallin.ecs.soton.ac.uk/challenges/VOC/
21. L. Itti and C. Koch, "Computational modelling of visual attention," Nature Rev. Neurosci., vol. 2, pp. 194-203, Mar. 2001. (Pubitemid 33673372)
22. D. Heinke and G. W. Humphreys, "Computational models of visual selective attention: A review," in Connectionist Models in Psychology, G. Houghton, Ed. Florence, KY: Psychology Press, 2004, pp. 273-312.
23. S. Frintrop, E. Rome, and H. I. Christensen, "Computational visual attention systems and their cognitive foundations: A survey," ACM Trans. Appl. Percept., vol. 7, no. 1, pp. 1-46, 2010.
24. A. Borji and L. Itti, "State-of-the-art in visual attention modeling," IEEE Trans. Pattern Anal. Mach. Intell., to be published.
25. D. Ballard, M. Hayhoe, and J. Pelz, "Memory representations in natural tasks," J. Cognit. Neurosci., vol. 7, no. 1, pp. 66-80, 1995.
26. M. F. Land and D. N. Lee, "Where we look when we steer," Nature, vol. 369, pp. 742-744, Jun. 1994. (Pubitemid 24225362)
27. R. J. Peters and L. Itti, "Beyond bottom-up: Incorporating task-dependent influences into a computational model of spatial attention," in Proc. Comput. Vis. Pattern Recognit., 2007, pp. 1-8.
28. M. Hayhoe and D. Ballard, "Eye movements in natural behavior," Trends Cognit. Sci., vol. 9, no. 4, pp. 188-194, 2005. (Pubitemid 40543254)
29. J. M. Henderson and A. Hollingworth, "High-level scene perception," Annu. Rev. Psychol., vol. 50, pp. 243-271, Feb. 1999.
30. W. Einhauser, M. Spain, and P. Perona, "Objects predict fixations better than early saliency," J. Vis., vol. 8, no. 14, pp. 1-26, 2008.
31. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254-1259, Nov. 1998. (Pubitemid 128742001)
32. iLab Neuromorphic Vision C++ Toolkit (iNVT). (2010, Nov. 17) [Online]. Available: http://ilab.usc.edu/toolkit/
33. L. Itti and C. Koch, "A saliency-based search mechanism for overt and covert shifts of visual attention," Vis. Res., vol. 40, nos. 10-12, pp. 1489-1506, 2000. (Pubitemid 30216462)
34. L. Itti, N. Dhavale, and F. Pighin, "Realistic avatar eye and head animation using a neurobiological model of visual attention," Proc. SPIE, vol. 5200, pp. 64-78, Dec. 2003.
35. A. Torralba, "Modeling global scene factors in attention," J. Opt. Soc. Amer., vol. 20, no. 7, pp. 1407-1418, 2003.
36. L. Itti and P. Baldi, "Bayesian surprise attracts human attention," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2006.
37. N. D. B. Bruce and J. K. Tsotsos, "Saliency based on information maximization," in Advances in Neural Information Processing Systems. New York: Academic, 2005.
38. O. Le Meur, P. Le Callet, D. Barba, and D. Thoreau, "A coherent computational approach to model bottom-up visual attention," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 5, pp. 802-817, May 2006.
39. O. Le Meur, P. Le Callet, and D. Barba, "Predicting visual fixations on video based on low-level visual features," Vis. Res., vol. 47, no. 19, pp. 2483-2493, 2007.
40. J. Harel, C. Koch, and P. Perona, "Graph-based visual saliency," in Advances in Neural Information Processing Systems, vol. 19. Cambridge, MA: MIT Press, 2006, pp. 545-552.
41. X. Hou and L. Zhang, "Saliency detection: A spectral residual approach," in Proc. Comput. Vis. Pattern Recognit., 2007, pp. 1-8.
42. M. Mancas, "Computational attention: Modelisation and application to audio and image processing," Ph.D. thesis, Faculté Polytechnique de Mons, Arrondissement of Mons, Belgium, 2007.
43. X. Hou and L. Zhang, "Dynamic visual attention: Searching for coding length increments," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2008.
44. G. Kootstra and L. R. B. Schomaker, "Prediction of human eye fixations using symmetry," in Proc. 31st Annu. Conf. Cognit. Sci. Soc., 2009.
45. L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell, "SUN: A Bayesian framework for saliency using natural statistics," J. Vis., vol. 8, no. 7, pp. 1-20, 2008.
46. S. Marat, T. Ho-Phuoc, L. Granjon, N. Guyader, D. Pellerin, and A. Guérin-Dugué, "Modelling spatio-temporal saliency to predict gaze direction for short videos," Int. J. Comput. Vis., vol. 82, no. 3, pp. 231-243, 2009.
47. C. Guo and L. Zhang, "A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression," IEEE Trans. Image Process., vol. 19, no. 1, pp. 185-198, Jan. 2010.
48. Y. Li, Y. Zhou, J. Yan, and J. Yang, "Visual saliency based on conditional entropy," in Proc. Asian Conf. Comput. Vis., 2009, pp. 246-257.
49. H. J. Seo and P. Milanfar, "Static and space-time visual saliency detection by self-resemblance," J. Vis., vol. 9, no. 12, pp. 1-27, 2009.
50. T. Judd, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look," in Proc. Int. Conf. Comput. Vis., 2009, pp. 1-7.
51. P. Bian and L. Zhang, "Biological plausibility of spectral domain approach for spatiotemporal visual saliency," in Advances in NeuroInformation Processing (Lecture Notes in Computer Science). New York: Springer-Verlag, 2009.
52. T. Avraham and M. Lindenbaum, "Esaliency (extended saliency): Meaningful attention using stochastic image modeling," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 4, pp. 693-708, Apr. 2010.
53. J. Yan, J. Liu, Y. Li, and Y. Liu, "Visual saliency detection via rank-sparsity decomposition," in Proc. Int. Conf. Image Process., 2010, pp. 1089-1092.
54. A. Garcia-Diaz, X. R. Fdez-Vidal, X. M. Pardo, and R. Dosil, "Saliency from hierarchical adaptation through decorrelation and variance normalization," Image Vis. Comput., vol. 30, no. 1, pp. 51-64, 2012.
55. J. Li, Y. Tian, T. Huang, and W. Gao, "Probabilistic multi-task learning for visual saliency estimation in video," Int. J. comput. Vis., vol. 90, no. 2, pp. 150-165, 2010.
56. H.R. Tavakoli, E. Rahtu, and J. Heikkilä, "Fast and efficient saliency detection using sparse sampling and kernel density estimation," in Proc. 17th Scandin. Conf. Image Anal., 2011, pp. 666-675.
57. N. Murray, M. Vanrell, X. Otazu, and C. A. Parraga, "Saliency estimation using a non-parametric low-level vision model," in Proc. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 433-440.
58. D. Gao, V. Mahadevan, and N. Vasconcelos, "The discriminant center-surround hypothesis for bottom-up saliency," in Proc. Neural Inf. Process. Syst., 2007, pp. 1-8.
59. A. M. Treisman and G. Gelade, "A feature integration theory of attention," Cognit. Psychol., vol. 12, no. 1, pp. 97-136, Jan. 1980.
60. J. M. Wolfe and T. S. Horowitz, "What attributes guide the deployment of visual attention and how do they do it?" Nature Rev. Neurosci., vol. 5, pp. 1-7, Jun. 2004.
61. Q. Zhao and C. Koch, "Learning a saliency map using fixated locations in natural scenes," J. Vis., vol. 11, no. 3, pp. 1-15, 2011.
62. CRCNS-Collaborative Research in Computational Neuroscience-Data Sharing (2005). [Online]. Available: http://crcns.org/data-sets/eye/eye-1
63. DIEM Dataset. [Online]. Available: http://thediemproject.wordpress.com/
64. T. Jost, N. Ouerhani, R. von Wartburg, R. Mäuri, and H. Häugli, "Assessing the contribution of color in visual attention," Comput. Vis. Image Understand., vol. 100, nos. 1-2, pp. 107-123, 2005. (Pubitemid 41329898)
65. U. Rajashekar, A. C. Bovik, and L. K. Cormack, "Visual search in noise: Revealing the influence of structural cues by gaze-contingent classification image analysis," J. Vis., vol. 6, no. 4, pp. 379-386, 2006. (Pubitemid 43731164)
66. R. Peters, A. Iyer, L. Itti, and C. Koch, "Components of bottom-up gaze allocation in natural images," Vis. Res., vol. 45, no. 18, pp. 2397-2416, 2005. (Pubitemid 40848284)
67. D. Parkhurst, K. Law, and E. Niebur, "Modeling the role of salience in the allocation of overt visual attention," Vis. Res., vol. 42, no. 1, pp. 107-123, 2002. (Pubitemid 34135927)
68. D. Green and J. Swets, Signal Detection Theory and Psychophysics. New York: Wiley, 1966.
69. M. Cerf, P. Frady, and C. Koch, "Faces and text attract gaze independent of the task: Experimental data and computer model," J. Vis., vol. 9, no. 12, pp. 1-15, 2009.
70. L. Elazary and L. Itti, "A Bayesian model for efficient visual search and recognition," Vis. Res., vol. 50, no. 14, pp. 1338-1352, 2010.
71. B. W. Tatler, "The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions," J. Vis., vol. 14, no. 7, pp. 1-17, 2007.
72. P. Tseng, R. Carmi, I. G. M. Cameron, D. P. Munoz, and L. Itti, "Quantifying center bias of observers in free viewing of dynamic natural scenes," J. Vis., vol. 9, no. 7, pp. 1-16, 2009.
73. S. A. Brandt and L. W. Stark, "Spontaneous eye movements during visual imagery reflect the content of the visual scene," J. Cognit. Neurosci., vol. 9, no. 1, pp. 27-38, 1997. (Pubitemid 27152047)
74. C. M. Privitera and L. W. Stark, "Algorithms for defining visual regions-of-interest: Comparison with eye fixations," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 9, pp. 970-982, Sep. 2000.
75. X. Hou, J. Harel, and C. Koch, "Image signature: Highlighting sparse salient regions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp. 194-201, Jan. 2012.
76. H. C. Nothdurft, "Salience of feature contrast," in Neurobiology of Attention, L. Itti, G. Rees, and J. K. Tsotsos, Eds. San Diego, CA: Elsevier, 2005.
77. S. Han and N. Vasconcelos, "Biologically plausible saliency mechanisms improve feedforward object recognition," Vis. Res., vol. 50, no. 22, pp. 2295-2307, 2010.
78. N. Mennie, M. Hayhoe, and B. Sullivan, "Look-ahead fixations: Anticipatory eye movements in natural tasks," Experim. Brain Res., vol. 179, no. 3, pp. 427-442, 2007. (Pubitemid 46716600)
79. T. Judd, "Understanding and predicting where people look," Ph.D. thesis, Dept. Electr. Eng. Comput. Sci., MIT, Cambrige, 2011.
80. A. Borji, M. N. Ahmadabadi, and B. N. Araabi, "Cost-sensitive learning of top-down modulation for attentional control," Mach. Vis. Appl., vol. 22, no. 1, pp. 61-76, 2011.
81. T. Judd, F. Durand, and A. Torralba, "Fixations on low-resolution images," J. Vis., vol. 11, no. 4, pp. 1-14, 2011.
82. S. Ramanathan, H. Katti, N. Sebe, M. Kankanhalli, and T. S. Chua, "An eye fixation database for saliency detection in images," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 30-43.
83. B. A. Olshausen and D. J. Field, "How close are we to understanding V1?" Neural Comput., vol. 17, no. 8, pp. 1665-1699, 2005. (Pubitemid 40864606)
84. E. Simoncelli and B. Olshausen, "Natural image statistics and neural representation," Annu. Rev. Neurosci., vol. 24, pp. 1193-1216, Mar. 2001. (Pubitemid 32695258)
85. Z. Li, "A saliency map in primary visual cortex," Trends Cognit. Sci., vol. 6, no. 1, pp. 9-16, 2002. (Pubitemid 34081841)
86. J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226-239, Mar. 1998. (Pubitemid 128741312)
87. B. Han and B. Zhou, "High speed visual saliency computation on GPU," in Proc. Int. Conf. Image Process., 2007, pp. 361-364.
88. T. Xu, T. Pototschnig, K. Kühnlenz, and M. Buss, "A high-speed multi-GPU implementation of bottom-up attention using CUDA," in Proc. Int. Conf. Robot. Auotm., 2009, pp. 41-47.
89. W. Wang, C. Chen, Y. Wang, T. Jiang, F. Fang, and Y. Yao, "Simulating human saccadic scanpaths on natural images," in Proc. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 441-448.
90. B. W. Tatler, M. M. Hayhoe, M. F. Land, and D. H. Ballard, "Eye guidance in natural vision: Reinterpreting salience," J. Vis., vol. 11, no. 5, pp. 1-23, May 2011.
91. C. M. Masciocchi, S. Mihalas, D. Parkhurst, and E. Niebur, "Everyone knows what is interesting: Salient locations which should be fixated," J. Vis., vol. 9, no. 11, pp. 1-22, 2009.
92. C. Koch and S. Ullman, "Shifts in selective visual attention: Toward the underlying neural circuitry," Human Neurobiol., vol. 4, no. 4, pp. 219-227, 1985. (Pubitemid 16176695)
93. D. Reisfeld, H. Wolfson, and Y. Yeshurun, "Context-free attentional operators: The generalized symmetry transform," in Proc. Int. Joint Conf. Vis., vol. 14. 1995, pp. 119-130.
94. A. Oliva, A. Torralba, M. S. Castelhano, and J. M. Henderson, "Top-down control of visual attention in object detection," in Proc. Int. Conf. Image Proces., 2003, pp. 1-4.
95. L. Zhang, M. H. Tong, and G.W. Cottrell, "SUNDAy: Saliency using natural statistics for dynamic analysis of scenes," in Proc. 31st Annu. Cognit. Sci. Soc., 2009, pp. 2944-2949.
96. W. Kienzle, M. O. Franz, B. Schölkopf, and F. A. Wichmann, "Center-surround patterns emerge as optimal predictors for human saccade targets," J. Vis., vol. 9, no. 5, pp. 1-15, 2009.
97. G. Heidemann, "Focus-of-attention from local color symmetries," IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 7, pp. 817-830, 2004.
98. B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman, "LabelMe: A database and web-based tool for image annotation," Int. J. Comput. Vis., vol. 77, nos. 1-3, pp. 157-173, 2008.
99. J. Yuen, B. C. Russell, C. Liu, and A. Torralba, "LabelMe video: Building a video database with human annotations," in Proc. Int. Conf. Comput. Vis., 2009, pp. 1-8.
100. P. Reinagel and A. Zador, "Natural scene statistics at the centre of gaze," Network, vol. 10, no. 4, pp. 341-350, 1999.
101. D. J. Simons and C. F. Chabris, "Gorillas in our midst: Sustained inattentional blindness for dynamic events," Perception, vol. 28, no. 9, pp. 1059-1074, 1999.