Learning a Combined Model of Visual Saliency for Fixation Prediction

IEEE Transactions on Image Processing

Volumn 25, Issue 4, 2016, Pages 1566-1579

  Wang, Jingwei ^a   Borji, Ali ^b   Kuo, C C Jay ^a   Itti, Laurent ^a  

^a University of Southern California   (United States)

^b University of Central Florida   (United States)

Author keywords

bottom up attention; eye movements; para boosting learner; regions of interest; Saliency; scene understanding; score level fusion

Indexed keywords

IMAGE SEGMENTATION; PROBABILITY DENSITY FUNCTION; SEMANTICS;

BOTTOM-UP ATTENTION; PARA-BOOSTING LEARNER; REGIONS OF INTEREST; SALIENCY; SCENE UNDERSTANDING; SCORE-LEVEL FUSION;

EYE MOVEMENTS;

EID: 84962598208     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2016.2522380     Document Type: Article

References (93)

1. A. Borji and L. Itti, "State-of-the-art in visual attention modeling, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 185-207, Jan. 2013.
2. A. Borji, D. N. Sihite, and L. Itti, "Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study, " IEEE Trans. Image Process., vol. 22, no. 1, pp. 55-69, Jan. 2013.
3. A. Borji, H. R. Tavakoli, D. N. Sihite, and L. Itti, "Analysis of scores, datasets, and models in visual saliency prediction, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2013, pp. 921-928.
4. O. Le Meur and Z. Liu, "Saccadic model of eye movements for freeviewing condition, " Vis. Res., vol. 116, pp. 152-164, Nov. 2015.
5. Z. Liu, R. Shi, L. Shen, Y. Xue, K. N. Ngan, and Z. Zhang, "Unsupervised salient object segmentation based on kernel density estimation and two-phase graph cut, " IEEE Trans. Multimedia, vol. 14, no. 4, pp. 1275-1289, Aug. 2012.
6. R. Shi, Z. Liu, H. Du, X. Zhang, and L. Shen, "Region diversity maximization for salient object detection, " IEEE Signal Process. Lett., vol. 19, no. 4, pp. 215-218, Apr. 2012.
7. C. Guo and L. Zhang, "A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression, " IEEE Trans. Image Process., vol. 19, no. 1, pp. 185-198, Jan. 2009.
8. L. Shen, Z. Liu, and Z. Zhang, "A novel H. 264 rate control algorithm with consideration of visual attention, " J. Multimedia Tools Appl., vol. 63, no. 3, pp. 709-727, 2013.
9. D. Culibrk, M. Mirkovic, V. Zlokolica, M. Pokric, V. Crnojevic, and D. Kukolj, "Salient motion features for video quality assessment, " IEEE Trans. Image Process., vol. 20, no. 4, pp. 948-958, Apr. 2010.
10. H. Du, Z. Liu, J. Jiang, and L. Shen, "Stretchability-aware block scaling for image retargeting, " J. Vis. Commun. Image Represent., vol. 24, no. 4, pp. 499-508, 2013.
11. Z. Liu, X. Zhang, S. Luo, and O. Le Meur, "Superpixel-based spatiotemporal saliency detection, " IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 9, pp. 1522-1540, Sep. 2014.
12. Y. Luo, J. Yuan, P. Xue, and Q. Tian, "Saliency density maximization for efficient visual objects discovery, " IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 12, pp. 1822-1834, Dec. 2011.
13. A. Torralba and A. A. Efros, "Unbiased look at dataset bias, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2011, pp. 1521-1528.
14. A. Borji, D. N. Sihite, and L. Itti, "Salient object detection: A benchmark, " in Proc. Eur. Conf. Comput. Vis. (ECCV), 2012, pp. 414-429.
15. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254-1259, Nov. 1998.
16. C. Koch and S. Ullman, "Shifts in selective visual attention: Towards the underlying neural circuitry, " Human Neurobiol., vol. 4, no. 4, pp. 219-227, 1985.
17. A. M. Treisman and G. Gelade, "A feature-integration theory of attention, " Cognit. Psychol., vol. 12, no. 1, pp. 97-136, 1980.
18. J. Harel, C. Koch, and P. Perona, "Graph-based visual saliency, " in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2006, pp. 542-552.
19. T. Liu et al., "Learning to detect a salient object, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 2, pp. 353-367, Feb. 2011.
20. N. D. B. Bruce and J. K. Tsotsos, "Saliency based on information maximization, " in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2005, pp. 155-162.
21. X. Hou and L. Zhang, "Dynamic attention: Searching for coding length increments, " in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2008, pp. 681-688.
22. L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell, "SUN: A Bayesian framework for saliency using natural statistics, " J. Vis., vol. 8, no. 7, pp. 1-20, 2008.
23. L. Kong, L. Duan, W. Yang, and Y. Dou, "Salient region detection: An integration approach based on image pyramid and region property, " IET Comput. Vis., vol. 9, no. 1, pp. 85-97, 2015.
24. Y. Fang, W. Lin, Z. Chen, C.-M. Tsai, and C.-W. Lin, "A video saliency detection model in compressed domain, " IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 1, pp. 27-38, Jan. 2014.
25. H. Jiang, J. Wang, Z. Yuan, T. Liu, N. Zheng, and S. Li, "Automatic salient object segmentation based on context and shape prior, " in Proc. Brit. Mach. Vis. Conf. (BMVC), 2011, pp. 1-12.
26. A. Borji and L. Itti, "Exploiting local and global patch rarities for saliency detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2012, pp. 478-485.
27. M. Cerf, E. P. Frady, and C. Koch, "Faces and text attract gaze independent of the task: Experimental data and computer model, " J. Vis., vol. 9, no. 12, pp. 1-15, 2009.
28. T. Judd, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look, " in Proc. IEEE 12th Int. Conf. Comput. Vis. (ICCV), Sep. /Oct. 2009, pp. 795-825.
29. A. Nuthmann and J. M. Henderson, "Object-based attentional selection in scene viewing, " J. Vis., vol. 10, no. 8, pp. 2237-2242, 2010.
30. K.-Y. Chang, T.-L. Liu, H.-T. Chen, and S.-H. Lai, "Fusing generic objectness and visual saliency for salient object detection, " in Proc. Int. Conf. Comput. Vis. (ICCV), Nov. 2011, pp. 914-921.
31. B. Alexe, T. Deselaers, and V. Ferrari, "What is an object?" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 73-80.
32. G. Kootstra, A. Nederveen, and B. de Boer, "Paying attention to symmetry, " in Proc. Brit. Mach. Vis. Conf. (BMVC), 2008, pp. 1115-1125.
33. M. Pajak and A. Nuthmann, "Object-based saccadic selection during scene perception: Evidence from viewing position effects, " J. Vis., vol. 13, no. 5, pp. 1-21, 2013.
34. B. W. Tatler, "The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions, " J. Vis., vol. 7, no. 14, pp. 1-17, 2007.
35. K. A. Ehinger, B. Hidalgo-Sotelo, A. Torralba, and A. Oliva, "Modelling search for people in 900 scenes: A combined source model of eye guidance, " Vis. Cognit., vol. 17, nos. 6-7, pp. 945-978, 2009.
36. Z. Li, "A saliency map in primary visual cortex, " Trends Cognit. Sci., vol. 6, no. 1, pp. 9-16, Jan. 2002.
37. L. Itti, and C. Koch, "Feature combination strategies for saliency-based visual attention systems, " J. Electron. Imag., vol. 10, no. 1, pp. 161-169, 2001.
38. P. Viola and M. J. Jones, "Robust real-time face detection, " Int. J. Comput. Vis., vol. 57, no. 2, pp. 137-154, 2004.
39. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, "Object detection with discriminatively trained part-based models, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 9, pp. 1627-1645, Sep. 2010.
40. Q. Zhao and C. Koch, "Learning a saliency map using fixated locations in natural scenes, " J. Vis., vol. 11, no. 3, pp. 1-15, 2011.
41. A. Borji, "Boosting bottom-up and top-down visual features for saliency estimation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2012, pp. 438-445.
42. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting, " J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, Aug. 1997.
43. P. Reinagel and A. M. Zador, "Natural scene statistics at the centre of gaze, " Netw., Comput. Neural Syst., vol. 10, no. 4, pp. 341-350, 1999.
44. W. Kienzle, A. F. Wichmann, B. Schölkopf, and M. O. Franz, "A nonparametric approach to bottom-up visual saliency, " in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2006, pp. 689-696.
45. L. Zhang, Y. Xia, R. Ji, and X. Li, "Spatial-aware object-level saliency prediction by learning graphlet hierarchies, " IEEE Trans. Ind. Electron., vol. 62, no. 2, pp. 1301-1308, Feb. 2015.
46. L. Mai, Y. Niu, and F. Liu, "Saliency aggregation: A data-driven approach, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2013, pp. 1131-1138.
47. O. Le Meur and Z. Liu, "Saliency aggregation: Does unity make strength?" in Proc. 12th Asia Conf. Comput. Vis. (ACCV), 2014, pp. 18-32.
48. Y. Xie, H. Lu, and M.-H. Yang, "Bayesian saliency via low and mid level cues, " IEEE Trans. Image Process., vol. 22, no. 5, pp. 1689-1698, May 2013.
49. X. Li, H. Lu, L. Zhang, X. Ruan, and M.-H. Yang, "Saliency detection via dense and sparse reconstruction, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2013, pp. 2976-2983.
50. E. Rahtu, J. Kannala, M. Salo, and J. Heikkilä, "Segmenting salient objects from images and videos, " in Proc. 11th Eur. Conf. Comput. Vis. (ECCV), 2010, pp. 366-379.
51. V. Navalpakkam and L. Itti, "An integrated model of top-down and bottom-up attention for optimizing detection speed, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2006, pp. 2049-2056.
52. J. M. Wolfe, E. F. Risko, and A. Kingstone, Visual Search. Hove, U. K: Psychology Press, 1998, pp. 13-71.
53. A. Borji, M. N. Ahmadabadi, and B. N. Araabi, "Cost-sensitive learning of top-down modulation for attentional control, " Mach. Vis. Appl., vol. 22, no. 1, pp. 61-76, 2011.
54. X. Hou and L. Zhang, "Saliency detection: A spectral residual approach, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2007, pp. 1-8.
55. A. Jain, K. Nandakumar, and A. Ross, "Score normalization in multimodal biometric systems, " Pattern Recognit., vol. 38, no. 12, pp. 2270-2285, Dec. 2005.
56. R. Snelick, U. Uludag, A. Mink, M. Indovina, and A. K. Jain, "Largescale evaluation of multimodal biometric authentication using state-ofthe-art systems, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp. 450-455, Mar. 2005.
57. J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, "On combining classifiers, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226-239, Mar. 1998.
58. A. Hagen, "Robust speech recognition based on multi-stream processing, " Ph. D. dissertation, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, 2001.
59. J. Ming and F. J. Smith, "Speech recognition with unknown partial feature corruption-A review of the union model, " Comput. Speech Lang., vol. 17, nos. 2-3, pp. 287-305, 2003.
60. T. Chen and R. R. Rao, "Audio-visual integration in multimodal communication, " Proc. IEEE, vol. 86, no. 5, pp. 837-852, May 1998.
61. N. Poh and S. Bengio, "EER of fixed and trainable fusion classifiers: A theoretical study with application to biometric authentication tasks, " Multiple Classier Syst., vol. 17, no. 5, pp. 74-85, 2005.
62. K. Nandakumar, Y. Chen, S. C. Dass, and A. K. Jain, "Likelihood ratiobased biometric score fusion, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 2, pp. 342-347, Feb. 2008.
63. A. GarciaDiaz, X. R. FdezVidal, X. M. Pardo, and R. Dosil, "Decorrelation and distinctiveness provide with humanlike saliency, " in Advanced Concepts for Intelligent Vision Systems (ACIVS), vol. 5807. Berlin, Germany: Springer, 2009, pp. 343-354.
64. W. Zhu, S. Liang, Y. Wei, and J. Sun, "Saliency optimization from robust background detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2014, pp. 2814-2821.
65. Z. Liu, W. Zou, and O. Le Meur, "Saliency tree: A novel saliency detection framework, " IEEE Trans. Image Process., vol. 23, no. 5, pp. 1937-1952, May 2014.
66. S. Prabhakar and A. K. Jain, "Decision-level fusion in fingerprint verification, " Pattern Recognit., vol. 35, no. 4, pp. 861-874, 2002.
67. R. P. W. Duin and D. M. J. Tax, "Experiments with classifier combining rules, " in Multiple Classifier Systems (Lecture Notes in Computer Science), vol. 1857. 2000, pp. 16-29.
68. J. Li, M. D. Levine, X. An, and H. He, "Saliency detection based on frequency and spatial domain analyses, " in Proc. Brit. Mach. Vis. Conf. (BMVC), 2011, pp. 1-11.
69. F. Wang and J. Han, "Multimodal biometric authentication based on score level fusion using support vector machine, " Opto-Electron. Rev., vol. 17, no. 1, pp. 59-64, 2009.
70. P. Mehrani and O. Veksler, "Saliency segmentation based on learning and graph cut refinement, " in Proc. Brit. Mach. Vis. Conf. (BMVC), vol. 101. 2010, pp. 1-12.
71. M.-M. Cheng, G.-X. Zhang, N. J. Mitra, X. Huang, and S.-M. Hu, "Global contrast based salient region detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2011, pp. 409-416.
72. H. J. Seo and P. Milanfar, "Static and space-time visual saliency detection by self-resemblance, " J. Vis., vol. 9, no. 12, pp. 1-27, 2009.
73. S. Goferman, L. Zelnik-Manor, and A. Tal, "Context-aware saliency detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 2376-2383.
74. Y. Zhai and M. Shah, "Visual attention detection in video sequences using spatiotemporal cues, " in Proc. ACM Multimedia, 2006, pp. 815-824.
75. R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, "Frequency-tuned salient region detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2009, pp. 1597-1604.
76. D. Gao and N. Vasconcelos, "Discriminant saliency for visual recognition from cluttered scenes, " in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2005, pp. 481-488.
77. Y.-F. Ma and H.-J. Zhang, "Contrast-based image attention analysis by using fuzzy growing, " in Proc. ACM Multimedia, 2003, pp. 374-381.
78. V. Gopalakrishnan, Y. Hu, and D. Rajan, "Random walks on graphs to model saliency in images, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2009, pp. 1698-1705.
79. V. Movahedi and J. H. Elder, "Design and perceptual validation of performance measures for salient object segmentation, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2010, pp. 49-56.
80. B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman, "LabelMe: A database and web-based tool for image annotation, " Int. J. Comput. Vis., vol. 77, nos. 1-3, pp. 157-173, 2008.
81. L. Itti, "Automatic foveation for video compression using a neurobiological model of visual attention, " IEEE Trans. Image Process., vol. 20, no. 11, pp. 1304-1318, Oct. 2004.
82. W. Einhauser, M. Spain, and P. Perona, "Objects predict fixations better than early saliency, " J. Vis., vol. 8, no. 14, pp. 1-26, 2008.
83. C. Kanan and G. Cottrell, "Robust classification of objects, faces, and flowers using natural image statistics, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2010, pp. 2472-2479.
84. Y.-F. Ma, X.-S. Hua, L. Lu, and H.-J. Zhang, "A generic framework of user attention model and its application in video summarization, " IEEE Trans. Multimedia, vol. 7, no. 5, pp. 907-919, Oct. 2005.
85. L. Elazary and L. Itti, "Interesting objects are visually salient, " J. Vis., vol. 8, no. 3, pp. 1-15, 2008.
86. S. Ramanathan, H. Katti, N. Sebe, M. Kankanhalli, and T.-S. Chua, "An eye fixation database for saliency detection in images, " in Proc. Eur. Conf. Comput. Vis. (ECCV), 2010, pp. 30-43.
87. H. F. Chua, J. E. Boland, and R. E. Nisbett, "Cultural variation in eye movements during scene perception, " Proc. Nat. Acad. Sci. USA, vol. 102, no. 35, pp. 12629-12633, 2005.
88. R. Carmi and L. Itti, "Visual causes versus correlates of attentional selection in dynamic scenes, " Vis. Res., vol. 46, no. 26, pp. 4333-4345, 2006.
89. J. Richiardi, "Probabilistic models for multi-classifier biometric authentication using quality measures, " Ph. D. dissertation, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, 2007.
90. J. P. Hube, "Methods for estimating biometric score level fusion, " in Proc. IEEE 4th Int. Conf. Biometrics, Theory, Appl., Syst., Sep. 2010, pp. 1-6.
91. S. Frintrop, "VOCUS: A visual attention system for object detection and goal-directed search, " Ph. D. dissertation, Univ. Bonn, Bonn, Germany, 2005.
92. Q. Zhao and C. Koch, "Learning visual saliency by combining feature maps in a nonlinear manner using AdaBoost, " J. Vis., vol. 12, no. 6, pp. 1-15, 2012.
93. Y. Li, K. Fu, Z. Liu, and J. Yang, "Efficient saliency-model-guided visual co-saliency detection, " IEEE Signal Process. Lett., vol. 22, no. 5, pp. 588-592, May 2014.