Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features

Neurocomputing

Volumn 244, Issue , 2017, Pages 10-18

  R Tavakoli, Hamed ^a   Borji, Ali ^b   Laaksonen, Jorma ^a   Rahtu, Esa ^c,d  

^a AALTO UNIVERSITY   (Finland)

^b University of Central Florida   (United States)

^c UNIVERSITY OF OULU   (Finland)

^d TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

Author keywords

Extreme learning machines; Fixation prediction; Inter image similarity; Saliency prediction; Visual attention

Indexed keywords

BEHAVIORAL RESEARCH; EYE MOVEMENTS; IMAGE ANALYSIS; KNOWLEDGE ACQUISITION; LEARNING SYSTEMS;

CONTEXTUAL INFORMATION; EXTREME LEARNING MACHINE; EYE MOVEMENT PATTERNS; IMAGE SIMILARITY; LOW-LEVEL VISUAL CUES; SALIENCY MODELING; VISUAL ATTENTION; VISUAL EXPERIENCES;

FORECASTING;

ARTICLE; ATTENTION; EXTREME LEARNING MACHINE; EYE MOVEMENT; IMAGE PROCESSING; IMAGE RETRIEVAL; MACHINE LEARNING; PREDICTION; VISUAL MEMORY;

EID: 85017026850     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2017.03.018     Document Type: Article

References (103)

1. [1] Tsotsos, J.K., A Computational Perspective on Visual Attention. 2011, The MIT Press.
2. [2] Frintrop, S., VOCUS: A Visual Attention System for Object Detection and Goal-directed Search, Ph.D. thesis, 2006, University of Bonn.
3. [3] Salah, A., Alpaydin, E., Akarun, L., A selective attention-based method for visual pattern recognition with application to handwritten digit recognition and face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 24:3 (2002), 420–425.
4. [4] Siagian, C., Itti, L., Rapid biologically-inspired scene classification using features shared with visual attention. IEEE Trans. Pattern Anal. Mach. Intell. 29:2 (2007), 300–312.
5. [5] Gao, D., Han, S., Vasconcelos, N., Discriminant saliency, the detection of suspicious coincidences, and applications to visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31:6 (2009), 989–1005.
6. [6] Kanan, C., Cottrell, G., Robust classification of objects, faces, and flowers using natural image statistics. Proceedings of the 2010 Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
7. [7] Papageorgiou, C., Poggio, T., A trainable system for object detection. Int. J. Comput. Vis. 38:1 (2000), 15–33.
8. [8] Bouchard, G., Triggs, B., Hierarchical part-based visual object categorization. Proceedings of the 2005 Conference on Computer Vision and Pattern Recognition (CVPR), 2005.
9. [9] Oliva, A., Torralba, A., Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42 (2001), 145–175.
10. [10] Torralba, A., Contextual priming for object detection. Int. J. Comput. Vis. 53:2 (2003), 169–191.
11. [11] Ehinger, K., Hidalgo-Sotelo, B., Torralba, A., Oliva, A., Modeling search for people in 900 scenes. Vis. Cognit. 17 (2009), 945–978.
12. [12] Fritz, G., Seifert, C., Paletta, L., Bischof, H., Attentive object detection using an information theoretic saliency measure. Proceedings of the 2005 International Workshop on Attention and Performance in Computer Vision (WAPCV), 2005.
13. [13] Kunt, M., Ikonomopoulos, A., Kocher, M., Second-generation image-coding techniques. Proc. IEEE 73:4 (1985), 549–574.
14. [14] Ouerhani, N., Bracamonte, J., Hugli, H., Ansorge, M., Pellandini, F., Adaptive color image compression based on visual attention. Proceedings of the 2001 IEEE International Conference on Image Processing (ICIP), 2001.
15. [15] Dhavale, N., Itti, L., Saliency-based multifoveated MPEG compression. Proceedings of the 2003 International Symposium on Signal Processing and its Applications (ISSPA), 2003.
16. [16] Guo, C., Zhang, L., A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans. Image Process. 19:1 (2010), 185–198.
17. [17] Mahadevan, V., Vasconcelos, N., Saliency-based discriminant tracking. Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
18. [18] Frintrop, S., General object tracking with a component-based target descriptor. Proceedings of the 2010 IEEE International Conference on Robotics and Automation (ICRA), 2010.
19. [19] Borji, A., Frintrop, S., Sihite, D., Itti, L., Adaptive object tracking by learning background context. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012.
20. [20] R.-Tavakoli, H., Shahram Moin, M., Heikkilä, J., Local similarity number and its application to object tracking. Int. J. Adv. Robot. Syst., 10(184), 2013, 10.5772/55337.
21. [21] Mishra, A., Aloimonos, Y., Fah, C.L., Active segmentation with fixation. Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
22. [22] Fu, Y., Cheng, J., Li, Z., Lu, H., Saliency cuts: An automatic approach to object segmentation. Proceedings of the 2008 International Conference on Pattern Recognition (ICPR), 2008.
23. [23] Yanulevskaya, V., Uijlings, J., Geusebroek, J.-M., Salient object detection: from pixels to segments. Image Vis. Comput. 31:1 (2013), 31–42.
24. [24] Sadaka, N.G., Karam, L.J., Efficient perceptual attentive super-resolution. Proceedings of the 2009 IEEE International Conference on Image Processing (ICIP), 2009.
25. [25] Liu, H., Jiang, S., Huang, Q., Xu, C., A generic virtual content insertion system based on visual attention analysis. Proceedings of the 2008 ACM International Conference on Multimedia (MM), 2008.
26. [26] Rosenholtz, R., Dorai, A., Freeman, R., Do predictions of visual perception aid design?. ACM Trans. Appl. Percept. 8:2 (2011), 12:1–12:20.
27. [27] Ninassi, A., Meur, O.L., Callet, P.L., Barba, D., Does where you gaze on an image affect your perception of quality? applying visual attention to image quality metric. Proceedings of the 2007 IEEE International Conference on Image Processing (ICIP), 2007.
28. [28] Ma, Q., Zhang, L., Saliency-based image quality assessment criterion. Proceedings of the 2008 International Conference on Intercultural Collaboration (ICIC), 2008.
29. [29] Mahadevan, V., Vasconcelos, N., Spatiotemporal saliency in dynamic scenes. IEEE Trans. Pattern Anal. Mach. Intell. 32:1 (2010), 171–177.
30. [30] R.-Tavakoli, H., Rahtu, E., Heikkilä, J., Saliency detection using joint temporal and spatial decorrelation. Proceedings of the 2013 Scandinavian Conference on Image Analysis (SCIA), 2013.
31. [31] R.-Tavakoli, H., Rahtu, E., Heikkil, J., Temporal saliency for fast motion detection. Proceedings of the 2013 Asian Conference on Computer Vision (ACCV), 2013.
32. [32] Mancas, M., Meur, O.L., Memorability of natural scenes: the role of attention. Proceedings of the 2013 IEEE International Conference on Image Processing (ICIP), 2013.
33. [33] Butko, N., Movellan, J.R., Optimal scanning for faster object detection. Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
34. [34] Cheng, M.-M., Zhang, Z., Lin, W.-Y., Torr, P.H.S., BING: binarized normed gradients for objectness estimation at 300fps. Proceedings of the 2014 Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
35. [35] Kasturi, R., Goldgof, D., Ekambaram, R., Sharma, R., Pratt, G., Anderson, M., Peot, M., Aguilar, M., Krotkov, E., Hackett, D., Khosla, D., Chen, Y., Kim, K., Ran, Y., Zheng, Q., Elazary, L., Voorhies, R., Parks, D., Itti, L., Performance evaluation of neuromorphic-vision object recognition algorithms. Proceedings of the 2014 International Conference on Pattern Recognition (ICPR), 2014.
36. [36] Hollingworth, A., Henderson, J.M., Accurate visual memory for previously attended objects in natural scenes. J. Exp. Psychol. Hum. Percept. Perform. 28 (2002), 113–136.
37. [37] Castelhano, M., Henderson, J., Initial scene representations facilitate eye movement guidance in visual search. J. Exp. Psychol. Hum. Percept. Perform. 33:4 (2007), 753–763.
38. [38] Kit, D., Katz, L., Sullivan, B., Snyder, K., Ballard, D., Hayhoe, M., Eye movements, visual search and scene memory, in an immersive virtual environment. PLoS One, 9(4), 2014.
39. [39] Bulling, A., Roggen, D., Recognition of visual memory recall processes using eye movement analysis. Proceedings of the 2011 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2011.
40. [40] Treisman, A.M., Gelade, G., A feature-integration theory of attention. Cognit. Psychol. 12:1 (1980), 97–136.
41. [41] Torralba, A., Oliva, A., Castelhano, M., Henderson, J., Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychol. Rev. 113:4 (2006), 766–786.
42. [42] Oliva, A., Torralba, A., The role of context in object recognition. Trends Cognit. Sci., 11(12), 2007.
43. [43] Torresani, L., Szummer, M., Fitzgibbon, A., Efficient object category recognition using classemes. Proceedings of the 2010 European Conference on Computer Vision (ECCV), 2010.
44. [44] Judd, T., Durand, F., Torralba, A., A Benchmark of Computational Models of Saliency to Predict Human Fixations. Technical Report, MIT-CSAIL-TR-2012-001, 2012, Massachusetts Institute of Technology.
45. [45] Itti, L., Koch, C., Niebur, E., A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20:11 (1998), 1254–1259.
46. [46] Choi, S.-B., Jung, B.-S., Ban, S.-W., Niitsuma, H., Lee, M., Biologically motivated vergence control system using human-like selective attention model. Neurocomputing 69 (2006), 537–558.
47. [47] Le Meur, O., Le Callet, P., Barba, D., Thoreau, D., A coherent computational approach to model bottom-up visual attention. IEEE Trans. Pattern Anal. Mach. Intell. 28:5 (2006), 802–817.
48. [48] Murray, N., Vanrell, M., Otazu, X., Parraga, C., Saliency estimation using a non-parametric low-level vision model. Proceedings of the 2011 Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
49. [49] Gao, D., Mahadevan, V., Vasconcelos, N., The discriminant center-surround hypothesis for bottom-up saliency. Proceedings of the 2007 Neural Information Processing Systems (NIPS), 2007.
50. [50] Seo, H.J., Milanfar, P., Nonparametric bottom-up saliency detection by self-resemblance. Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition CVPR, 2009.
51. [51] R.-Tavakoli, H., Rahtu, E., Heikkil, J., Fast and efficient saliency detection using sparse sampling and kernel density estimation. Proceedings of the 2011 Scandinavian Conference on Image Analysis (SCIA), 2011.
52. [52] Erdem, E., Erdem, A., Visual saliency estimation by nonlinearly integrating features using region covariances. J. Vis. 13:4 (2013), 1–20, 10.1167/13.4.11.
53. [53] Wang, Q., Yuan, Y., Yan, P., Visual saliency by selective contrast. IEEE Trans. Circuits Syst. Video Technol. 23:7 (2013), 1150–1155.
54. [54] Tavakoli, H.R., Rahtu, E., Heikkilä, J., Stochastic bottom-up fixation prediction and saccade generation. Image Vis. Comput. 31:9 (2013), 686–693.
55. [55] Bruce, N.D.B., Tsotsos, J.K., Saliency based on information maximization. Proceedings of the 2006 Neural Information Processing Systems (NIPS), 2006.
56. [56] Bruce, N.D.B., Tsotsos, J.K., Saliency, attention, and visual search: an information theoretic approach. J. Vis., 9(3), 2009.
57. [57] Mancas, M., Computational Attention: Towards Attentive Computers, 2007, CIACO University Ph.D. thesis.
58. [58] Hou, X., Zhang, L., Dynamic visual attention: searching for coding length increments. Proceedings of the 2008 Neural Information Processing Systems (NIPS), 2008.
59. [59] Li, Y., Zhou, Y., Yan, J., Niu, Z., Yang, J., Visual saliency based on conditional entropy. Proceedings of the 2010 Asian Conference on Computer Vision (ACCV), 2010.
60. [60] Li, Y., Zhou, Y., Xu, L., Yang, X., Yang, J., Incremental sparse saliency detection. Proceedings of the 2009 IEEE International Conference on Image Processing (ICIP), 2009.
61. [61] Hou, X., Zhang, L., Saliency detection: a spectral residual approach. Proceedings of the 2007 Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
62. [62] Guo, C., Ma, Q., Zhang, L., Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. Proceedings of the 2008 Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
63. [63] Bian, P., Zhang, L., Biological plausibility of spectral domain approach for spatiotemporal visual saliency. Proceedings of the 2008 International Conference on Neural Information Processing (ICONIP), 2008.
64. [64] Bian, P., Zhang, L., Visual saliency: a biologically plausible Contourlet-like frequency domain approach. Cognit. Neurodyn. 4:3 (2010), 189–198.
65. [65] Li, J., Levine, M., An, X., He, H., Saliency detection based on frequency and spatial domain analyses. Proceedings of the 2011 British Machine Vision Conference (BMVC), 2011.
66. [66] Li, J., Levine, M., An, X., Xu, X., He, H., Visual saliency based on scale-space analysis in the frequency domain. IEEE Trans. Pattern Anal. Mach. Intell. 35:4 (2013), 996–1010.
67. [67] Hou, X., Harel, J., Koch, C., Image signature: highlighting sparse salient regions. IEEE Trans. Pattern Anal. Mach. Intell., 34(1), 2012.
68. [68] Schauerte, B., Stiefelhagen, R., Predicting human gaze using quaternion DCT image signature saliency and face detection. Proceedings of the 2012 Winter Conference on Applications of Computer Vision (WACV), 2012.
69. [69] Harel, J., Koch, C., Perona, P., Graph-based visual saliency. Proceedings of the 2007 Neural Information Processing Systems (NIPS), 2007.
70. [70] Gopalakrishnan, V., Hu, Y., Rajan, D., Random walks on graphs to model saliency in images. Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
71. [71] Wang, W., Wang, Y., Huang, Q., Gao, W., Measuring visual saliency by site entropy rate. Proceedings of the 2010 Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
72. [72] Toet, A., Computational versus psychophysical bottom-up image saliency: a comparative evaluation study. IEEE Trans. Pattern Anal. Mach. Intell. 33:11 (2011), 2131–2146.
73. [73] Borji, A., Itti, L., State-of-the-art in visual attention modeling. IEEE Trans. Pattern Anal. Mach. Intell. 35:PrePrints (2013), 185–207.
74. [74] Kienzle, W., Franz, M.O., Schlkopf, B., Wichmann, F.A., Center-surround patterns emerge as optimal predictors for human saccade targets. J. Vis., 9(5), 2009.
75. [75] Judd, T., Ehinger, K., Durand, F., Torralba, A., Learning to predict where humans look. Proceedings of the 2009 IEEE International Conference on Computer Vision (ICCV), 2009.
76. [76] Wang, Q., Yuan, Y., Yan, P., Li, X., Saliency detection by multiple-instance learning. IEEE Trans. Cybern. 43:2 (2013), 660–672.
77. [77] Zhao, Q., Koch, C., Learning a saliency map using fixated locations in natural scenes. J. Vis., 11(3), 2011.
78. [78] Zhao, Q., Koch, C., Learning visual saliency by combining feature maps in a nonlinear manner using adaboost. J. Vis., 12(6), 2012.
79. [79] Vig, E., Dorr, M., Cox, D., Large-scale optimization of hierarchical features for saliency prediction in natural images. Proceedings of the 2014 Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
80. [80] Kümmerer, M., Theis, L., Bethge, M., Deep gaze I: boosting saliency prediction with feature maps trained on imagenet. Proceedings of the 2015 International Conference on Learning Representations (ICLR), 2015.
81. [81] Pan, J., McGuinness, K., Sayrol, E., O'Connor, N., Giro-i Nieto, X., Shallow and deep convolutional networks for saliency prediction. Proceedings of the 2016 Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
82. [82] Simonyan, K., Zisserman, A., Very deep convolutional networks for large-scale image recognition. Proceedings of the 2015 International Conference on Learning Representations (ICLR), 2015.
83. [83] Liu, N., Han, J., Zhang, D., Wen, S., Liu, T., Predicting eye fixations using convolutional neural networks. Proceedings of the 2015 Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
84. [84] Huang, X., Shen, C., Boix, X., Zhao, Q., Salicon: reducing the semantic gap in saliency prediction by adapting deep neural networks. Proceedings of the 2015 International Conference on Computer Vision (ICCV), 2015.
85. [85] Jetley, S., Murray, N., Vig, E., End-to-end saliency mapping via probability distribution prediction. Proceedings of the 2016 Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
86. [86] Borji, A., R.-Tavakoli, H., Sihite, D.N., Itti, L., Analysis of scores, datasets, and models in visual saliency prediction. Proceedings of the 2013 International Conference on Computer Vision (ICCV), 2013.
87. [87] Cimpoi, M., Maji, S., Vedaldi, A., Deep filter banks for texture recognition and segmentation. Proceedings of the 2015 Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
88. [88] Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Proceedings of the 2012 Workshop on Neural Information Processing Systems (NIPS), 2012.
89. [89] Oliva, A., Gist of the scene. Neurobiol. Atten. 696:64 (2005), 251–258.
90. [90] Xu, J., Jiang, M., Wang, S., Kankanhalli, M.S., Zhao, Q., Predicting human gaze beyond pixels. J. Vis 14:1 (2014), 1–20.
91. [91] Minsky, M., Papert, S., Perceptrons: An Introduction to Computational Geometry. 1969, MIT Press.
92. [92] Schmidt, W., Kraaijveld, M., Duin, R., Feedforward neural networks with random weights. Proceedings of the 1992 International Conference on Pattern Recognition (ICPR), 1992.
93. [93] Pao, Y.-H., Park, G.-H., Sobajic, D.J., Learning and generalization characteristics of the random vector functional-link net. Neurocomputing 6:2 (1994), 163–180.
94. [94] Huang, G.-B., Zhu, Q.-Y., Siew, C.-K., Extreme learning machine: a new learning scheme of feedforward neural networks. Proceedings of the 2004 International Joint Conference on Neural Networks (IJCNN), 2004.
95. [95] Huang, G.-B., Zhu, Q.-Y., Siew, C.-K., Extereme learning machine: theory and applicatons. Neurocomputing 70 (2006), 489–501.
96. [96] Huang, G.-B., Zhu, Q.-Y., Siew, C.-K., Real-time learning capability of neural netwroks. IEEE Trans. Neural Netw. 17:4 (2006), 863–878.
97. [97] Huang, G., Huang, G.-B., Song, S., You, K., Trends in extreme learning machines: a review. Neural Netw. 61 (2015), 32–48.
98. [98] Tatler, B.W., Vincent, B.T., The prominence of behavioural biases in eye guidance. Vis. Cognit. 17:6–7 (2009), 1029–1054.
99. [99] Z. Bylinskii, T. Judd, A. Borji, L. Itti, F. Durand, A. Oliva, A. Torralba, Mit saliency benchmark, 2016 http://saliency.mit.edu/.
100. [100] Borji, A., Itti, L., Cat2000: a large scale fixation dataset for boosting saliency research. Proceedings of the 2015 Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
101. [101] Garcia-Diaz, A., Fdez-Vidal, X., Pardo, X., Dosil, R., Decorrelation and distinctiveness provide with human-like saliency. Proceedings of the 2009 International Conference on Advanced Concepts for Intelligent Vision Systems (ACIVS), 2009.
102. [102] Cornia, M., Baraldi, L., Serra, G., Cucchiara, R., A deep multi-level network for saliency prediction. Proceedings of the 2016 International Conference on Pattern Recognition (ICPR), 2016.
103. [103] Zhang, J., Sclaroff, S., saliency detection: a Boolean map approach. Proceedings of the 2013 International Conference on Computer Vision (ICCV), 2013.