DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations

IEEE Transactions on Image Processing

Volumn 26, Issue 9, 2017, Pages 4446-4456

  Kruthiventi, Srinivas S S ^a   Ayush, Kumar ^b   Babu, R Venkatesh ^a  

^a INDIAN INSTITUTE OF SCIENCE   (India)

^b INDIAN INSTITUTE OF TECHNOLOGY   (India)

Author keywords

convolutional neural network; deep learning; eye fixations; Saliency prediction

Indexed keywords

BEHAVIORAL RESEARCH; CONVOLUTION; DEEP LEARNING; FORECASTING; NEURAL NETWORKS; SEMANTICS;

CONVOLUTIONAL NEURAL NETWORK; EYE FIXATIONS; HUMAN VISUAL ATTENTION; MODELING LOCATION; RECEPTIVE FIELDS; SPATIALLY INVARIANTS; STATE OF THE ART; VISUAL ATTENTION;

NETWORK LAYERS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; ATTENTION; EYE FIXATION; HUMAN; IMAGE PROCESSING; MACHINE LEARNING; PHYSIOLOGY; PROCEDURES; STATISTICAL MODEL;

ALGORITHMS; ATTENTION; FIXATION, OCULAR; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MACHINE LEARNING; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER);

EID: 85029208405     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2017.2710620     Document Type: Article

References (73)

1. C. E. Connor, H. E. Egeth, and S. Yantis, "Visual attention: Bottom-up versus top-down, " Current Biol., vol. 14, no. 19, pp. R850-R852, 2004.
2. S. Frintrop, E. Rome, and H. I. Christensen, "Computational visual attention systems and their cognitive foundations: A survey, " ACM Trans. Appl. Perception, vol. 7, no. 1, p. 6, 2010.
3. E. Awh, E. K. Vogel, and S.-H. Oh, "Interactions between attention and working memory, " Neuroscience, vol. 139, no. 1, pp. 201-208, 2006.
4. A. M. Treisman and G. Gelade, "A feature-integration theory of attention, " Cognit. Psychol., vol. 12, no. 1, pp. 97-136, Jan. 1980.
5. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254-1259, Nov. 1998.
6. H. Hadizadeh and I. V. Bajíc, "Saliency-aware video compression, " IEEE Trans. Image Process., vol. 23, no. 1, pp. 19-33, Jan. 2014.
7. D. Walther, L. Itti, M. Riesenhuber, T. Poggio, and C. Koch, "Attentional selection for object recognition-A gentle way, " in Biologically Motivated Computer Vision. Berlin, Germany: Springer, 2002, pp. 472-479.
8. K. Rapantzikos, Y. Avrithis, and S. Kollias, "Dense saliency-based spatiotemporal feature points for action recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1454-1461.
9. L.-Q. Chen, X. Xie, X. Fan, W.-Y. Ma, H.-J. Zhang, and H.-Q. Zhou, "A visual attention model for adapting images on small displays, " Multimedia Syst., vol. 9, no. 4, pp. 353-364, Oct. 2003.
10. T. Yubing, F. A. Cheikh, F. F. E. Guraya, H. Konik, and A. Trémeau, "A spatiotemporal saliency model for video surveillance, " Cognit. Comput., vol. 3, no. 1, pp. 241-263, 2011.
11. E. Niebur, "Saliency map, " Scholarpedia, vol. 2, no. 8, p. 2675, 2007.
12. J. Harel, C. Koch, and P. Perona, "Graph-based visual saliency, " in Proc. Adv. Neural Inf. Process. Syst., 2006, pp. 545-552.
13. G. Hinton et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
14. R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning, " in Proc. Int. Conf. Mach. Learn., 2008, pp. 160-167.
15. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
16. L. Bottou, "Large-scale machine learning with stochastic gradient descent, " in Proc. COMPSTAT, 2010, pp. 177-186.
17. O. Russakovsky et al., "ImageNet large scale visual recognition challenge, " Int. J. Comput. Vis., vol. 115, no. 3, pp. 211-252, Apr. 2015.
18. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, "Learning deep features for scene recognition using places database, " in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 487-495.
19. K. Simonyan and A. Zisserman. (2014). "Very deep convolutional networks for large-scale image recognition." [Online]. Available: https://arxiv.org/abs/1409.1556
20. C. Szegedy et al. (2014). "Going deeper with convolutions." [Online]. Available: https://arxiv.org/abs/1409.4842
21. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 3431-3440.
22. T. Judd, F. Durand, and A. Torralba, "A benchmark of computational models of saliency to predict human fixations, " Dept. Comput. Sci. Artif. Intell. Lab, MIT, Cambridge, MA, USA, Tech. Rep. MIT-CSAIL-TR-2012-001, 2012. [Online]. Available: http://hdl.handle.net/1721.1/68590
23. A. Borji and L. Itti. (2015). "Cat2000: A large scale fixation dataset for boosting saliency research." [Online]. Available: https://arxiv.org/abs/1505.03581
24. Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, "The secrets of salient object segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 280-287.
25. J. Xu, M. Jiang, S. Wang, M. S. Kankanhalli, and Q. Zhao, "Predicting human gaze beyond pixels, " J. Vis., vol. 14, no. 1, pp. 1-20, 2014.
26. Z. Bylinskii, P. Isola, C. Bainbridge, A. Torralba, and A. Oliva, "Intrinsic and extrinsic effects on image memorability, " Vis. Res., vol. 116, pp. 165-178, Nov. 2015.
27. M. Jiang, S. Huang, J. Duan, and Q. Zhao, "SALICON: Saliency in context, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 1072-1080.
28. C. Koch and S. Ullman, "Shifts in selective visual attention: Towards the underlying neural circuitry, " Human Neurobiol., vol. 4, no. 4, pp. 219-227, 1985.
29. R. Valenti, N. Sebe, and T. Gevers, "Image saliency by isocentric curvedness and color, " in Proc. Int. Conf. Comput. Vis., 2009, pp. 2185-2192.
30. Z. Liu, O. Le Meur, S. Luo, and L. Shen, "Saliency detection using regional histograms, " Opt. Lett., vol. 38, no. 5, pp. 700-702, 2013.
31. C. Lang, T. V. Nguyen, H. Katti, K. Yadati, M. Kankanhalli, and S. Yan, "Depth matters: Influence of depth cues on visual saliency, " in Proc. Eur. Conf. Comput. Vis., 2012, pp. 101-115.
32. E. Erdem and A. Erdem, "Visual saliency estimation by nonlinearly integrating features using region covariances, " J. Vis., vol. 13, no. 4, p. 11, Mar. 2013.
33. F. Porikli, O. Tuzel, and P. Meer, "Covariance tracking using model update based on lie algebra, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2006, pp. 728-735.
34. R. Kountchev and K. Nakamatsu, Advances in Reasoning-Based Image Processing Intelligent Systems: Conventional and Intelligent Paradigms, vol. 29. Springer, 2012. [Online]. Available: http://www.springer.com/in/book/9783642246920
35. A. Oliva, A. Torralba, M. S. Castelhano, and J. M. Henderson, "Topdown control of visual attention in object detection, " in Proc. Int. Conf. Image Process., 2003, pp. I-253-I-256.
36. N. Bruce and J. Tsotsos, "Saliency based on information maximization, " in Proc. Adv. Neural Inf. Process. Syst., 2005, pp. 1-8.
37. S. H. Khatoonabadi, N. Vasconcelos, I. V. Bajíc, and Y. Shan, "How many bits does it take for a stimulus to be salient?" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 5501-5510.
38. M. Kümmerer, L. Theis, and M. Bethge. (2014). "Deep gaze I: Boosting saliency prediction with feature maps trained on ImageNet." [Online]. Available: https://arxiv.org/abs/1411.1045
39. C. Shen and Q. Zhao, "Learning to predict eye fixations for semantic contents using multi-layer sparse network, " Neurocomputing, vol. 138, pp. 61-68, Aug. 2014.
40. E. Vig, M. Dorr, and D. Cox, "Large-scale optimization of hierarchical features for saliency prediction in natural images, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 2798-2805.
41. N. Liu, J. Han, D. Zhang, S. Wen, and T. Liu, "Predicting eye fixations using convolutional neural networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 362-370.
42. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition, " in Proc. Comput. Vis. Pattern Recognit. Workshops, 2014, pp. 512-519.
43. X. Huang, C. Shen, X. Boix, and Q. Zhao, "SALICON: Reducing the semantic gap in saliency prediction by adapting deep neural networks, " in Proc. IEEE Int. Conf. Comput. Vis., Jun. 2015, pp. 262-270.
44. R. Zhao, W. Ouyang, H. Li, and X. Wang, "Saliency detection by multi-context deep learning, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 1265-1274.
45. S. R. Srivatsa and R. V. Babu, "Salient object detection via object-ness measure, " in Proc. IEEE Int. Conf. Image Process., Sep. 2015, pp. 4481-4485.
46. C. Sheth and R. V. Babu, "Object saliency using a background prior, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Mar. 2016, pp. 1931-1935.
47. S. S. S. Kruthiventi, V. Gudisa, J. H. Dholakiya, and R. V. Babu, "Saliency unified: A deep architecture for simultaneous eye fixation prediction and salient object segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 5781-5790.
48. Deeplearning.net, accessed on Aug. 5, 2016. [Online]. Available: http://deeplearning.net/tutorial/lenet.html#maxpooling
49. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. (2014). "Semantic image segmentation with deep convolutional nets and fully connected CRFs." [Online]. Available: https://arxiv.org/abs/1412.7062
50. K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. (2014). "Return of the devil in the details: Delving deep into convolutional nets." [Online]. Available: https://arxiv.org/abs/1405.3531
51. G Li and Y. Yu. (2015). "Visual saliency based on multiscale deep features." [Online]. Available: https://arxiv.org/abs/1503.08663
52. L. Itti, "Visual salience, " Scholarpedia, vol. 2, no. 9, p. 3327, 2007.
53. N. Srivastava, G Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting, " J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929-1958, 2014.
54. B. W Tatler, "The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions, " J. Vis., vol. 7, no. 14, p. 4, 2007.
55. P.-H. Tseng, R. Carmi, I. G M. Cameron, D. P. Munoz, and L. Itti, "Quantifying center bias of observers in free viewing of dynamic natural scenes, " J. Vis., vol. 9, no. 7, p. 4, 2009.
56. Y Yang, M. Song, N. Li, J. Bu, and C. Chen, "What is the chance of happening: A new way to predict where people look, " in Proc. Eur. Conf. Comput. Vis., 2010, pp. 631-643.
57. T Judd, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look, " in Proc. IEEE Int. Conf. Comput. Vis., Sep./Oct. 2009, pp. 2106-2113.
58. Y Jia et al. (2014). "Caffe: Convolutional architecture for fast feature embedding." [Online]. Available: https://arxiv.org/abs/1408.5093
59. A. Borji, "What is a salient object? A dataset and a baseline model for salient object detection, " IEEE Trans. Image Process., vol. 24, no. 2, pp. 742-756, Feb. 2015.
60. O. Le Meur and T Baccino, "Methods for comparing scanpaths and saliency maps: Strengths and weaknesses, " Behavior Res. Methods, vol. 45, no. 1, pp. 251-266, Mar. 2013.
61. T.-Y Lin et al, "Microsoft COCO: Common objects in context, " in Proc. Eur. Conf. Comput. Vis., 2014, pp. 740-755.
62. Z. Bylinskii et al. MIT Saliency Benchmark, accessed on Aug. 5, 2016. [Online]. Available: http://saliency.mit.edu/
63. N. Riche, M. Duvinage, M. Mancas, B. Gosselin, and T. Dutoit, "Saliency and human fixations: State-of-the-art and study of com-parison metrics, " in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 1153-1160.
64. R. J. Peters and L. Itti, "Applying computational tools to predict gaze direction in interactive visual environments, " ACM Trans. Appl. Perception, vol. 5, no. 2, p. 9, 2008.
65. R. J. Peters, A. Iyer, L. Itti, and C. Koch, "Components of bottom-up gaze allocation in natural images, " Vis. Res., vol. 45, no. 8, pp. 2397-2416, 2005.
66. S. Barthelmé, H. Trukenbrod, R. Engbert, and F. Wichmann, "Modeling fixation locations using spatial point processes, " J. Vis., vol. 13, no. 12, p. 1, 2013.
67. M.-M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S.-M. Hu, "Global contrast based salient region detection, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 3, pp. 569-582, Mar. 2015.
68. J. Zhang and S. Sclaroff, "Saliency detection: A Boolean map approach, " in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 153-160.
69. J. Pan, K. McGuinness, E. Sayrol, N. O'Connor, and X. Giro-i-Nieto. (2016). "Shallow and deep convolutional networks for saliency prediction." [Online]. Available: https://arxiv.org/abs/1603.00845
70. A. Garcia-Diaz, V. Leborán, X. R. Fdez-Vidal, and X. M. Pardo, "On the relationship between optical variability, visual saliency, and eye fixations: A computational approach, " J. Vis., vol. 12, no. 6, p. 17, 2012.
71. A. Borji, H. R. Tavakoli, D. N. Sihite, and L. Itti, "Analysis of scores, datasets, and models in visual saliency prediction, " in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 921-928.
72. Q. Zhao and C. Koch, "Learning a saliency map using fixated locations in natural scenes, " J. Vis., vol. 11, no. 3, p. 9, 2011.
73. (2016). Large-Scale Scene Understanding (LSUN): Challenge Leaderboard. [Online]. Available: http://lsun.cs.princeton.edu/leaderboard/index-2016.html