Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

Volumn 2017-January, Issue , 2017, Pages 3539-3548

  Zhang, Mengmi ^a,b   Ma, Keng Teck ^b   Lim, Joo Hwee ^b   Zhao, Qi ^a,c   Feng, Jiashi ^a  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^b INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

^c UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; PATTERN RECOGNITION;

ADVERSARIAL NETWORKS; CURRENT FRAME; NETWORK-BASED MODELING; ON CURRENTS; PREDICTION PROBLEM; SEARCH TASKS; SPATIAL TEMPORALS; STATE-OF-THE-ART METHODS;

PROBLEM SOLVING;

EID: 85040649121     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2017.377     Document Type: Conference Paper

References (41)

1. S. O. Ba and J.-M. Odobez. Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):101-116, 2011. 2
2. A. Betancourt, P. Morerio, C. S. Regazzoni, and M. Rauterberg. The evolution of first person vision methods: A survey. IEEE Transactions on Circuits and Systems for Video Technology, 25(5):744-760, 2015. 1
3. A. Borji, D. N. Sihite, and L. Itti. Probabilistic learning of task-specific visual attention. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 470-477. IEEE, 2012. 2
4. A. Borji, H. R. Tavakoli, D. N. Sihite, and L. Itti. Analysis of scores, datasets, and models in visual saliency prediction. In IEEE ICCV, pages 921-928. IEEE, 2013. 5
5. T. Brox, C. Bregler, and J. Malik. Large displacement optical flow. In CVPR, pages 41-48. IEEE, 2009. 5
6. N. Bruce and J. Tsotsos. Saliency based on information maximization. In Advances in neural information processing systems, pages 155-162, 2005. 5, 7
7. N. D. Bruce and J. K. Tsotsos. Saliency, attention, and visual search: An information theoretic approach. Journal of vision, 9(3):5-5, 2009. 2
8. D. Christopoulos, A. Gaitatzes, and G. Papaioannou. Imagebased techniques for enhancing virtual reality environments. In 2nd International Workshop on ICT's, Arts and Cultural Heritage, 2003. 1, 2
9. W. Ding, P. Chen, H. Al-Mubaid, and M. Pomplun. A gazecontrolled interface to virtual reality applications for motorand speech-impaired users. HCI International, San Diego, CA, 2009. 1
10. P. Dollár, V. Rabaud, G. Cottrell, and S. Belongie. Behavior recognition via sparse spatio-temporal features. In 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pages 65-72. IEEE, 2005. 8
11. A. Fathi, Y. Li, and J. M. Rehg. Learning to recognize daily actions using gaze. In European Conference on Computer Vision, pages 314-327. Springer, 2012. 5
12. A. Garcia-Diaz, X. R. Fdez-Vidal, X. M. Pardo, and R. Dosil. Saliency from hierarchical adaptation through decorrelation and variance normalization. Image and Vision Computing, 30(1):51-64, 2012. 5, 7
13. J. Harel, C. Koch, and P. Perona. Graph-based visual saliency. In NIPS, pages 545-552, 2006. 2, 5, 7
14. X. Hou, J. Harel, and C. Koch. Image signature: Highlighting sparse salient regions. TPAMI, 34(1):194-201, 2012. 2
15. X. Hou, J. Harel, and C. Koch. Image signature: Highlighting sparse salient regions. IEEE transactions on pattern analysis and machine intelligence, 34(1):194-201, 2012. 5, 7
16. C.-M. Huang, S. Andrist, A. Sauppé, and B. Mutlu. Using gaze patterns to predict task intent in collaboration. Frontiers in psychology, 6, 2015. 1
17. X. Huang, C. Shen, X. Boix, and Q. Zhao. Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In IEEE ICCV, pages 262-270, 2015. 2, 5, 7
18. L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision research, 40(10):1489-1506, 2000. 5, 7
19. L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. TPAMI, (11):1254-1259, 1998. 2
20. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 4
21. C. Koch and S. Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. In Matters of intelligence, pages 115-141. Springer, 1987. 2
22. M. Kumar. Gaze-enhanced user interface design. PhD thesis, Citeseer, 2007. 1
23. M. F. Land. The coordination of rotations of the eyes, head and trunk in saccadic turns produced in natural situations. Experimental brain research, 159(2):151-160, 2004. 2
24. I. Laptev. On space-time interest points. International Journal of Computer Vision, 64(2-3):107-123, 2005. 8
25. Y. Li, A. Fathi, and J. M. Rehg. Learning to predict gaze in egocentric video. In Proceedings of the IEEE International Conference on Computer Vision, pages 3216-3223, 2013. 1, 2, 5, 7, 8
26. Y. Lin, S. Kong, D.Wang, and Y. Zhuang. Saliency detection within a deep convolutional architecture. 2
27. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440, 2015. 2, 3, 4
28. F. Multon, L. France, M.-P. Cani-Gascuel, and G. Debunne. Computer animation of human walking: a survey. The journal of visualization and computer animation, 10(1):39-54, 1999. 1, 2
29. R. Ohme, M. Matukin, and B. Pacula-Lesniak. Biometric measures for interactive advertising research. Journal of Interactive Advertising, 11(2):60-72, 2011. 1
30. N. Riche, M. Duvinage, M. Mancas, B. Gosselin, and T. Dutoit. Saliency and human fixations: state-of-the-art and study of comparison metrics. In Proceedings of the IEEE International Conference on Computer Vision, pages 1153-1160, 2013. 5
31. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, pages 568-576, 2014. 3
32. M. Stengel, S. Grogorick, M. Eisemann, E. Eisemann, and M. A. Magnor. An affordable solution for binocular eye tracking and calibration in head-mounted displays. In Proceedings of the 23rd ACM international conference on Multimedia, pages 15-24. ACM, 2015. 1
33. A. Torralba, A. Oliva, M. S. Castelhano, and J. M. Henderson. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4):766, 2006. 2
34. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. arXiv preprint arXiv:1412.0767, 2014. 8
35. A. M. Treisman and G. Gelade. A feature-integration theory of attention. Cognitive psychology, 12(1):97-136, 1980. 2, 8
36. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. arXiv preprint arXiv: 1609.02612, 2016. 2, 3, 4, 7
37. K. Yamada, Y. Sugano, T. Okabe, Y. Sato, A. Sugimoto, and K. Hiraki. Attention prediction in egocentric video using motion and visual saliency. In Pacific-Rim Symposium on Image and Video Technology, pages 277-288. Springer, 2011. 1, 2
38. K. Yun, Y. Peng, D. Samaras, G. J. Zelinsky, and T. L. Berg. Exploring the role of gaze behavior and object detection in scene understanding. Frontiers in psychology, 4:917, 2013. 1
39. R. C. Zeleznik, A. S. Forsberg, and J. P. Schulze. Look-thatthere: Exploiting gaze in virtual reality interactions. Technical report, Technical Report CS-05, 2005. 1
40. J. Zhang and S. Sclaroff. Saliency detection: A boolean map approach. In Proceedings of the IEEE International Conference on Computer Vision, pages 153-160, 2013. 2
41. L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G.W. Cottrell. Sun: A bayesian framework for saliency using natural statistics. Journal of vision, 8(7):32-32, 2008. 2, 5, 7