Can computers learn from humans to see better? Inferring scene semantics from viewers' eye movements

MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops

Volumn , Issue , 2011, Pages 33-42

  Subramanian, Ramanathan ^a   Yanulevskaya, Victoria ^a   Sebe, Nicu ^a  

^a UNIVERSITY OF TRENTO   (Italy)

Author keywords

Eye movements; Fixations and saccades; Salient entities and interactions; Scene semantics

Indexed keywords

COMPLEX SCENES; COMPUTER VISION ALGORITHMS; ENTITY-RELATIONSHIP; FACIAL EXPRESSIONS; HUMAN ACTIVITIES; LINGUISTIC DESCRIPTIONS; SCENE DESCRIPTION; SCENE OBJECT; SCENE UNDERSTANDING; SEMANTIC GAP; SEMANTIC RELATIONSHIPS; STRONG CORRELATION; VISION AND SCENE UNDERSTANDING;

COMPUTER VISION; SEMANTICS;

EYE MOVEMENTS;

EID: 84455173231     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2072298.2072305     Document Type: Conference Paper

References (37)

1. S. Avidan and A. Shamir. Seam carving for content-aware image resizing. ACM Transactions on Graphics, 26(3), 2007.
2. E. Birmingham, W. F. Bischof, and A. Kingstone. Saliency does not account for fixations to eyes within social scenes. Vision research, 49(24):2992-3000, 2009.
3. W. Einhauser, M. Spain, and P. Perona. Objects predict fixations better than early saliency. Journal of Vision, 8(14):1-26, 11 2008.
4. L. Elazary and L. Itti. Interesting objects are visually salient. Journal of Vision, 8(3):1-15, 2008.
5. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results.
6. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 99, 2009.
7. D. A. Forsyth and J. Ponce. Computer Vision: A Modern Approach. Prentice Hall, 2002.
8. A. Hanjalic and L.-Q. Xu. Affective video content representation and modeling. IEEE Transactions on Multimedia, 7(1):143-154, 2005.
9. J. Henderson. Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11):498-504, November 2003.
10. L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40:1489-1506, 2000.
11. R. Jain and P. Sinha. Content without context is meaningless. In ACM International Conference on Multimedia, 2010.
12. T. Judd, K. Ehinger, F. Durand, and A. Torralba. Learning to predict where humans look. In International Conference on Computer Vision, 2009.
13. A. Kembhavi, T. Yeh, and L. S. Davis. Why did the person cross the road (there)? scene understanding using probabilistic logic models and common sense reasoning. In European Conference on Computer Vision, pages II: 693-706, 2010.
14. G. Kuhn, B. Tatler, and G. Cole. You look where i look! effect of gaze cues on overt and covert attention in misdirection. Visual Cognition, 17(6-7):925-944, 2009.
15. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating simple image descriptions. In Computer Vision and Pattern Recognition, pages 1601-1608, 2011.
16. J.-S. Lee, Y.-M. Kuo, P.-C. Chung, and E.-L. Chen. Naked image detection based on adaptive and extensible skin color model. Pattern Recognition, 40:2261-2270, 2007.
17. A. D. Lykins, M. Meana, and G. Kambe. Detection of differential viewing patterns to erotic and non-erotic stimuli using eye-tracking methodology. Archives of sexual behavior, 35(5):569-575, 2006.
18. J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In Proceedings of the international conference on Multimedia, ACM International Conference on Multimedia, pages 83-92, 2010.
19. C. M. M. Masciocchi, S. Mihalas, D. Parkhurst, and E. Niebur. Everyone knows what is interesting: salient locations which should be fixated. Journal of vision, 9(11), 2009.
20. S. Ramanathan, H. Katti, N. Sebe, M. Kankanhalli, and T. Chua. An eye fixation database for saliency detection in images. In European Conference on Computer Vision, pages IV: 30-43, 2010.
21. R. A. Rensink, J. K. O'Regan, and J. J. Clark. To See or not to See: The Need for Attention to Perceive Changes in Scenes. Psychological Science, 8(5):368-373, 1997.
22. H. Rowley, S. Baluja, and T. Kanade. Neural network-based face detection. IIEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1):237-28, 1998.
23. B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman. Labelme: A database and web-based tool for image annotation. Technical report, Tech. Rep. MIT-CSAIL-TR-2005-056, 2005.
24. A. Santella and D. DeCarlo. Robust clustering of eye movement recordings for quantification of visual interest. In Proceedings of the symposium on Eye tracking research & applications, pages 27-34, 2004.
25. J. Shotton, J. Winn, C. Rother, and A. Criminisi. Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International Journal of Comput. Vision, 81(1):2-23, 2009.
26. M. Spain and P. Perona. Measuring and predicting object importance. International Journal of Computer Vision, 2010.
27. R. Subramanian, H. Katti, R. Huang, T.-S. Chua, and M. Kankanhalli. Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. In ACM International Conference on Multimedia, 2009.
28. A. Torralba, A. Oliva, M. S. Castelhano, and J. M. Henderson. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4):766-786, 2006.
29. R. Valenti, N. Sebe, and T. Gevers. Image saliency by isocentric curvedness and color. In International Conference on Computer Vision, 2009.
30. R. Valenti, J. Staiano, N. Sebe, and T. Gevers. Webcam-based visual gaze estimation. In International Conference on Image Analysis and Processing, 2009.
31. P. Viola and M. J. Jones. Robust real-time face detection. International Journal of Computer Vision, 57(2):137-154, May 2004.
32. Z. Wang, L. Lu, and A. C. Bovik. Foveation scalable video coding with automatic fixation selection. IEEE Transactions on Image Processing, 12(2), 2003.
33. A. Whiten. Evolutionary and developmental origins of the mindreading system. In Evolution and Development. Lawrence Erlbaum, 1997.
34. Y. Yang, M. Song, N. Li, J. Bu, and C. Chen. V: What is the chance of happening: A new way to predict where people look. In European Conference on Computer Vision, pages 631-643, 2010.
35. A. Yarbus. Eye Movements and Vision. Plenum Press, 1967.
36. Z. Zeng and Q. Ji. Knowledge based activity recognition with dynamic bayesian network. In European Conference on Computer Vision, pages VI: 532-546, 2010.
37. Q. Zhao and C. Koch. Learning a saliency map using fixated locations in natural scenes. Journal of Vision, 11(3):1-15, 2011.