Adopting abstract images for semantic scene understanding

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 4, 2016, Pages 627-638

  Zitnick, C Lawrence ^a   Vedantam, Ramakrishna ^b   Parikh, Devi ^b  

^a MICROSOFT RESEARCH   (United States)

^b State University   (United States)

Author keywords

Abstract Images; Linguistic Meaning; Memorability; Saliency; Semantic Scene Understanding

Indexed keywords

COMPUTATIONAL LINGUISTICS; LINGUISTICS; SEMANTICS;

HIGH LEVEL SEMANTICS; MEMORABILITY; SALIENCY; SCENE UNDERSTANDING; SEMANTIC IMPORTANCE; SEMANTIC INFORMATION; SEMANTIC SIMILARITY; WRITTEN DESCRIPTION;

IMAGE PROCESSING;

EID: 84963603427     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2014.2366143     Document Type: Article

References (76)

1. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254-1259, Nov. 1998.
2. C. Privitera and L. Stark, "Algorithms for defining visual regions-of-interest: Comparison with eye fixations," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 9, pp. 970-982, Sep. 2000.
3. L. Elazary and L. Itti, "Interesting objects are visually salient," J. Vis., vol. 8, no. 3, pp. 1-15, 2008.
4. S. Hwang and K. Grauman, "Learning the relative importance of objects from tagged images for retrieval and cross-modal search," Int. J. Comput. Vis., vol. 100, pp. 134-153, 2011.
5. M. Spain and P. Perona, "Measuring and predicting object importance," Int. J. Comput. Vis., vol. 91, no. 1, pp. 59-76, 2011.
6. A. Berg, T. Berg, H. Daume, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, and K. Yamaguchi, "Understanding and predicting importance in images," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 3562-3569.
7. A. Farhadi, M. Hejrati, M. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, "Every picture tells a story: Generating sentences from images," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 15-29.
8. V. Ordonez, G Kulkarni, and T. Berg, "Im2text: Describing images using 1 million captioned photographs," in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 1143-1151.
9. Y. Yang, C. Teo, H. Daume III, and Y. Aloimonos, "Corpus-guided sentence generation of natural images," in Proc. Conf. Empirical Methods Natural Lang. Process., 2011, pp. 444-454.
10. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. Berg, and T. Berg, "Baby talk: Understanding and generating simple image descriptions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1601-1608.
11. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, "Describing objects by their attributes," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 1778-1785.
12. T. Berg, A. Berg, and J. Shih, "Automatic attribute discovery and characterization from noisy web data," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 663-676.
13. A. Gupta and L. Davis, "Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers," in Proc. 10th Eur. Conf. Comput. Vis., 2008, pp. 16-29.
14. F. Heider and M. Simmel, "An experimental study of apparent behavior," The Am. J. Psychol., vol. 57, pp. 243-259,1944.
15. K. Oatley and N. Yuill, "Perception of personal and interpersonal action in a cartoon film," Brit. J. Soc. Psychol., vol. 24, no. 2, pp. 115-124, 2011.
16. T. Judd, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look," in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 2106-2113.
17. P. Isola, J. Xiao, A. Torralba, and A. Oliva, "What makes an image memorable?" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 145-152.
18. P. Isola, D. Parikh, A. Torralba, and A. Oliva, "Understanding the intrinsic memorability of images," in Proc. Adv. Neural Inform. Process. Syst., 2011, pp. 2429-2437.
19. K. Weinberger, J. Blitzer, and L. Saul, "Distance metric learning for large margin nearest neighbor classification," in Proc. Adv. Neural Inf. Process. Syst., 2006, pp. 1473-1480.
20. P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan, "Object detection with discriminatively trained part-based models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 9, pp. 1627-1645, Sep. 2010.
21. D. Parikh, and K. Grauman, "Relative attributes," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 503-510.
22. W.-H. Lin and A. Hauptmann, "Which thousand words are worth a picture? experiments on video retrieval using a thousand concepts," in Proc. IEEE Int. Conf. Multimedia Expo, 2006, pp. 41-44.
23. M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele. (2013, Dec). Translating video content to natural language descriptions in Proc. IEEE Int. Conf. Comput. Vis., pp. 433-440, [Online]. Available: https://www.d2.mpi-inf.mpg.de/nlg-for-videos-and-images
24. N. Krishnamoorthy, G Malkarnenkar, R. J. Mooney, K. Saenko, and S. Guadarrama, "Generating natural-language video descriptions using text-mined knowledge," in Proc. Workshop Vis. Natural Lang. Process., Jul. 2013, pp. 10-19.
25. P. Das, C. Xu, R. F. Doell, and J. J. Corso, "A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 2634-2641.
26. H. Yu and J. M. Siskind. (2013). Grounded language learning from video described with sentences in Proc. 51st Annu. Meet. Assoc. Comput. Linguistics, pp. 53-63 [Online]. Available: http://haona-nyu.com/wp-content/uploads/2013/05/yu13.pdf http://haona-nyu.com/research/acl2013
27. M. Grubinger, P. Clough, H. Muller, and T. Deselaers, "The IAPR tc-12 benchmark: A new evaluation resource for visual information systems," in Proc. Int. Workshop OntoImage, 2006, pp. 13-23.
28. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, "Collecting image annotations using Amazon's mechanical turk," in Proc. NAACL HLT Workshop Creating Speech Lang. Data Amazon's MT, 2010, pp. 139-147.
29. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, "Microsoft COCO: Common objects in context," CoRR, vol. abs/1405.0312, 2014.
30. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "Sun database: Large-scale scene recognition from abbey to zoo," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 3485-3492.
31. B. Russell, A. Torralba, K. Murphy, and W. Freeman, "Labelme: A database and web-based tool for image annotation," Int. J. Comput. Vis., vol. 77, pp. 151-173, 2008.
32. I. Biederman, R. Mezzanotte, and J. Rabinowitz, "Scene perception: Detecting and judging objects undergoing relational violations," Cogn. Psychol., vol. 14, no. 2, pp. 143-177,1982.
33. M. Sadeghi and A. Farhadi, "Recognition using visual phrases," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1745-1752.
34. B. Yao and L. Fei-Fei, "Modeling mutual context of object and human pose in human-object interaction activities," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 17-24.
35. W. Einhuser, M. Spain, and P. Perona. (2008). Objects predict fixations better than early saliency J. Vis., vol. 8, no. 14, p. 18 [Online]. Available: http://www.journalofvision.org/content/8/14/18. abstract
36. T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and H.-Y. Shum, "Learning to detect a salient object," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 2, pp. 353-367, Feb. 2011.
37. A. Borji and L. Itti, "State-of-the-art in visual attention modeling," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 185-207, Jan. 2013.
38. N. Turakhia and D. Parikh, "Attribute dominance: What pops out?" in Proc. Int. Conf. Comput. Vis., 2013, pp. 1225-1232.
39. S. J. Hwang and K. Grauman, "Reading between the lines: Object localization using implicit cues from image tags," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 6, pp. 1145-1158, Jun. 2012.
40. L. Standing, "Learning 10,000 pictures," Quart. J. Exp. Psychol., vol. 25, pp. 207-222,1973.
41. I. Rock and P. Englestein, "A study of memory for visual form," The Amer. J. Psychol., vol. 72, pp. 221-229, 1959.
42. T. F. Brady, T. Konkle, G. A. Alvarez, and A. Oliva, "Visual long-term memory has a massive storage capacity for object details," Proc. Nat. Acad. Sci. USA, vol. 105, pp. 14325-14329, 2008.
43. R. R. Hunt, and J. B. Worthen, Distinctiveness and Memory. New York, NY, USA: Oxford Univ. Press, 2006.
44. A. Khosla, W. A. Bainbridge, A. Torralba, and A. Oliva, "Modifying the memorability of face photographs," in Proc. Int. Conf. Comput. Vis., 2013, pp. 3200-3207.
45. A. Khosla, J. Xiao, A. Torralba, and A. Oliva, "Memorability of image regions," in Proc. Adv. Neural Inf. Process. Syst., Lake Tahoe, USA, Dec. 2012, pp. 1378-1386.
46. M. W. Howard and M. J. Kahana, "A distributed representation of temporal context," J. Math. Psychol., vol. 46, pp. 269-299, 2001.
47. G. D. A. Brown, I. Neath, and N. Chater, "A temporal ratio model of memory," Psychol. Rev., vol. 114, pp. 539-576, 2007.
48. R. M. Shiffrin and M. Steyvers, "A model for recognition memory: REM-retrieving effectively from memory," Psychonomic Bull. Rev., vol. 4, pp. 145-166, 1997.
49. J. L. McClelland, B. L. McNaughton, and R. C. O'Reilly, "Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connec-tionist models of learning and memory," Psychol. Rev., vol. 102, pp. 419-457, 1995.
50. Y. Luo and X. Tang, "Photo and video quality evaluation: Focusing on the subject," in Proc. Eur. Conf. Comput. Vis., 2008, pp. 386-399.
51. T. Leyvand, D. Cohen-Or, G. Dror, and D. Lischinski, "Data-driven enhancement of facial attractiveness," in ACM Trans. Graph., vol. 27, p. 38, 2008.
52. B. Gooch, E. Reinhard, C. Moulding, and P. Shirley, "Artistic composition for image creation," in Proc. Eurograph. Workshop Rendering, 2001, pp. 125-136.
53. L. Renjie, C. L. Wolf, and D. Cohen-Or, "Optimizing photo composition," Tech. Rep., Tel-Aviv Univ., Israel, 2010.
54. D. Cohen-Or, O. Sorkine, R. Gal, T. Leyvand, and Y.-Q. Xu, "Color harmonization," in ACM Trans. Graph., vol. 25, pp. 624-630, 2006.
55. S. Dhar, V. Ordonez, and T. L. Berg, "High level describable attributes for predicting aesthetics and interestingness," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1657-1664.
56. G. R. Taylor, A. J. Chosak, and P. C. Brewer, "OVVV: Using virtual worlds to design and evaluate surveillance systems," in Proc. IEEE Conf. Comput Vis. Pattern Recognit., 2007, pp. 1-8.
57. J. Marin, D. Vazquez, D. Geronimo, and A. Lopez, "Learning appearance in virtual scenarios for pedestrian detection," in Proc. IEEE Conf. Comput Vis. Pattern Recognit., 2007, pp. 137-144.
58. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images," in Proc. IEEE Conf. Comput Vis. Pattern Recognit., 2011, pp. 1397-1304.
59. A. Saxena, J. Driemeyer, and A. Y. Ng, "Robotic grasping of novel objects using vision," Int. J. Robotics Res., vol. 27, pp. 157-173, 2008.
60. B. Keneva, A. Torralba, and W. Freeman, "Evaluation of image features using a photorealistic virtual world," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2282-2289.
61. C. L. Zitnick and D. Parikh, "Bringing semantics into focus using visual abstraction," in Proc. IEEE Conf. Comput Vis. Pattern Recog-nit., 2013, pp. 3009-3016.
62. C. L. Zitnick, D. Parikh, and L. Vanderwende, "Learning the visual interpretation of sentences," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1681-1688.
63. S. Ullman, M. Vidal-Naquet, and E. Sali, "Visual features of intermediate complexity and their use in classification," Nature Neuro-sci., vol. 5, no. 7, pp. 682-687, 2002.
64. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey," Pattern Recognit., vol. 36, no. 1, pp. 259-275, 2003.
65. Y. Yang and D. Ramanan, "Articulated pose estimation with flexible mixtures-of-parts," in Proc. IEEE Conf. Comput. Vis. Pattern Rec-ognit., 2011, pp. 1385-1392.
66. A. Torralba, K. Murphy, and W. Freeman, "Contextual models for object detection using boosted random fields," in Proc. Adv. Neural Inf. Process. Syst., 2004.
67. A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, and S. Belongie, "Objects in context," in Proc. IEEE Int. Conf. Comput. Vis., 2007, pp. 1-8.
68. P. Tseng, R. Carmi, I. Cameron, D. Munoz, and L. Itti, "Quantifying center bias of observers in free viewing of dynamic natural scenes," J. Vis., vol. 9, no. 7, pp. 1-16, 2009.
69. A. Oliva and A. Torralba, "The role of context in object recognition," Trends Cogn. Sci., vol. 11, no. 12, pp. 520-527, 2007.
70. C. Galleguillos, A. Rabinovich, and S. Belongie, "Object categorization using co-occurrence, location and appearance," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2008, pp. 1-8.
71. C. Desai, D. Ramanan, and C. Fowlkes, "Discriminative models for multi-class object layout," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 229-236.
72. D. Hoiem, A. Efros, and M. Hebert, "Putting objects in perspective," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2006, pp. 3-15.
73. A. Gupta, A. Efros, and M. Hebert, "Blocks world revisited: Image understanding using qualitative geometry and mechanics," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 482-496.
74. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," Int. J. Comput. Vis., vol. 42, no. 3, pp. 145-175, 2001.
75. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2006, pp. 2169-2178.
76. R. A. Rensink, "Change detection," Annu. Rev. Psychol., vol. 53, no. 1, pp. 245-277, 2002.