Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

International Journal of Computer Vision

Volumn 123, Issue 1, 2017, Pages 32-73

  Krishna, Ranjay ^a   Zhu, Yuke ^a   Groth, Oliver ^b   Johnson, Justin ^a   Hata, Kenji ^a   Kravitz, Joshua ^a   Chen, Stephanie ^a   Kalantidis, Yannis ^c   Li, Li Jia ^d   Shamma, David A ^e   Bernstein, Michael S ^a   Fei Fei, Li ^a  

^a STANFORD UNIVERSITY   (United States)

^b DRESDEN UNIVERSITY OF TECHNOLOGY   (Germany)

^c YAHOO INC   (United States)

^d SNAP INC   (United States)

^e CWI   (Netherlands)

Author keywords

Attributes; Computer vision; Crowdsourcing; Dataset; Image; Knowledge; Language; Objects; Question answering; Relationships; Scene graph

Indexed keywords

COGNITIVE SYSTEMS; COMPUTER VISION; CROWDSOURCING; GENES; VEHICLES;

ATTRIBUTES; DATASET; IMAGE; KNOWLEDGE; LANGUAGE; OBJECTS; QUESTION ANSWERING; RELATIONSHIPS; SCENE GRAPH;

VISUAL LANGUAGES;

EID: 85011596790     PISSN: 09205691     EISSN: 15731405     Source Type: Journal    
DOI: 10.1007/s11263-016-0981-7     Document Type: Article

References (112)

1. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., et al. (2015). VQA: Visual question answering. In International conference on computer vision (ICCV).
2. Antol, S., Zitnick, C. L., & Parikh, D. (2014). Zero-shot learning via visual abstraction. In European conference on computer vision (pp. 401–416). Springer.
3. Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The Berkeley framenet project. In Proceedings of the 36th annual meeting of the association for computational linguistics and 17th international conference on computational linguistics—Volume 1, ACL’98 (pp. 86–90). Stroudsburg, PA: Association for Computational Linguistics.
4. Betteridge, J., Carlson, A., Hong, S. A., Hruschka, E. R, Jr., Law, E. L., Mitchell, T. M., et al. (2009). Toward never ending language learning. In AAAI spring symposium: Learning by reading and learning to read (pp. 1–2).
5. Bird, S. (2006). NLTK: The natural language toolkit. In Proceedings of the COLING/ACL on interactive presentation sessions (pp. 69–72). Association for Computational Linguistics.
6. Bruner, J. (1990). Culture and human development: A new look. Human Development, 33(6), 344–355.
7. Bunescu, R. C., & Mooney, R. J. (2005). A shortest path dependency kernel for relation extraction. In Proceedings of the conference on human language technology and empirical methods in natural language processing (pp. 724–731). Association for Computational Linguistics.
8. Chang, A. X., Savva, M., & Manning, C. D. (2014). Semantic parsing for text to 3D scene generation. In ACL 2014 (p. 17).
9. Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., et al. (2015). Microsoft COCO captions: Data collection and evaluation server. arXiv:1504.00325.
10. Chen, X., & Lawrence Zitnick, C. (2015). Mind’s eye: A recurrent visual representation for image caption generation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2422–2431).
11. Chen, X., Liu, Z., & Sun, M. (2014). A unified model for word sense representation and disambiguation. In EMNLP (pp. 1025–1035). Citeseer.
12. Chen, X., Shrivastava, A., & Gupta, A. (2013). Neil: Extracting visual knowledge from web data. In 2013 IEEE international conference on computer vision (ICCV) (pp. 1409–1416). IEEE.
13. Choi, W., Chao, Y.-W., Pantofaru, C., & Savarese, S. (2013). Understanding indoor scenes using 3D geometric phrases. In 2013 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 33–40). IEEE.
14. Culotta, A., & Sorensen, J. (2004). Dependency tree kernels for relation extraction. In Proceedings of the 42nd annual meeting on association for computational linguistics (p. 423). Association for Computational Linguistics.
15. Dauphin, Y., de Vries, H., & Bengio, Y. (2015). Equilibrated adaptive learning rates for non-convex optimization. In Advances in neural information processing systems (pp. 1504–1512).
16. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, 2009 (CVPR 2009) (pp. 248–255). IEEE.
17. Denkowski, M., & Lavie, A. (2014). Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation. Citeseer.
18. Dollar, P., Wojek, C., Schiele, B., & Perona, P. (2012). Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 743–761.
19. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., et al. (2015). Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2625–2634).
20. Everingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303–338.
21. Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., et al. (2015). From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1473–1482).
22. Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009). Describing objects by their attributes. In IEEE conference on computer vision and pattern recognition, 2009 (CVPR 2009) (pp. 1778–1785). IEEE.
23. Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., et al. (2010). Every picture tells a story: Generating sentences from images. In Computer vision–ECCV 2010 (pp. 15–29). Springer.
24. Fei-Fei, L., Fergus, R., & Perona, P. (2007). Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1), 59–70.
25. Ferrari, V., & Zisserman, A. (2007). Learning visual attributes. In Advances in neural information processing systems (pp. 433–440).
26. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., et al. (2010). Building watson: An overview of the deepqa project. AI Magazine, 31(3), 59–79.
27. Firestone, C., & Scholl, B. J. (2015). Cognition does not affect perception: Evaluating the evidence for top-down effects. Behavioral and brain sciences (pp. 1–72).
28. Forbus, K. D. (1984). Qualitative process theory. Artificial Intelligence, 24(1), 85–168.
29. Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015). Are you talking to a machine? Dataset and methods for multilingual image question. In Advances in neural information processing systems (pp. 2296–2304).
30. Geman, D., Geman, S., Hallonquist, N., & Younes, L. (2015). Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12), 3618–3623.
31. Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE international conference on computer vision (pp. 1440–1448).
32. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 580–587). IEEE.
33. Goering, C., Rodner, E., Freytag, A., & Denzler, J. (2014). Nonparametric part transfer for fine-grained recognition. In 2014 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2489–2496). IEEE.
34. Griffin, G., Holub, A., & Perona, P. (2007). Caltech-256 object category dataset. Technical Report 7694.
35. GuoDong, Z., Jian, S., Jie, Z., & Min, Z. (2005). Exploring various knowledge in relation extraction. In Proceedings of the 43rd annual meeting on association for computational linguistics (pp. 427–434). Association for Computational Linguistics.
36. Gupta, A., & Davis, L. S. (2008). Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In Computer vision–ECCV 2008 (pp. 16–29). Springer.
37. Gupta, A., Kembhavi, A., & Davis, L. S. (2009). Observing human–object interactions: Using spatial and functional compatibility for recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10), 1775–1789.
38. Hayes, P. J. (1978). The naive physics manifesto. Geneva: Institut pour les études sémantiques et cognitives/Université de Genève.
39. Hayes, P. J. (1985). The second naive physics manifesto. Theories of the commonsense world (pp. 1–36).
40. Hearst, M. A., Dumais, S. T., Osman, E., Platt, J., & Scholkopf, B. (1998). Support vector machines. IEEE Intelligent Systems and their Applications, 13(4), 18–28.
41. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
42. Hodosh, M., Young, P., & Hockenmaier, J. (2013). Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47(1), 853–899.
43. Hou, C.-S. J., Noy, N. F., & Musen, M. A. (2002). A template-based approach toward acquisition of logical sentences. In Intelligent information processing (pp. 77–89). Springer.
44. Huang, G. B., Mattar, M., Berg, T., & Learned-Miller, E. (2008). Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in ’real-life’ images: Detection, alignment, and recognition.
45. Isola, P., Lim, J. J., & Adelson, E. H. (2015). Discovering states and transformations in image collections. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1383–1391).
46. Izadinia, H., Sadeghi, F., & Farhadi, A. (2014). Incorporating scene context and object layout into appearance modeling. In 2014 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 232–239). IEEE.
47. Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M., et al. (2015). Image retrieval using scene graphs. In IEEE conference on computer vision and pattern recognition (CVPR).
48. Karpathy, A., & Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3128–3137).
49. Kiros, R., Salakhutdinov, R., & Zemel, R. (2014). Multimodal neural language models. In Proceedings of the 31st international conference on machine learning (ICML-14) (pp. 595–603).
50. Krishna, R., Hata, K., Chen, S., Kravitz, J., Shamma, D. A., Fei-Fei, L., et al. (2016). Embracing error to enable rapid crowdsourcing. In CHI’16-SIGCHI conference on human factors in computing system.
51. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).
52. Lampert, C. H., Nickisch, H., & Harmeling, S. (2009). Learning to detect unseen object classes by between-class attribute transfer. In IEEE conference on computer vision and pattern recognition, 2009 (CVPR 2009) (pp. 951–958). IEEE.
53. Leacock, C., Miller, G. A., & Chodorow, M. (1998). Using corpus statistics and wordnet relations for sense identification. Computational Linguistics, 24(1), 147–165.
54. Lebret, R., Pinheiro, P. O., & Collobert, R. (2015). Phrase-based image captioning. arXiv:1502.03671.
55. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). Microsoft COCO: Common objects in context. In Computer vision–ECCV 2014 (pp. 740–755). Springer.
56. Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016). Visual relationship detection using language priors. In European conference on computer vision (ECCV). IEEE.
57. Ma, L., Lu, Z., & Li, H. (2015). Learning to answer questions from image using convolutional neural network. arXiv:1506.00333.
58. Malinowski, M., & Fritz, M. (2014). A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in neural information processing systems (pp. 1682–1690).
59. Malinowski, M., Rohrbach, M., & Fritz, M. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE international conference on computer vision (pp. 1–9).
60. Malisiewicz, T., Efros, A., et al. (2008). Recognition by association via learning per-exemplar distances. In IEEE conference on computer vision and pattern recognition, 2008 (CVPR 2008) (pp. 1–8). IEEE.
61. Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations (pp. 55–60).
62. Mao, J., Xu, W., Yang, Y., Wang, J., & Yuille, A. L. (2014). Explain images with multimodal recurrent neural networks. arXiv:1410.1090.
63. Mihalcea, R., Chklovski, T. A., & Kilgarriff, A. (2004). The senseval-3 English lexical sample task. Association for Computational Linguistics, UNT Digital Library.
64. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781.
65. Miller, G. A. (1995). Wordnet: a lexical database for english. Communications of the ACM, 38(11), 39–41.
66. Niu, F., Zhang, C., Ré, C., & Shavlik, J. (2012). Elementary: Large-scale knowledge-base construction via machine learning and statistical inference. International Journal on Semantic Web and Information Systems (IJSWIS), 8(3), 42–73.
67. Ordonez, V., Kulkarni, G., & Berg, T. L. (2011). Im2text: Describing images using 1 million captioned photographs. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, & K. Weinberger (Eds.), Advances in neural information processing systems (Vol. 24, pp. 1143–1151). Red Hook: Curran Associates, Inc.
68. Pal, A. R., & Saha, D. (2015). Word sense disambiguation: A survey. arXiv:1508.01346.
69. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (pp. 311–318). Association for Computational Linguistics.
70. Patterson, G., Xu, C., Su, H., & Hays, J. (2014). The sun attribute database: Beyond categories for deeper scene understanding. International Journal of Computer Vision, 108(1–2), 59–81.
71. Perronnin, F., Sánchez, J., & Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In Computer vision–ECCV 2010 (pp. 143–156). Springer.
72. Prest, A., Schmid, C., & Ferrari, V. (2012). Weakly supervised learning of interactions between humans and objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(3), 601–614.
73. Ramanathan, V., Li, C., Deng, J., Han, W., Li, Z., Gu, K., et al. (2015). Learning semantic relationships for better action retrieval in images. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1100–1109).
74. Ren, M., Kiros, R., & Zemel, R. (2015a). Image question answering: A visual semantic embedding model and a new dataset. arXiv:1505.02074.
75. Ren, S., He, K., Girshick, R., & Sun, J. (2015b). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91–99).
76. Ronchi, M. R., & Perona, P. (2015). Describing common human visual actions in images. In X. Xie, M. W. Jones, & G. K. L. Tam (Eds.), Proceedings of the British machine vision conference (BMVC 2015) (pp. 52.1–52.12). BMVA Press.
77. Rothe, S., & Schütze, H. (2015). Autoextend: Extending word embeddings to embeddings for synsets and lexemes. arXiv:1507.01127.
78. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al. (2015). ImageNet large scale visual recognition challenge. International journal of computer vision (IJCV) (pp. 1–42).
79. Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173.
80. Sadeghi, F., Divvala, S. K., & Farhadi, A. (2015). Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1456–1464).
81. Sadeghi, M. A., & Farhadi, A. (2011). Recognition using visual phrases. In 2011 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1745–1752). IEEE.
82. Salehi, N., Irani, L. C., & Bernstein, M. S. (2015). We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd annual ACM conference on human factors in computing systems (pp. 1621–1630). ACM.
83. Schank, R. C., & Abelson, R. P. (2013). Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Hove: Psychology Press.
84. Schuler, K. K. (2005). VerbNet: A broad-coverage, comprehensive verb lexicon. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA, USA (AAI3179808).
85. Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., & Manning, C. D. (2015). Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language (pp. 70–80). Citeseer.
86. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv:1312.6229.
87. Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012). Indoor segmentation and support inference from RGBD images. In ECCV.
88. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.
89. Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast—But is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing (pp. 254–263). Association for Computational Linguistics.
90. Socher, R., Huval, B., Manning, C. D., & Ng, A. Y. (2012). Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning (pp. 1201–1211). Association for Computational Linguistics.
91. Steinbach, M., Karypis, G., Kumar, V., et al. (2000). A comparison of document clustering techniques. In KDD workshop on text mining, Boston (Vol. 400, pp. 525–526).
92. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., et al. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9).
93. Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., et al. (2016). YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2), 64–73.
94. Torralba, A., Fergus, R., & Freeman, W. T. (2008). 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11), 1958–1970.
95. Varma, M., & Zisserman, A. (2005). A statistical approach to texture classification from single images. International Journal of Computer Vision, 62(1–2), 61–81.
96. Vedantam, R., Lawrence Zitnick, C., & Parikh, D. (2015a). Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4566–4575).
97. Vedantam, R., Lin, X., Batra, T., Lawrence Zitnick, C., & Parikh, D. (2015b). Learning common sense through visual abstraction. In Proceedings of the IEEE international conference on computer vision (pp. 2542–2550).
98. Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3156–3164).
99. Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The Caltech-UCSD birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology.
100. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A., et al. (2010). Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3485–3492). IEEE.
101. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A. C., Salakhutdinov, R., Zemel, R. S., and Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. CoRR. arXiv:1502.03044.
102. Yang, Y., Baker, S., Kannan, A., & Ramanan, D. (2012). Recognizing proxemics in personal photos. In 2012 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3522–3529). IEEE.
103. Yao, B., & Fei-Fei, L. (2010). Modeling mutual context of object and human pose in human–object interaction activities. In 2010 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 17–24). IEEE.
104. Yao, B., Yang, X., & Zhu, S.-C. (2007). Introduction to a large-scale general purpose ground truth database: methodology, annotation tool and benchmarks. In Energy minimization methods in computer vision and pattern recognition (pp. 169–183). Springer.
105. Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2, 67–78.
106. Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015). Visual madlibs: Fill in the blank image generation and question answering. arXiv:1506.00278.
107. Zeng, D., Liu, K., Lai, S., Zhou, G., & Zhao, J. (2014). Relation classification via convolutional deep neural network. In Proceedings of COLING (pp. 2335–2344).
108. Zhou, G., Zhang, M., Ji, D. H., & Zhu, Q. (2007). Tree kernel-based relation extraction with context-sensitive structured parse tree information. In EMNLP-CoNLL 2007 (p. 728).
109. Zhu, J., Nie, Z., Liu, X., Zhang, B., & Wen, J.-R. (2009). Statsnowball: A statistical approach to extracting entity relationships. In Proceedings of the 18th international conference on world wide web (pp. 101–110). ACM.
110. Zhu, Y., Fathi, A., & Fei-Fei, L. (2014). Reasoning about object affordances in a knowledge base representation. In European conference on computer vision.
111. Zhu, Y., Zhang, C., Ré, C., & Fei-Fei, L. (2015). Building a large-scale multimodal knowledge base system for answering visual queries. arXiv:1507.05670.
112. Zitnick, C. L., & Parikh, D. (2013). Bringing semantics into focus using visual abstraction. In 2013 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3009–3016). IEEE.