Automatic description generation from images: A survey of models, datasets, and evaluation measures

Journal of Artificial Intelligence Research

Volumn 55, Issue , 2016, Pages 409-442

  Bernardi, Raffaella ^a   Cakici, Ruket ^b   Elliott, Desmond ^c   Erdem, Aykut ^d   Erdem, Erkut ^d   Ikizler Cinbis, Nazli ^d   Keller, Frank ^e   Muscat, Adrian ^f   Plank, Barbara ^g  

^a UNIVERSITY OF TRENTO   (Italy)

^b MIDDLE EAST TECHNICAL UNIVERSITY   (Turkey)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

^d HACETTEPE UNIVERSITY   (Turkey)

^e UNIVERSITY OF EDINBURGH   (United Kingdom)

^f UNIVERSITY OF MALTA   (Malta)

^g UNIVERSITY OF COPENHAGEN   (Denmark)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; IMAGE ANALYSIS; NATURAL LANGUAGE PROCESSING SYSTEMS; SURVEYS;

EVALUATION MEASURES; IMAGE DATASETS; IMAGE DESCRIPTIONS; LARGE AMOUNTS; MULTI-MODAL; NATURAL IMAGES; NATURAL LANGUAGE PROCESSING;

QUALITY CONTROL;

EID: 84960130911     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.4900     Document Type: Article

References (105)

1. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015). Vqa: Visual question answering. In International Conference on Computer Vision.
2. Banerjee, S., & Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Annual Meeting of the Association for Computational Linguistics Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization.
3. Berg, T. L., Berg, A. C., & Shih, J. (2010). Automatic attribute discovery and characterization from noisy web data. In European Conference on Computer Vision.
4. Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). Return of the devil in the details: Delving deep into convolutional nets. In British Machine Vision Conference.
5. Chen, J., Kuznetsova, P., Warren, D., & Choi, Y. (2015). Déjà image-captions: A corpus of expressive descriptions in repetition. In North American Chapter of the Association for Computational Linguistics.
6. Chen, X., & Zitnick, C. L. (2015). Mind's eye: A recurrent visual representation for image caption generation. In IEEE Conference on Computer Vision and Pattern Recognition.
7. Dale, R., & White, M. E. (Eds.). (2007). Workshop on Shared Tasks and Comparative Evaluation in Natural Language Generation: Position Papers.
8. Denkowski, M., & Lavie, A. (2014). Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Conference of the European Chapter of the Association for Computational Linguistics Workshop on Statistical Machine Translation.
9. Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015). Language Models for Image Captioning: The Quirks and What Works. In Annual Meeting of the Association for Computational Linguistics.
10. Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2015). Long-term recurrent convolutional networks for visual recognition and description. In IEEE Conference on Computer Vision and Pattern Recognition.
11. Elliott, D., & de Vries, A. P. (2015). Describing images using inferred visual dependency representations. In Annual Meeting of the Association for Computational Linguistics.
12. Elliott, D., & Keller, F. (2013). Image Description using Visual Dependency Representations. In Conference on Empirical Methods in Natural Language Processing.
13. Elliott, D., & Keller, F. (2014). Comparing Automatic Evaluation Measures for Image Description. In Annual Meeting of the Association for Computational Linguistics.
14. Elliott, D., Lavrenko, V., & Keller, F. (2014). Query-by-Example Image Retrieval using Visual Dependency Representations. In International Conference on Computational Linguistics.
15. Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010). The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 88 (2), 303-338.
16. Fader, A., Zettlemoyer, L., & Etzioni, O. (2013). Paraphrase-driven learning for open question answering. In Annual Meeting of the Association for Computational Linguistics.
17. Fader, A., Zettlemoyer, L., & Etzioni, O. (2014). Open question answering over curated and extracted knowledge bases. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
18. Fang, H., Gupta, S., Iandola, F., Srivastava, R., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J., Zitnick, C. L., & Zweig, G. (2015). From captions to visual concepts and back. In IEEE Conference on Computer Vision and Pattern Recognition.
19. Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010). Every picture tells a story: Generating sentences from images. In European Conference on Computer Vision.
20. Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (9), 1627-1645.
21. Feng, Y., & Lapata, M. (2008). Automatic Image Annotation Using Auxiliary Text Information. In Annual Meeting of the Association for Computational Linguistics.
22. Feng, Y., & Lapata, M. (2013). Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35 (4), 797-812.
23. Ferraro, F., Mostafazadeh, N., Huang, T., Vanderwende, L., Devlin, J., Galley, M., & Mitchell, M. (2015). A survey of current datasets for vision and language research. In Conference on Empirical Methods in Natural Language Processing.
24. Gao, H., Mao, J., Zhou, J., Huang, Z., & Yuille, A. (2015). Are you talking to a machine? dataset and methods for multilingual image question answering. In International Conference on Learning Representations.
25. Geman, D., Geman, S., Hallonquist, N., & Younes, L. (2015). Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112 (12), 3618-3623.
26. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition.
27. Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., & Lazebnik, S. (2014). Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections. In European Conference on Computer Vision.
28. Grubinger, M., Clough, P., Muller, H., & Deselaers, T. (2006). The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In International Conference on Language Resources and Evaluation.
29. Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., & Saenko, K. (2013). Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In International Conference on Computer Vision.
30. Gupta, A., Verma, Y., & Jawahar, C. V. (2012). Choosing linguistics over vision to describe images. In AAAI Conference on Artificial Intelligence.
31. Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004). Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16 (12), 2639-2664.
32. Hodosh, M., & Hockenmaier, J. (2013). Sentence-based image description with scalable, explicit models. In IEEE Conference on Computer Vision and Pattern Recognition Workshops.
33. Hodosh, M., Young, P., & Hockenmaier, J. (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. Journal of Artificial Intelligence Research, 47, 853-899.
34. Hotelling, H. (1936). Relations between two sets of variates. Biometrika, 0, 321-377.
35. Jaimes, A., & Chang, S.-F. (2000). A conceptual framework for indexing visual information at multiple levels. In IST SPIE Internet Imaging.
36. Jas, M., & Parikh, D. (2015). Image specificity. In IEEE Conference on Computer Vision and Pattern Recognition.
37. Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015). Guiding the long-short term memory model for image caption generation. In International Conference on Computer Vision.
38. Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M., & Fei-Fei, L. (2015). Image retrieval using scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition.
39. Karpathy, A., & Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition.
40. Karpathy, A., Joulin, A., & Fei-Fei, L. (2014). Deep Fragment Embeddings for Bidirectional Image Sentence Mapping. In Advances in Neural Information Processing Systems.
41. Khan, M. U. G., Zhang, L., & Gotoh, Y. (2011). Towards coherent natural language description of video streams. In International Conference on Computer Vision Workshops.
42. Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2015). Unifying visual-semantic embeddings with multimodal neural language models. In Advances in Neural Information Processing Systems Deep Learning Workshop.
43. Krishnamoorthy, N., Malkarnenkar, G., Mooney, R., Saenko, K., & Guadarrama, S. (2013). Generating Natural-Language Video Descriptions Using Text-Mined Knowledge. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
44. Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011). Baby talk: Understanding and generating simple image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition.
45. Kuznetsova, P., Ordonez, V., Berg, A. C., Berg, T. L., & Choi, Y. (2012). Collective Generation of Natural Image Descriptions. In Annual Meeting of the Association for Computational Linguistics.
46. Kuznetsova, P., Ordonezz, V., Berg, T. L., & Choi, Y. (2014). TREETALK: Composition and compression of trees for image descriptions. In Conference on Empirical Methods in Natural Language Processing.
47. Lampert, C. H., Nickisch, H., & Harmeling, S. (2009). Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conference on Computer Vision and Pattern Recognition.
48. Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In IEEE Conference on Computer Vision and Pattern Recognition.
49. Lebret, R., Pinheiro, P. O., & Collobert, R. (2015). Phrase-based image captioning. In International Conference on Machine Learning.
50. Li, S., Kulkarni, G., Berg, T. L., Berg, A. C., & Choi, Y. (2011). Composing simple image descriptions using web-scale n-grams. In The SIGNLL Conference on Computational Natural Language Learning.
51. Liang, P., Jordan, M. I., & Klein, D. (2012). Learning dependency-based compositional semantics. Computational Linguistics, 39 (2), 389-446.
52. Lin, C.-Y., & Hovy, E. (2008). Automatic evaluation of summaries using n-gram cooccurrence statistics. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
53. Lin, D., Fidler, S., Kong, C., & Urtasun, R. (2015). Generating multi-sentence natural language descriptions of indoor scenes. In British Machine Vision Conference.
54. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014). Microsoft COCO: Common objects in context. In European Conference on Computer Vision.
55. Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60 (4), 91-110.
56. Ma, L., Lu, Z., & Li, H. (2016). Learning to answer questions from image using convolutional neural network. In AAAI Conference on Artificial Intelligence.
57. Malinowski, M., & Fritz, M. (2014a). A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems.
58. Malinowski, M., & Fritz, M. (2014b). Towards a visual turing challenge. In Advances in Neural Information Processing Systems Workshop on Learning Semantics.
59. Malinowski, M., Rohrbach, M., & Fritz, M. (2015). Ask your neurons: A neural-based approach to answering questions about images. In International Conference on Computer Vision.
60. Mao, J., Xu, W., Yang, Y., Wang, J., & Yuille, A. L. (2015a). Deep captioning with multimodal recurrent neural networks (m-RNN). In International Conference on Learning Representations.
61. Mao, J., Wei, X., Yang, Y., Wang, J., Huang, Z., & Yuille, A. L. (2015b). Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In International Conference on Computer Vision.
62. Mason, R., & Charniak, E. (2014). Nonparametric Method for Data-driven Image Captioning. In Annual Meeting of the Association for Computational Linguistics.
63. Mason, W. A., & Watts, D. J. (2009). Financial incentives and the "performance of crowds". In ACM SIGKDD Workshop on Human Computation.
64. Mitchell, M., Han, X., Dodge, J., Mensch, A., Goyal, A., Berg, A. C., Yamaguchi, K., Berg, T. L., Stratos, K., Daume, III, H., & III (2012). Midge: generating image descriptions from computer vision detections. In Conference of the European Chapter of the Association for Computational Linguistics.
65. Nenkova, A., & Vanderwende, L. (2005). The impact of frequency on summarization. Tech. rep., Microsoft Research.
66. Oliva, A., & Torralba, A. (2001). Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42 (3), 145-175.
67. Ordonez, V., Kulkarni, G., & Berg, T. L. (2011). Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems.
68. Ortiz, L. M. G., Wolff, C., & Lapata, M. (2015). Learning to Interpret and Describe Abstract Scenes. In Conference of the North American Chapter of the Association of Computational Linguistics.
69. Panofsky, E. (1939). Studies in Iconology. Oxford University Press.
70. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics.
71. Parikh, D., & Grauman, K. (2011). Relative attributes. In International Conference on Computer Vision.
72. Park, C., & Kim, G. (2015). Expressing an image stream with a sequence of natural sentences. In Advances in Neural Information Processing Systems.
73. Patterson, G., Xu, C., Su, H., & Hays, J. (2014). The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding. International Journal of Computer Vision, 108 (1-2), 59-81.
74. Pinheiro, P., Lebret, R., & Collobert, R. (2015). Simple image description generator via a linear phrase-based model. In International Conference on Learning Representations Workshop.
75. Prest, A., Schmid, C., & Ferrari, V. (2012). Weakly supervised learning of interactions between humans and objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34 (3), 601-614.
76. Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010). Collecting image annotations using amazon's mechanical turk. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk.
77. Reiter, E., & Belz, A. (2009). An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35 (4), 529-588.
78. Reiter, E., & Dale, R. (2006). Building Natural Language Generation Systems. Cambridge University Press.
79. Ren, M., Kiros, R., & Zemel, R. (2015). Image question answering: A visual semantic embedding model and a new dataset. In International Conference on Machine Learningt Deep Learning Workshop.
80. Richardson, M., Burges, C. J., & Renshaw, E. (2013). MCTest: A challenge dataset for the open-domain machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing.
81. Rohrbach, A., Rohrback, M., Tandon, N., & Schiele, B. (2015). A dataset for movie description. In International Conference on Computer Vision.
82. Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., & Schiele, B. (2013). Translating Video Content to Natural Language Descriptions. In International Conference on Computer Vision.
83. Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., & Manning, C. D. (2015). Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Conference on Empirical Methods in Natural Language Processing Vision and Language Workshop.
84. Shatford, S. (1986). Analyzing the subject of a picture: A theoretical approach. Cataloging & Classification Quarterly, 6, 39-62.
85. Silberman, N., Kohli, P., Hoiem, D., & Fergus, R. (2012). Indoor segmentation and support inference from RGBD images. In European Conference on Computer Vision.
86. Socher, R., & Fei-Fei, L. (2010). Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In IEEE Conference on Computer Vision and Pattern Recognition.
87. Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. (2014). Grounded Compositional Semantics for Finding and Describing Images with Sentences. Transactions of the Association for Computational Linguistics, 2, 207-218.
88. Sun, C., Gan, C., & Nevatia, R. (2015). Automatic concept discovery from parallel text and visual corpora. In International Conference on Computer Vision.
89. Thomason, J., Venugopalan, S., Guadarrama, S., Saenko, K., & Mooney, R. (2014). Integrating Language and Vision to Generate Natural Language Descriptions of Videos in the Wild. In International Conference on Computational Linguistics.
90. Torralba, A., Fergus, R., & Freeman, W. T. (2008). 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30 (11), 1958-1970.
91. Ushiku, Y., Yamaguchi, M., Mukuta, Y., & Harada, T. (2015). Common subspace for model and similarity: Phrase learning for caption generation from images. In International Conference on Computer Vision.
92. Vedantam, R., Lawrence Zitnick, C., & Parikh, D. (2015). Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition.
93. Verma, Y., & Jawahar, C. V. (2014). Im2Text and Text2Im: Associating Images and Texts for Cross-Modal Retrieval. In British Machine Vision Conference.
94. Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition.
95. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning.
96. Yagcioglu, S., Erdem, E., Erdem, A., & Cakici, R. (2015). A Distributed Representation Based Query Expansion Approach for Image Captioning. In Annual Meeting of the Association for Computational Linguistics.
97. Yang, Y., Teo, C. L., Daume, III, H., & Aloimonos, Y. (2011). Corpus-guided sentence generation of natural images. In Conference on Empirical Methods in Natural Language Processing.
98. Yao, B., & Fei-Fei, L. (2010). Grouplet: A structured image representation for recognizing human and object interactions. In IEEE Conference on Computer Vision and Pattern Recognition.
99. Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015). Describing videos by exploiting temporal structure. In International Conference on Computer Vision.
100. Yatskar, M., Galley, M., Vanderwende, L., & Zettlemoyer, L. (2014). See No Evil, Say No Evil: Description Generation from Densely Labeled Images. In Joint Conference on Lexical and Computation Semantics.
101. Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2, 67-78.
102. Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015). Visual madlibs: Fill in the blank description generation and question answering. In International Conference on Computer Vision.
103. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In International Conference on Computer Vision.
104. Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013). Learning the visual interpretation of sentences. In International Conference on Computer Vision.
105. Zitnick, C. L., & Parikh, D. (2013). Bringing semantics into focus using visual abstraction. In IEEE Conference on Computer Vision and Pattern Recognition.