What value do explicit high level concepts have in vision to language problems?

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 2016-December, Issue , 2016, Pages 203-212

  Wu, Qi ^a   Shen, Chunhua ^a   Liu, Lingqiao ^a   Dick, Anthony ^a   Van Den Hengel, Anton ^a  

^a UNIVERSITY OF ADELAIDE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; NEURAL NETWORKS; PATTERN RECOGNITION; RECURRENT NEURAL NETWORKS; SEMANTICS;

BENCHMARK DATASETS; CONVOLUTIONAL NEURAL NETWORK; EXPLICIT REPRESENTATION; HIGH LEVEL SEMANTICS; HIGH-LEVEL INFORMATION; QUESTION ANSWERING; RECURRENT NEURAL NETWORK (RNNS); SEMANTIC INFORMATION;

HIGH LEVEL LANGUAGES;

EID: 84986301177     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2016.29     Document Type: Conference Paper

References (59)

1. A. Aker and R. Gaizauskas. Generating image descriptions using dependency relational patterns. In Proc. Conf. Associ-ation for Computational Linguistics, 2010.
2. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question Answering. In Proc. IEEE Int. Conf. Comp. Vis., 2015.
3. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Proc. Int. Conf. Learn. Representations, 2015.
4. S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005.
5. X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. ArXiv: 1504. 00325, 2015.
6. X. Chen and C. Lawrence Zitnick. Mind's Eye: A Recurrent Visual Representation for Image Caption Generation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., June 2015.
7. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proc. Conf. Empirical Methods in Natural Language Processing, 2014.
8. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2009.
9. J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell. Language models for image captioning: The quirks and what works. In Proc. IEEE Int. Conf. Comp. Vis., 2015.
10. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2015.
11. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al. From captions to visual concepts and back. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2015.
12. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In Proc. Eur. Conf. Comp. Vis. 2010.
13. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are You Talking to a Machine Dataset and Methods for Multilingual Image Question Answering. In Proc. Advances in Neural Inf. Process. Syst., 2015.
14. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proc. Int. Conf. Artificial Intell. & Stat., pages 249-256, 2010.
15. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In Proc. Eur. Conf. Comp. Vis. 2014.
16. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9 (8): 1735-1780, 1997.
17. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, pages 853-899, 2013.
18. X. Jia, E. Gavves, B. Fernando, and T. Tuytelaars. Guiding Long-Short Term Memory for Image Caption Generation. In Proc. IEEE Int. Conf. Comp. Vis., 2015.
19. Y. Jia, M. Salzmann, and T. Darrell. Learning cross-modality similarity for multinomial data. In Proc. IEEE Int. Conf. Comp. Vis., 2011.
20. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional Architecture for Fast Feature Embedding. ArXiv: 1408. 5093, 2014.
21. J. Jin, K. Fu, R. Cui, F. Sha, and C. Zhang. Aligning where to see and what to tell: image caption with region-based attention and scene factorization. ArXiv: 1506. 06272, 2015.
22. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2015.
23. A. Karpathy, A. Joulin, and F. F. Li. Deep fragment embeddings for bidirectional image sentence mapping. In Proc. Advances in Neural Inf. Process. Syst., 2014.
24. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. In Proc. Conf. Association for Computational Linguistics, 2015.
25. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Proc. Advances in Neural Inf. Process. Syst., 2012.
26. G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. IEEE Trans. Pattern Anal. Mach. Intell.
27. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In Proc. Conf. Association for Computational Linguistics, 2012.
28. P. Kuznetsova, V. Ordonez, T. L. Berg, and Y. Choi. Treetalk: Composition and compression of trees for image descriptions. Proc. Conf. Association for Computational Linguistics, 2014.
29. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proc. IEEE, 86 (11): 2278-2324, 1998.
30. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In CoNLL, 2011.
31. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In Proc. Eur. Conf. Comp. Vis. 2014.
32. L. Ma, Z. Lu, and H. Li. Learning to Answer Questions From Image using Convolutional Neural Network. In AAAI, 2016.
33. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Proc. Advances in Neural Inf. Process. Syst., pages 1682-1690, 2014.
34. M. Malinowski and M. Fritz. Hard to Cheat: A Turing Test based on Answering Questions about Images. ArXiv: 1501. 03302, 2015.
35. M. Malinowski, M. Rohrbach, and M. Fritz. Ask Your Neurons: A Neural-based Approach to Answering Questions about Images. In Proc. IEEE Int. Conf. Comp. Vis., 2015.
36. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). In Proc. Int. Conf. Learn. Representations, 2015.
37. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Proc. Advances in Neural Inf. Process. Syst., pages 3111-3119, 2013.
38. G. A. Miller. WordNet: A lexical database for English. Communications of the ACM, 38 (11): 39-41, 1995.
39. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III. Midge: Generating image descriptions from computer vision detections. In EACL, 2012.
40. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In Proc. Advances in Neural Inf. Process. Syst., 2011.
41. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evaluation of machine translation. In Proc. Conf. Association for Computational Linguistics, 2002.
42. J. Pont-Tuset, P. Arbeláez, J. Barron, F. Marques, and J. Malik. Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation. In arXiv: 1503. 00848, March 2015.
43. M. Ren, R. Kiros, and R. Zemel. Image Question Answering: A Visual Semantic Embedding Model and a New Dataset. In Proc. Advances in Neural Inf. Process. Syst., 2015.
44. M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele. Translating video content to natural language descriptions. In Proc. IEEE Int. Conf. Comp. Vis., 2013.
45. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proc. Int. Conf. Learn. Representations, 2015.
46. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng. Grounded compositional semantics for finding and describing images with sentences. Proc. Conf. Association for Computational Linguistics, 2014.
47. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Proc. Advances in Neural Inf. Process. Syst., 2014.
48. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2015.
49. R. Vedantam, C. L. Zitnick, and D. Parikh. CIDEr: Consensus-based Image Description Evaluation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2015.
50. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2014.
51. Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan. CNN: Single-label to multi-label. ArXiv: 1406. 5726, 2014.
52. Z. Wu and M. Palmer. Verbs semantics and lexical selection. In Proc. Conf. Association for Computational Linguistics, 1994.
53. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proc. Int. Conf. Mach. Learn., 2015.
54. Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos. Corpus-guided sentence generation of natural images. In Proc. Conf. Empirical Methods in Natural Language Processing, 2011.
55. L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting temporal structure. In Proc. IEEE Int. Conf. Comp. Vis., 2015.
56. Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo. Image captioning with semantic attention. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., June 2016.
57. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Proc. Conf. Association for Computational Linguistics, 2, 2014.
58. C. Zhang, J. C. Platt, and P. A. Viola. Multiple instance boosting for object detection. In Proc. Advances in Neural Inf. Process. Syst., 2005.
59. Y. Zhu, C. Zhang, C. Ré, and L. Fei-Fei. Building a Largescale Multimodal Knowledge Base for Visual Question Answering. ArXiv: 1507. 05670, 2015.