Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 4, 2017, Pages 652-663

  Vinyals, Oriol ^a   Toshev, Alexander ^a   Bengio, Samy ^a   Erhan, Dumitru ^a  

^a GOOGLE INC   (United States)

Author keywords

Image captioning; language model; recurrent neural network; sequence to sequence

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; RECURRENT NEURAL NETWORKS;

IMAGE CAPTIONING; IMAGE DESCRIPTIONS; LANGUAGE MODEL; MACHINE TRANSLATIONS; MICROSOFT RESEARCHES; NATURAL LANGUAGE PROCESSING; SEQUENCE-TO-SEQUENCE; TARGET DESCRIPTIONS;

COMPUTER VISION;

EID: 85015770940     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2587640     Document Type: Article

References (49)

1. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet large scale visual recognition challenge," Int. J. Comput. Vis. (IJCV), vol. 115, no. 3, pp. 211-252, 2015.
2. A. Farhadi "Every picture tells a story: Generating sentences from images," in Proc. 11th Eur. Conf. Comput. Vis.: Part IV, 2010, pp. 15-29.
3. G. Kulkarni, "Baby talk: Understanding and generating simple image descriptions," in Proc. IEEE Conf Comput. Vis. Pattern Recog., 2011, pp. 1601-1608.
4. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proc. Empirical Methods Natural Lang. Process., 2014.
5. D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," arXiv:1409.0473, 2014.
6. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," in Proc. Neural Inf. Process. Syst., 2014.
7. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "Overfeat: Integrated recognition, localization and detection using convolutional networks," arXiv:1312.6229, 2013.
8. R. Gerber and H.-H. Nagel, "Knowledge representation for the generation of quantified natural language descriptions of vehicle traffic in image sequences," in Proc. Int. Conf. Image Process, 1996, pp. 805-808.
9. B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu, "I2t: Image parsing to text description," in Proc. IEEE, vol. 98, no. 8, pp. 1485-1508, Aug. 2010.
10. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi, "Composing simple image descriptions using web-scale n-grams," in Proc. Conf. Comput. Natural Lang. Learn., 2011, pp. 220-228.
11. M. Mitchell, "Midge: Generating image descriptions from computer vision detections," in Proc. 13th Conf. Eur. Chapter Assoc. Comput. Linguistics, 2012, pp. 747-756.
12. A. Aker and R. Gaizauskas, "Generating image descriptions using dependency relational patterns," in Proc. 48th Annu. Meet. Assoc. Comput. Linguistics, 2010, pp. 1250-1258.
13. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, "Collective generation of natural image descriptions," in Proc. 50th Annu. Meet. Assoc. Comput. Linguistics: Long Papers-Vol. 1, 2012, pp. 359-368.
14. P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi, "Treetalk: Composition and compression of trees for image descriptions," in Proc. Assoc. Comput. Linguistics, vol. 2, no. 10, 2014.
15. D. Elliott and F. Keller, "Image description using visual dependency representations," in Proc. Conf. Empirical Methods Natural Lang. Process., 2013, pp. 1292-1302.
16. M. Hodosh, P. Young, and J. Hockenmaier, "Framing image description as a ranking task: Data, models and evaluation metrics," J. Artif. Intell. Res., vol. 47, pp. 853-899, 2013.
17. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik, "Improving image-sentence embeddings using large weakly annotated photo collections," in Proc. Eur. Conf. Comput. Vis., 2014, pp. 529-545.
18. V. Ordonez, G. Kulkarni, and T. L. Berg, "Im2text: Describing images using 1 million captioned photographs," in Proc. Neural Inf. Process. Syst., 2011.
19. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, "Exploring nearest neighbor approaches for image captioning," arXiv:1505.04467, 2015.
20. M. Kolář, M. Hradiš, and P. Zemčík, "Technical report: Image captioning with semantically similar images," arXiv:1506.03995, 2015.
21. R. Socher, A. Karpathy, Q. V. Le, C. Manning, and A. Y. Ng, "Grounded compositional semantics for finding and describing images with sentences," in Proc. Assoc. Comput. Linguistics, 2014.
22. A. Karpathy, A. Joulin, and L. Fei-Fei, "Deep fragment embeddings for bidirectional image sentence mapping," in Proc. Neural Inf. Process. Syst., 2014.
23. H. Fang, "From captions to visual concepts and back," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015.
24. S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in Proc. Int. Conf. Mach. Learn., 2015.
25. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
26. R. Kiros and R. Z. R. Salakhutdinov, "Multimodal neural language models," in Proc. Neural Inf. Process. Syst. Deep Learn. Workshop, 2013.
27. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille, "Explain images with multimodal recurrent neural networks," in arXiv:1410.1090, 2014.
28. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille, "Deep captioning with multimodal recurrent neural networks (m-RNN)," Int. Conf. Learn. Representations, 2015.
29. R. Kiros, R. Salakhutdinov, and R. S. Zemel, "Unifying visualsemantic embeddings with multimodal neural language models," Trans. Assoc. Comput. Linguistics, 2015.
30. J. Donahue, "Long-term recurrent convolutional networks for visual recognition and description," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015.
31. K. Xu, "Show, attend and tell: Neural image caption generation with visual attention," in Proc. Int. Conf. Mach. Learn., 2015.
32. A. Karpathy and F.-F. Li, "Deep visual-semantic alignments for generating image descriptions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015.
33. X. Chen and C. L. Zitnick, "Mind's eye: A recurrent visual representation for image caption generation," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
34. J. Devlin, "Language models for image captioning: The quirks and what works," in Proc. Assoc. Comput. Linguistics, 2015.
35. J. Donahue, et al., "Decaf: A deep convolutional activation feature for generic visual recognition," in Proc. Int. Conf. Mach. Learn., 2014.
36. T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," in Int. Conf. Learn. Representations, 2013.
37. A. Graves, "Generating sequenceswith recurrent neural networks," arXiv:1308.0850, 2013.
38. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, "BLEU: A method for automatic evaluation of machine translation," in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics, 2002, pp. 311-318.
39. R. Vedantam, C. L. Zitnick, and D. Parikh, "CIDEr: Consensusbased image description evaluation," in arXiv:1411.5726, 2015.
40. S. Banerjee and A. Lavie, "Meteor: An automatic metric for MT evaluation with improved correlation with human judgments," in Proc. ACL Workshop Intrinsic Extrinsic Eval. Measures Mach. Transl. Summarization, vol. 29, 2005, pp. 65-72.
41. C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Proc. ACL Workshop Text Summarization Branches Out, vol. 8, 2004.
42. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, "Collecting image annotations using Amazon's Mechanical Turk," in Proc. NAACL HLT Workshop Creating Speech Lang. Data Amazon's Mech. Turk, 2010, pp. 139-147.
43. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, "From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions," Trans. Assoc. Comput. Linguistics, vol. 2, pp. 67-78, 2014.
44. T.-Y. Lin, et al., "Microsoft COCO: Common objects in context," arXiv:1405.0312, 2014.
45. W. Zaremba, I. Sutskever, and O. Vinyals, "Recurrent neural network regularization," in arXiv:1409.2329, 2014.
46. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3156-3164.
47. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, "Scheduled sampling for sequence prediction with recurrent neural networks," Adv. Neural Inf. Process. Syst., pp. 1171-1179, 2015.
48. C. Szegedy, et al., "Going deeper with convolutions," in arXiv:1409.4842, 2014.
49. L. Breiman, "Bagging predictors," in Proc. Mach. Learn., vol. 24, 1996, pp. 123-140.