Attention-based LSTM with semantic consistency for videos captioning

MM 2016 - Proceedings of the 2016 ACM Multimedia Conference

Volumn , Issue , 2016, Pages 357-361

  Guo, Zhao ^a   Gao, Lianli ^a   Song, Jingkuan ^b   Xu, Xing ^a   Shao, Jie ^a   Shen, Heng Tao ^a,c  

^a UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b Columbia University ^*  (United States)

^c UNIVERSITY OF QUEENSLAND   (Australia)

Author keywords

Attention mechanism; LSTM; Multimodal embedding; Semantic consistence; Video description

Indexed keywords

NEURAL NETWORKS;

ATTENTION MECHANISMS; CONVOLUTIONAL NEURAL NETWORK; LONG SHORT TERM MEMORY; LSTM; MULTI-MODAL; SEMANTIC CONSISTENCY; STATIC REPRESENTATION; VIDEO DESCRIPTION;

SEMANTICS;

EID: 84994560125     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2964284.2967242     Document Type: Conference Paper

References (27)

1. S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, volume 29, pages 65-72, 2005.
2. D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, pages 190-200. Association for Computational Linguistics, 2011.
3. X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014.
4. K. Cho, A. Courville, and Y. Bengio. Describing multimedia content using attention-based encoder-decoder networks. Multimedia, IEEE Transactions on, 17(11):1875-1886, 2015.
5. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, T. Darrell, and K. Saenko. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, pages 2625-2634, 2015.
6. H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, et al. From captions to visual concepts and back. In CVPR, pages 1473-1482, 2015.
7. L. Gao, J. Song, F. Nie, Y. Yan, N. Sebe, and H. T. Shen. Optimal graph learning with partial tags and multiple features for image and video annotation. In CVPR, pages 4371-4379, 2015.
8. L. Gao, J. Song, F. Nie, F. Zou, N. Sebe, and H. T. Shen. Graph-without-cut: An ideal graph learning for image segmentation. In AAAI, pages 1188-1194, 2016.
9. X. Jia, E. Gavves, B. Fernando, and T. Tuytelaars. Guiding long-short term memory for image caption generation. arXiv preprint arXiv:1509.04942, 2015.
10. A. Karpathy, A. Joulin, and F. F. F. Li. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS, pages 1889-1897, 2014.
11. G. Li, S. Ma, and Y. Han. Summarization-based video caption via deep neural networks. In ACM Multimedia, pages 1191-1194. ACM, 2015.
12. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. arXiv preprint arXiv:1410.1090, 2014.
13. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318. Association for Computational Linguistics, 2002.
14. J. Song, Y. Yang, Z. Huang, H. T. Shen, and J. Luo. Effective multiple feature hashing for large-scale near-duplicate video retrieval. IEEE Trans. Multimedia, 15(8):1997-2008, 2013.
15. J. Song, Y. Yang, Y. Yang, Z. Huang, and H. T. Shen. Inter-media hashing for large-scale retrieval from heterogeneous data sources. In SIGMOD, pages 785-796, 2013.
16. K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
17. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.
18. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. arXiv preprint arXiv:1412.0767, 2014.
19. R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In CVPR, pages 4566-4575, 2015.
20. S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video to text. In ICCV, pages 4534-4542, 2015.
21. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko. Translating videos to natural language using deep recurrent neural networks. arXiv preprint arXiv:1412.4729, 2014.
22. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, pages 3156-3164, 2015.
23. Q. Wu, C. Shen, A. V. D. Hengel, L. Liu, and A. Dick. Image captioning with an intermediate attributes layer. arXiv preprint arXiv:1506.01144, 2015.
24. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015.
25. R. Xu, C. Xiong, W. Chen, and J. J. Corso. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In AAAI, pages 2346-2352. Citeseer, 2015.
26. L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting temporal structure. In ICCV, pages 4507-4515, 2015.
27. M. D. Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.