Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks

IEEE Transactions on Multimedia

Volumn 17, Issue 11, 2015, Pages 1875-1886

  Cho, Kyunghyun ^a,b   Courville, Aaron ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b POLYTECHNIC UNIVERSITY   (United States)

Author keywords

Attention mechanism; deep learning; recurrent neural networks

Indexed keywords

NEURAL NETWORKS; SPEECH RECOGNITION; SPEECH TRANSMISSION;

ATTENTION MECHANISMS; CLASSIFICATION TASKS; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; DEEP NEURAL NETWORKS; JOINT DISTRIBUTIONS; MACHINE TRANSLATIONS; MULTIMEDIA CONTENTS;

RECURRENT NEURAL NETWORKS;

EID: 84946763507     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2015.2477044     Document Type: Article

References (75)

1. G. Kulkarni et al., "BabyTalk: Understanding and generating simple image descriptions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 12, pp. 2891-2903, Dec. 2013.
2. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
3. K. Cho et al., "Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proc. Empiricial Methods Natural Language Process., Oct. 2014, pp. 1724-1734.
4. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," in Proc. NIPS Deep Learn. Workshop, 2014.
5. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994.
6. S. Hochreiter, F. F. Informatik, Y. Bengio, P. Frasconi, and J. Schmidhuber, "Gradient flow in recurrent nets: The difficulty of learning longterm dependencies," in Field Guide to Dynamical Recurrent Networks, J. Kolen and S. Kremer, Eds. Piscataway, NJ, USA: IEEE Press, 2000.
7. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., May 2011, pp. 5528-5531.
8. A. Graves, "Sequence transduction with recurrent neural networks," in Proc. ICML Workshop Representation Learn., 2012.
9. N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, "Audio chord recognition with recurrent neural networks," in Proc. ISMIR, 2013, pp. 335-340.
10. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
11. A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Adv. Neural Inf. Process. Syst. 25, 2012.
12. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in Proc. ECCV, 2014, pp. 818-833.
13. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. ICLR, 2015.
14. C. Szegedy et al., "Going deeper with convolutions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 1-9.
15. P. Sermanet et al., "Overfeat: Integrated recognition, localization and detection using convolutional networks," in Proc. Int. Conf. Learn. Representations, 2014.
16. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition," in Proc. IEEE Conf. IEEE Comput. Vis. Pattern Recog. Workshops, Jun. 2014, pp. 512-519.
17. A. Karpathy and F.-F. Li, "Deep visual-semantic alignments for generating image descriptions," in Proc. Comput. Vis. Pattern Recog., 2015, pp. 3128-3137.
18. H. Fang et al., "From captions to visual concepts and back," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1411.4952
19. J. Mao et al., "Deep captioning with multimodal recurrent neural networks," in Proc. ICLR, 2015.
20. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," in Proc. Comput. Vis. Pattern Recog., 2015, pp. 3156-3164.
21. R. Kiros, R. Salakhutdinov, and R. Zemel, "Unifying visual-semantic embeddings with multimodal neural language models," CoRR, Dec. 2014 [Online]. Available: http://arxiv.org/abs/1411.2539
22. K. Xu et al., "Show, attend and tell: Neural image caption generation with visual attention," in Proc. ICML, 2015.
23. L. Yao et al., "Describing videos by exploiting temporal structure," in Proc. Int. Conf. Comput. Vis., Dec. 2015, accepted for publication.
24. N. Kalchbrenner and P. Blunsom, "Recurrent continuous translation models," in Proc. EMNLP, 2013, pp. 1700-1709.
25. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," in Proc. NIPS, 2014.
26. G. Taylor, R. Fergus, Y. LeCun, and C. Bregler, "Convolutional learning of spatio-temporal features," in ECCV, 2010, pp. 140-153.
27. A. Mnih and G. E. Hinton, "Three new graphical models for statistical language modelling," in Proc. 4th Int. Conf. Mach. Learn., 2007, pp. 641-648.
28. S. Venugopalan et al., "Translating videos to natural language using deep recurrent neural networks," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1412.4729
29. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, "On the properties of neural machine translation: Encoder-decoder approaches," in Proc. 8th Workshop Syntax, Semantics Struct. Statist. Transl., Oct. 2014, pp. 103-111.
30. J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, "Striving for simplicity: The all convolutional net," in Proc. ICLR, 2015.
31. M. Denil, A. Demiraj, and N. de Freitas, "Extraction of salient sentences from labelled documents," Univ. of Oxford, Oxford, U.K., Tech. Rep, 2014.
32. D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in Proc. ICLR, 2015.
33. J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, "Attention- based models for speech recognition," in Proc. Adv. Neural Inf. Process. Syst. 27, Dec. 2015, accepted for publication.
34. A. Graves, "Generating sequences with recurrent neural networks," arXiv:1308.0850, Tech. Rep., 2013.
35. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, pp. 533-536, 1986.
36. G. E. Hinton and T. J. Sejnowski, "Learning and relearning in boltzmann machines," in Parallel Distributed Processing: Explorations in theMicrostructure of Cognition. Volume 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA, USA: MIT Press, 1986, pp. 282-317.
37. Y. Bengio, N. Léonard, and A. Courville, "Estimating or propagating gradients through stochastic neurons for conditional computation," CoRR, 2013 [Online]. Available: http://arxiv.org/abs/1308.3432
38. Y. Tang and R. Salakhutdinov, "Learning stochastic feedforward neural networks," in Proc. NIPS, 2013.
39. J. Ba, V. Mnih, and K. Kavukcuoglu, "Multiple object recognition with visual attention," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1412.7755
40. T. Raiko, M. Berglund, G. Alain, and L. Dinh, "Techniques for learning binary stochastic feedforward neural networks," in Proc. ICLR, 2015.
41. A. Mnih and K. Gregor, "Neural variational inference and learning in belief networks," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1402.0030
42. S. Jean, K. Cho, R. Memisevic, and Y. Bengio, "On using very large target vocabulary for neural machine translation," in Proc. ACL-IJCNLP, 2015, pp. 1-10.
43. N. Durrani, B. Haddow, P. Koehn, and K. Heafield, "Edinburgh's phrase-based machine translation systems for WMT-14," in Proc. 9th Workshop Statist. Mach. Transl. Assoc. Comput. Linguistics, Baltimore, MD, USA, 2014, pp. 97-104.
44. C. Gulcehre et al., "On using monolingual corpora in neural machine translation," CoRR, 2015 [Online]. Available: http://arxiv.org/abs/1503.03535
45. J. Devlin et al., "Language models for image captioning: The quirks and what works," CoRR, 2015 [Online]. Available: http://arxiv.org/abs/1505.01809
46. J. Donahue et al., "Long-term recurrent convolutional networks for visual recognition and description," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1411.4389
47. M. Hodosh, P. Young, and J. Hockenmaier, "Framing image description as a ranking task: Data, models and evaluation metrics," J. Artificial Intell. Res., vol. 47, no. 1, pp. 853-899, May 2013.
48. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, "From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions," in Proc. TACL, 2014, vol. 2, pp. 67-78.
49. T.-Y. Lin et al., "Microsoft COCO: Common objects in context," in Proc. ECCV, 2014, pp. 740-755.
50. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: A method for automatic evaluation of machine translation," in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics, 2002, pp. 311-318.
51. M. Denkowski and A. Lavie, "Meteor universal: Language specific translation evaluation for any target language," in Proc. EACL Workshop Statist. Mach. Transl., 2014.
52. C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Proc. ACL-04 Workshop, 2004, vol. 8, pp. 74-81.
53. R. Vedantam, C. L. Zitnick, and D. Parikh, "CIDEr: Consensus-based image description evaluation," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1411.5726
54. D. L. Chen and W. B. Dolan, "Collecting highly parallel data for paraphrase evaluation," in Proc. 49th Annu. Meeting Assoc. Comput. Linguistics, Jun. 2011, pp. 190-200.
55. A. Torabi, C. Pal, H. Larochelle, and A. Courville, "Using descriptive video services to create a large data source for video annotation research," CoRR, 2015 [Online]. Available: http://arxiv.org/abs/1503. 01070
56. G. E. Hinton et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
57. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks," in Proc. ICML, 2006, pp. 369-376.
58. A. Graves and N. Jaitly, "Towards end-to-end speech recognition with recurrent neural networks," in Proc. ICML, 2014.
59. A. Hannun et al., "Deep speech: Scaling up end-to-end speech recognition," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1412.5567
60. R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio, "How to construct deep recurrent neural networks," in Proc. ICLR, 2014.
61. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, "DARPA TIMIT acoustic-phonetic continous speech corpus," NASA STI, Hampton, VA, USA, Tech. Rep. NISTIR 4930, 1993, vol. 93.
62. A. Graves, A.-R. Mohamed, and G. Hinton, "Speech recognition with deep recurrent neural networks," in Proc. ICASSP'2013, 2013, pp. 6645-6649.
63. L. Tóth, "Combining time-and frequency-domain convolution in convolutional neural network-based phone recognition," in Proc. ICASSP, 2014, pp. 190-194.
64. O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton, "Grammar as a foreign language," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1412.7449
65. O. Vinyals, M. Fortunato, and N. Jaitly, "Pointer networks," CoRR, to be published.
66. S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus, "Weakly supervised memory networks," CoRR, 2015 [Online]. Available: http://arxiv.org/abs/1503.08895
67. V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, "Recurrent models of visual attention," in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds. Red Hook, NY, USA: Curran & Assoc., 2014, pp. 2204-2212.
68. K. Gregor, I. Danihelka, A. Graves, and D. Wierstra, "DRAW: A recurrent neural network for image generation," CoRR, 2015 [Online]. Available: http://arxiv.org/abs/1502.04623
69. H. Larochelle and G. E. Hinton, "Learning to combine foveal glimpses with a third-order Boltzmann machine," in Proc. Adv. Neural Inf. Process. Syst. 23, 2010, pp. 1243-1251.
70. M. Denil, L. Bazzani, H. Larochelle, and N. de Freitas, "Learning where to attend with deep architectures for image tracking," Neural Computation, vol. 24, no. 8, pp. 2151-2184, 2012.
71. Y. Zheng, R. S. Zemel, Y.-J. Zhang, and H. Larochelle, "A neural autoregressive approach to attention-based recognition," Int. J. Comput. Vis., vol. 113, no. 1, pp. 67-79, 2014.
72. G. Dulac-Arnold, L. Denoyer, N. Thome, M. Cord, and P. Gallinari, "Sequentially generated instance-dependent image representations for classification," in Proc. Int. Conf. Learn. Representations, 2014.
73. A. Graves, G. Wayne, and I. Danihelka, "Neural turing machines," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1410.5401
74. J. Weston, S. Chopra, and A. Bordes, "Memory networks," CoRR, 2014 [Online]. Available: http://arxiv.org/abs/1410.3916
75. L. Wehbe, B. Murphy, P. Talukdar, A. Fyshe, A. Ramdas, and T. Mitchell, "Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses," PLoS ONE, vol. 9, no. 11, p. e112575, Nov. 2014. The authors would like to thank the following for computing support: NSERC, FRQNT, Calcul Québec, Compute Canada, the Canada Research Chairs, CIFAR, and Samsung.