Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 4, 2017, Pages 677-691

  Donahue, Jeff ^a   Hendricks, Lisa Anne ^a   Rohrbach, Marcus ^a,b   Venugopalan, Subhashini ^c   Guadarrama, Sergio ^a   Saenko, Kate ^d   Darrell, Trevor ^a,b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b INTERNATIONAL COMPUTER SCIENCE INSTITUTE   (United States)

^c University of Texas at Austin   (United States)

^d University of Massachusetts Lowell   (United States)

Author keywords

Computer vision; convolutional nets; deep learning; transfer learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION; DEEP LEARNING;

COMPOSITIONAL REPRESENTATION; CONVOLUTIONAL NETS; CONVOLUTIONAL NETWORKS; LONG-TERM DEPENDENCIES; NATURAL LANGUAGE TEXT; PERCEPTUAL REPRESENTATIONS; TRANSFER LEARNING; VISUAL REPRESENTATIONS;

CONVOLUTION;

EID: 85020685307     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2599174     Document Type: Article

References (73)

1. S. Ji, W. Xu, M. Yang, and K. Yu, "3D convolutional neural networks for human action recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 221-231, Jan. 2013
2. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt, "Sequential deep learning for human action recognition," in Proc. 2nd Int. Conf. Human Behavior Understanding, 2011, pp. 29-39
3. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, "Large-scale video classification with convolutional neural networks," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 1725-1732
4. K. Simonyan and A. Zisserman, "Two-stream convolutional networks for action recognition in videos," in Proc. Advances Neural Inf. Process. Syst., 2014, pp. 568-576
5. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," DTIC Document, Fort Belvoir, VA, USA, Tech. Rep. ICS 8506, 1985
6. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," in Neural Computation. Cambridge, MA, USA: MIT Press, 1989
7. S. Hochreiter and J. Schmidhuber, "Long short-term memory, " in, Neural Computation. Cambridge, MA, USA: MIT Press, 1997
8. A. Graves and N. Jaitly, "Towards end-to-end speech recognition with recurrent neural networks," in Proc. 31st Int. Conf. Mach. Learn., 2014, pp. 1764-1772
9. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," in Proc. Advances Neural Inf. Process. Syst., 2014, pp. 3104-3112
10. K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, "On the properties of neural machine translation: Encoder-decoder approaches," in Proc. 8th Workshop Syntax Semantics Struct. Statistical Transl., 2014, pp. 103-111
11. M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele, "Translating video content to natural language descriptions," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 433-440
12. Y. Jia "Caffe: Convolutional architecture for fast feature embedding," in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 675-678
13. W. Zaremba and I. Sutskever, "Learning to execute," CoRR, vol. abs/1410.4615, 2014, http://arxiv.org/abs/1410.4615
14. A. Graves, "Generating sequences with recurrent neural networks," CoRR, vol. abs/1308.0850, 2013, http://arxiv.org/abs/1308.0850
15. O. Vinyals, S. V. Ravuri, and D. Povey, "Revisiting recurrent neural networks for robust ASR," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2012, pp. 4085-4088
16. I. Sutskever, J. Martens, and G. E. Hinton, "Generating text with recurrent neural networks," in Proc. 28th Int. Conf. Mach. Learn., 2011, pp. 1017-1024
17. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Advances Neural Inf. Process. Syst., 2012, pp. 1106-1114
18. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Representations, 2015
19. C. Szegedy, et al., "Going deeper with convolutions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 1-9
20. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proc. Conf. Empirical Methods Natural Language Process., 2014, pp. 1724-1734
21. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, "High accuracy optical flow estimation based on a theory for warping," in Proc. Eur. Conf. Comput. Vis., 2004, pp. 25-36
22. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in Proc. Eur. Conf. Comput. Vis., 2014, pp. 818-833
23. O. Russakovsky, et al., "ImageNet Large Scale Visual Recognition Challenge," Int. J. Comput. Vis., vol. 115, no. 3, pp. 211-252, 2015
24. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 248-255
25. K. Soomro, A. R. Zamir, and M. Shah, "UCF101: A dataset of 101 human actions classes from videos in the wild," Univ. Central Florida, Orlando, FL, USA, Tech. Rep. CRCV-TR-12-01, 2012
26. P. Y. Micah Hodosh and, J. Hockenmaier, "Framing image description as a ranking task: Data, models and evaluation metrics," in J. Artificial Intell. Res., vol. 47, no. 1, pp. 853-899, 2013
27. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille, "Deep captioning with multimodal recurrent neural networks (m-RNN)," presented at the Int. Conf. Learn. Representations, San Diego, CA, USA, 2015
28. A. Karpathy, A. Joulin, and L. Fei-Fei, "Deep fragment embeddings for bidirectional image sentence mapping," in Proc. Advances Neural Inf. Process. Syst., 2014, pp. 1889-1897
29. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng, "Grounded compositional semantics for finding and describing images with sentences," Trans. Assoc. Comput. Linguistics, vol. 2, pp. 207-218, 2014
30. A. Frome, et al., "Devise: A deep visual-semantic embedding model," in Advances Neural Inf. Process. Syst., 2013, pp. 2121-2129
31. R. Kiros, R. Salakhuditnov, and R. S. Zemel, "Unifying visualsemantic embeddings with multimodal neural language models," CoRR, vol. abs/1411.2539, 2014, http://arxiv.org/abs/1411.2539
32. M. H. Peter Young, A. Lai, and J. Hockenmaier, "From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions," Trans. Assoc. Comput. Linguistics, vol. 2, pp. 67-68, 2014
33. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, "Microsoft COCO: Common objects in context," Eur. Conf. Comput. Vis., Springer, pp. 740-755, 2014
34. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: A method for automatic evaluation of machine translation," in Proc. 40th Annu. Meet. Assoc. Comput. Linguistics, 2002, pp. 311-318
35. R. Vedantam, C. L. Zitnick, and D. Parikh, "CIDEr: Consensusbased image description evaluation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 4566-4575
36. S. Banerjee and A. Lavie, "METEOR: An automatic metric for MT evaluation with improved correlation with human judgments," in Proc. ACL Workshop Intrinsic Extrinsic Evaluation Measures Mach. Transl. Summarization, 2005, pp. 65-72
37. C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Proc. ACL Workshop Text Summarization Branches Out, 2004, pp. 25-26
38. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3156-3164
39. J. Devlin, et al., "Language models for image captioning: The quirks and what works," in Proc. 53rd Annu. Meeting Assoc. Comput. Linguistics, 2015, pp. 100-105
40. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, "Learning like a child: Fast novel visual concept learning from sentence descriptions of images," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 2533-2541
41. H. Fang, et al., "From captions to visual concepts and back," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 1473-1482
42. J. Devlin, S. Gupta, R. B. Girshick, M. Mitchell, and C. L. Zitnick, "Exploring nearest neighbor approaches for image captioning," CoRR, vol. abs/1505.04467, 2015, http://arxiv.org/abs/1505.04467
43. J. Donahue, et al., "Long-term recurrent convolutional networks for visual recognition and description," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 2625-2634
44. K. Xu, et al., "Show, attend and tell: Neural image caption generation with visual attention," presented at the 32nd Int. Conf. Mach. Learn., Lille, France, 2015
45. A. Karpathy and L. Fei-Fei, "Deep visual-semantic alignments for generating image descriptions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3128-3137
46. P. Kuznetsova, V. Ordonez, T. L. Berg, U. C. Hill, and Y. Choi, "TreeTalk: Composition and compression of trees for image descriptions," Trans. Assoc. Comput. Linguistics, vol. 2, no. 10, pp. 351-362, 2014
47. P. Koehn, et al., "Moses: Open source toolkit for statistical machine translation," in Proc. 45th Annu. Meeting ACL Interactive Poster Demonstration Sessions, 2007, pp. 177-180
48. A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, M. Pinkal, and B. Schiele, "Coherent multi-sentence video description with variable level of detail," in German Conf. Pattern Recog. (GCPR). Berlin, Germany: Springer, 2014
49. H. Wang, A. Klaser, C. Schmid, and C. Liu, "Dense trajectories and motion boundary descriptors for action recognition," Int. J. Comput. Vis., vol. 103, pp. 60-79, 2013
50. H. Wang and C. Schmid, "Action recognition with improved trajectories," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3551-3558
51. H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, "HMDB: A large video database for human motion recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2556-2563
52. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt, "Action classification in soccer videos with long short-term memory recurrent neural networks," in Proc. 20th Int. Conf. Artificial Neural Netw., 2010, pp. 154-159
53. A. Farhadi, et al., "Every picture tells a story: Generating sentences from images," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 15-29
54. G. Kulkarni, et al., "Baby talk: Understanding and generating simple image descriptions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1601-1608
55. Y. Yang, C. L. Teo, H. Daume III, and Y. Aloimonos, "Corpusguided sentence generation of natural images," in Proc. Conf. Empirical Methods Natural Language Process., 2011, pp. 444-454
56. M. Mitchell, et al., "Midge: Generating image descriptions from computer vision detections," in Proc. 13th Conf. Eur. Chapter. Assoc. Comput. Linguistics, 2012, pp. 747-756
57. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, "Collective generation of natural image descriptions," in Proc. 50th Annu. Meeting Assoc. Comput. Linguistics: Long Papers, 2012, pp. 359-368
58. R. Kiros, R. Salakhutdinov, and R. Zemel, "Multimodal neural language models," in Proc. 31st Int. Conf. Mach. Learn., 2014, 595-603
59. S. Guadarrama, et al., "YouTube2Text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shoot recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2712-2719
60. M. U. G. Khan, L. Zhang, and Y. Gotoh, "Human focused video description," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2011, pp. 1480-1487
61. A. Barbu, et al., "Video in sentences out," in Proc. Conf. Uncertainty Artificial Intell., 2012, pp. 102-112
62. P. Das, C. Xu, R. Doell, and J. Corso, "Thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 2634-2641
63. C. C. Tan, Y.-G. Jiang, and C.-W. Ngo, "Towards textually describing complex video contents with audio-visual concept classifiers," in Proc. 19th ACM Int. Conf. Multimedia, 2011, pp. 655-658
64. J. Thomason, S. Venugopalan, S. Guadarrama, K. Saenko, and R. J. Mooney, "Integrating language and vision to generate natural language descriptions of videos in the wild," in Proc. 25th Int. Conf. Comput. Linguistics, 2014, pp. 1218-1227
65. H. Sak, et al., "Sequence discriminative distributed training of long short-term memory recurrent neural networks," presented at the 15th Annu. Conf. Int. Speech Commun. Assoc., Singapore, 2014
66. J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, "Beyond short snippets: Deep networks for video classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 4694-4702
67. S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and F.-F. Li, "Every moment counts: Dense detailed labeling of actions in complex videos," CoRR, vol. abs/1507.05738, 2015, http://arxiv. org/abs/1507.05738
68. L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, "Deep compositional captioning: Describing novel object categories without paired training data," Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016
69. S. Venugopalan,H. Xu, J.Donahue,M. Rohrbach, R.Mooney, and K. Saenko, "Translating videos to natural language using deep recurrent neural networks," in Proc. North Amer. Chapter Assoc. Comput. Linguistics-Human Language Technol., 2015, pp. 1494-1504
70. S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko, "Sequence to sequence-video to text," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 4534-4542
71. L. Yao, et al., "Describing videos by exploiting temporal structure," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., 2015, pp. 4507-4515
72. A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele, "Grounding of textual phrases in images by reconstruction," CoRR, vol. abs/1511.03745, 2015, http://arxiv.org/abs/1511.03745
73. R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell, "Natural language object retrieval," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 4555-4564.