Integrating language and vision to generate natural language descriptions of videos in the wild

COLING 2014 - 25th International Conference on Computational Linguistics, Proceedings of COLING 2014: Technical Papers

Volumn , Issue , 2014, Pages 1218-1227

  Thomason, Jesse ^a   Venugopalan, Subhashini ^a   Guadarrama, Sergio ^b   Saenko, Kate ^c   Mooney, Raymond ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF MASSACHUSETTS LOWELL   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; LINGUISTICS; MODELING LANGUAGES; NATURAL LANGUAGE PROCESSING SYSTEMS; VIDEO SIGNAL PROCESSING; VISUAL LANGUAGES;

INDIVIDUAL COMPONENTS; N-GRAM LANGUAGE MODELS; NATURAL LANGUAGE PROCESSING; NATURAL LANGUAGES; PROBABILISTIC KNOWLEDGE; REAL WORLD VIDEOS; TEXTUAL DESCRIPTION; VISUAL RECOGNITION;

COMPUTATIONAL LINGUISTICS;

EID: 84959932469     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Video in sentences out. In Association for Uncertainty in Artificial Intelligence (UAI).
2. Tamara Berg and Julia Hockenmaier. 2013. Workshop on vision and language. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). NAACL.
3. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27.
4. David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190-200. Association for Computational Linguistics.
5. Stanley F. Chen and Joshua Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of the 34th annual meeting on Association for Computational Linguistics (ACL), pages 310-318. Association for Computational Linguistics.
6. Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. 2013. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
7. Jia Deng, Kai Li, Minh Do, Hao Su, and Li Fei-Fei. 2009. Construction and analysis of a large scale image ontology. Vision Sciences Society.
8. Jia Deng, Jonathan Krause, Alex Berg, and Li Fei-Fei. 2012. Hedging Your Bets: Optimizing Accuracy-Specificity Trade-offs in Large Scale Visual Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
9. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2013. Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531.
10. Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV), 88(2):303-338, June.
11. Yansong Feng and Mirella Lapata. 2013. Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 35(4):797-812.
12. Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In IEEE International Conference on Computer Vision (ICCV), December.
13. Niveda Krishnamoorthy, Girish Malkarnenkar, Raymond J. Mooney, Kate Saenko, and Sergio Guadarrama. 2013. Generating natural-language video descriptions using text-mined knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 541-547.
14. Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Alexander Berg, Yejin Choi, and Tamara Berg. 2011. Baby talk: Understanding and generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.
15. Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 2169-2178. IEEE.
16. Li-Jia Li, Hao Su, Eric Xing, and Li Fei-Fei. 2010. Object bank: A high-level image representation for scene classication and semantic feature sparsification. In Advances in Neural Information Processing Systems (NIPS).
17. Siming Li, Girish Kulkarni, Tamara L. Berg, Alexander C. Berg, and Yejin Choi. 2011. Composing simple image descriptions using web-scale n-grams. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning (CoNLL), pages 220-228, Stroudsburg, PA, USA. Association for Computational Linguistics.
18. Tanvi S. Motwani and Raymond J. Mooney. 2012. Improving video activity recognition using object recognition and text mining. In Proceedings of the European Conference on Artificial Intelligence (ECAI), pages 600-605.
19. Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems (NIPS), volume 24, pages 1143-1151.
20. Adam Pauls and Dan Klein. 2011. Faster and smaller n-gram language models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 258-267. Association for Computational Linguistics.
21. John C. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances In Large Margin Classifiers, pages 61-74. MIT Press.
22. Marcus Rohrbach, QiuWei, Ivan Titov, Stefan Thater, Manfred Pinkal, and Bernt Schiele. 2013. Translating video content to natural language descriptions. In IEEE International Conference on Computer Vision (ICCV).
23. Anna Senina, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. 2014. Coherent multi-sentence video description with variable level of detail. arXiv preprint arXiv:1403.6173.
24. Heng Wang, Alexander Klaser, Cordelia Schmid, and Cheng-Lin Liu. 2011. Action recognition by dense trajectories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3169-3176. IEEE.
25. Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Largescale scene recognition from abbey to zoo. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485-3492.
26. Yezhou Yang, Ching Lik Teo, Hal Daume, and Yiannis Aloimonos. 2011. Corpus-guided sentence generation of natural images. In Conference on Emperical Methods in Natural Language Processing (EMNLP), pages 444-454.
27. Haonan Yu and Jeffrey Mark Siskind. 2013. Grounded language learning from video described with sentences. In Proceedings of the Association for Computational Linguistics (ACL), pages 53-63.
28. Jianguo Zhang, Marcin Marszałek, Svetlana Lazebnik, and Cordelia Schmid. 2007. Local features and kernels for classification of texture and object categories: A comprehensive study. International Journal of Computer Vision (IJCV), 73(2):213-238.