Jointly modeling deep video and compositional text to bridge vision and language in a unified framework

Proceedings of the National Conference on Artificial Intelligence

Volumn 3, Issue , 2015, Pages 2346-2352

  Xu, Ran ^a   Xiong, Caiming ^b   Chen, Wei ^a   Corso, Jason J ^c  

^a State University of New York   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF MICHIGAN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTATIONAL LINGUISTICS; MODELING LANGUAGES; NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS; VECTOR SPACES; VISUAL LANGUAGES;

COMPOSITIONAL SEMANTICS; DEEP NEURAL NETWORKS; DEPENDENCY TREES; LANGUAGE MODEL; NATURAL LANGUAGE GENERATION; SEMANTIC INFORMATION; UNIFIED FRAMEWORK; VIDEO RETRIEVAL;

UNIFIED MODELING LANGUAGE;

EID: 84940762015     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. Barbu, A.; Bridge, A.; Burchill, Z.; Coroian, D.; Dickinson, S.; Fidler, S.; Michaux, A.; Mussman, S.; Narayanaswamy, S.; Salvi, D.; Schmidt, L.; Shangguan, J.; rey Mark Siskind, 1; Waggoner, J.; Wang, S.; Wei, J.; Yin, Y; and Zhang, Z. 2012. Video in sentences out. In UAL
2. Chen, D. L., and Dolan, W. B. 2011. Collecting highly parallel data for paraphrase evaluation. In ACL.
3. Das, P.; Xu, C; Doell, R. F.; and Corso, J. J. 2013. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In CVPR.
4. Das, P.; Srihari, R. K.; and Corso, J. J. 2013. Translating related words to videos and back through latent topics. In WS DM.
5. Deng, J.; Li, K.; Do, M.; Su, H.; and Fei-Fei, L. 2009. Construction and analysis of a large scale image ontology. In Vision Science Society.
6. Donahua, J.; Jia, Y.; Vinyals, O.; Hoffman, J.; Zhang, N.; Tzeng, E.; and Darrell, T. 2013. Decaf: A deep convoiutional activation feature for generic visual recognition. In arXiv:1310.1531.
7. Felzenszwalb, P. F.; Girshick, R. B.; McAllester, D.; and Ramanan, D. 2010. Object detection with discriminatively trained part based models. TPAMI.
8. Frome, A.; Corrado, G. S.; Shlens, J.; Bengio, S.; Dean, J.; Mikolov, T.; et al. 2013. Devise: A deep visual-semantic embedding model. In NIPS.
9. Goller, C, and Kuchler, A. 1996. Learning task-dependent distributed representations by backpropagation through structure. In International Conference on Neural Networks.
10. Guadarrama, S.; Krishnamoorthy, N.; Malkarnenkar, G.; Venu-gopalan, S.; Mooney, R.; Darrell, T.; and Saenko, K. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV.
11. Gupta, A.; Srinivasan, P.; Shi, J.; and Davis, L. S. 2009. Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In CVPR.
12. Karpathy, A.; Toderici, G.; Shetty, S.; Leung, T.; Sukthankar, R.; and Fei-Fei, L. 2014. Large-scale video classification with convoiutional neural networks. In CVPR.
13. Klein, D., and Manning, C. D. 2013. Accurate unlexicalized parsing. In ACL.
14. Krishnamoorthy, N.; Malkarnenkar, G.; Mooney, R. J.; Saenko, K.; and Guadarrama, S. 2013. Generating natural-language video descriptions using text-mined knowledge. In AAAI.
15. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. lmagenet classification with deep convolutional neural networks. In NIPS.
16. Kulkarni, G.; Premraj, V.; Dhar, S.; Li, S.; Choi, Y.; Berg, A. C; and Berg, T. L. 2011. Baby talk: Understanding and generating simple image descriptions. In CVPR.
17. Li, L.-.I.; Su, H.; Xing, E. P.; and Fei-Fei, L. 2011. Object bank: A high-level image representation for scene classification and semantic feature sparsification. In NIPS.
18. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In MPS.
19. Miller, G. A. 1995. Wordnet: A lexical database for english. In Communications of the ACM, 39-41.
20. Motwani, T., and Mooney, R. 2012. improving video activity recognition using object recognition and text mining. In ECAL.
21. Moxley, E.; Mei, T.; and Manjunath, B. S. 2010. Video annotation through search and graph reinforcement mining. In IEEE Transactions on Multimedia.
22. Nakayama, H.; Harada, T.; and Kuniyoshi, Y. 2010. Evaluation of dimensionality reduction methods for image auto-annotation. In BMVC.
23. Pedersen, T.; Patwardhan, S.; and Michelizzi, J. 2004. Word-net: similarity - measuring the relatedness of concepts. In HLT-NAACL.
24. Pereira, F.; Tishby, N.; and Lee, L. 1993. Distributional clustering of english words. In ACL.
25. Ramanathan, V.; Liang, P.; and Fei-Fei, L. 2013. Video event understanding using natural language description. In ICCV.
26. Rohrbach, M.; Qiu, W.; Titov, I.; Thater, S.; Pinkal, M.; and Schiele, B. 2013. Translating video content to natural language descriptions. In ICCV.
27. Sadanand, S., and Corso, J. J. 2012. Action bank: A high-level representation of activity in video. In CVPR.
28. Socher, R., and Fei-Fei, L. 2010. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In CVPR.
29. Socher, R.: Karpathy, A.; Le, Q. V.; Manning, C. D.; and Ng, A. Y. 2013. Grounded compositional semantics for finding and describing images with sentences. In Transactions of the ACL.
30. Song, J.; Yang, Y.; and Huang, Z. 2011. Multiple feature hashing for real-time large scale near-duplicate video retrieval. In ACM International Conference on Multimedia.
31. Thomason, J.; Venugopalan, S.; Guadarrama, S.; Saenko, K.; and Mooney, R. 2014. Integrating language and vision to generate natural language descriptions of videos in the wild. In COLING.
32. Tian, Y; Sukthankar, R.; and Shah, M. 2013. Spatiotemporal de-formable part models for action detection. In CVPR.
33. Wang, H.; Kläser, A.; Schmid, C; and Liu, C.-L. 2011. Action recognition by dense trajectories. In CVPR.
34. Yao, T; Mei, T.; Ngo, C.-W.; and Li, S. 2013. Annotation for free: Video tagging by mining user search behavior. In ACM International Conference on Multimedia.
35. Yu, H., and Siskind, J. M. 2013. Grounded language learning from video described with sentences. In ACL.
36. Zhang, W.; Zhu, M.; and Derpanis, K. G. 2013. From ademes to action: A strongly-supervised representation for detailed action understanding. In ICCV.