Joint video and text parsing for understanding events and answering queries

IEEE Multimedia

Volumn 21, Issue 2, 2014, Pages 42-70

  Tu, Kewei ^a   Meng, Meng ^b   Lee, Mun Wai ^c   Choe, Tae Eun ^d   Zhu, Song Chun ^b  

^a SHANGHAITECH UNIVERSITY   (China)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c INTELLIGENT AUTOMATION INC   (United States)

^d ObjectVideo ^*  (United States)

Author keywords

AND OR graph; joint video and text parsing; knowledge representation; multimedia; multimedia video analysis; query answering

Indexed keywords

GRAPHIC METHODS; KNOWLEDGE REPRESENTATION; PROBABILITY DISTRIBUTIONS; SEMANTICS;

AND-OR GRAPH; MULTIMEDIA; MULTIMEDIA VIDEO; QUERY ANSWERING; TEXT PARSING;

QUERY PROCESSING;

EID: 84901405262     PISSN: 1070986X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MMUL.2014.29     Document Type: Article

References (35)

1. S.L. Feng, R. Manmatha, and V. Lavrenko, "Multiple Bernoulli Relevance Models for Image and Video Annotation," Proc. 2004 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), vol. 2, 2004, pp. 1002-1009.
2. K. Barnard et al., "MatchingWords and Pictures," J. Machine Learning Research, vol. 3, Mar. 2003, pp. 1107-1135.
3. F. Monay and D. Gatica-Perez, "Modeling Semantic Aspects for Cross-Media Image Indexing," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, 2007, pp. 1802-1817.
4. C.Wang, D. Blei, and F.-F. Li, "Simultaneous Image Classification and Annotation," Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 1903-1910.
5. C.D. Manning and H. Schutze, Foundations of Statistical Natural Language Processing, MIT Press, 1999.
6. S.-C. Zhu and D. Mumford, "A Stochastic Grammar of Images," J. Foundations and Trends in Computer Graphics and Vision, vol. 2, no. 4, 2006, pp. 259-362.
7. Y. Zhao and S.-C. Zhu, "Image Parsing with Stochastic Scene Grammar," Proc. 24th Ann. Advances in Neural Information Processing Systems (NIPS), 2011, pp. 73-81.
8. Y. Zhao and S.-C. Zhu, "Scene Parsing by Integrating Function, Geometry and Appearance Models," Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), 2013, pp. 3119-3126.
9. M. Pei, Y. Jia, and S.-C. Zhu, "Parsing Video Events with Goal Inference and Intent Prediction," Proc. 2011 IEEE Int'l Conf. Computer Vision (ICCV), 2011, pp. 487-494.
10. A. Fire and S.-C. Zhu, "Using Causal Induction in Humans to Learn and Infer Causality from Video," Proc. Ann. Meeting of the Cognitive Science Soc. (CogSci), 2013, pp. 2297-2309; http://mindmodeling.org/cogsci2013/ papers/0419/paper0419.pdf.
11. T.R. Gruber, "A Translation Approach to Portable Ontology Specifications," Knowledge Acquisition, vol. 5, no. 2, 1993, pp. 199-220.
12. J.H. Gennari et al., "The Evolution of Prot-eg-e: An Environment for Knowledge-Based Systems Development," Int'l J. Human-Computer Studies, vol. 58, no. 1, 2003, pp. 89-123.
13. O. Corcho et al., "Webode: An IntegratedWorkbench for Ontology Representation, Reasoning, and Exchange," Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web (EKAW), LNCS 2473, Springer, 2002, pp. 138-153.
14. L. Zhang et al., "Orient: Integrate Ontology Engineering into Industry Tooling Environment," The Semantic Web-ISWC, LNCS 3298, Springer, 2004, pp. 823-837.
15. D. Foxvog, "Cyc," Theory and Applications of Ontology: Computer Applications, Springer, 2010, pp. 259-278.
16. A. Maedche and S. Staab, "Ontology Learning for the SemanticWeb," IEEE Intelligent Systems, vol. 16, no. 2, 2001, pp. 72-79.
17. P. Buitelaar and B. Magnini, "Ontology Learning from Text: An Overview," Ontology Learning from Text: Methods, Applications and Evaluation, P. Buitelaar, P. Cimiano, and B. Magnini, eds., IOS Press, 2005, pp. 3-12.
18. P. Cimiano, Ontology Learning and Population from Text: Algorithms, Evaluation and Applications, Springer, 2006.
19. E.T. Mueller, Commonsense Reasoning, Morgan Kaufmann, 2006.
20. S.J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 2nd ed., Prentice Hall, 2002.
21. Z. Si and S. Zhu, "Learning AND-OR Templates for Object Modeling and Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 35, no. 9, 2012, pp. 2189-2205.
22. Z. Si et al., "Unsupervised Learning of Event AND-OR Grammar and Semantics from Video," Proc. 2011 IEEE Int'l Conf. Computer Vision (ICCV), 2011, pp. 41-48.
23. J. Earley, "An Efficient Context-Free Parsing Algorithm," Comm. ACM, vol. 26, no. 1, 1983, pp. 57-61.
24. J.R. Finkel, T. Grenager, and C. Manning, "Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling," Proc. 43rd Ann. Meeting Assoc. Computational Linguistics (ACL), 2005, pp. 363-370.
25. M.-C. de Marneffe and C.D. Manning, "Stanford Typed Dependencies Manual," 2008; http://nlp. stanford.edu/software/dependencies-manual.pdf.
26. A. Hakeem et al., "Semantic Video Search Using Natural Language Queries," Proc. 17th ACM Int'l Conf. Multimedia (MM), 2009, pp. 605-608.
27. W.-N. Lee et al., "Comparison of Ontology-Based Semantic-Similarity Measures," AMIA Ann. Symp. Proc., vol. 2008, 2008, pp. 384-388.
28. R. Thiagarajan, G. Manjunath, and M. Stumptner, "Computing Semantic Similarity Using Ontologies," HP Labs, tech. report, 2008.
29. C. Pesquita et al., "Semantic Similarity in Biomedical Ontologies," PLoS Computational Biology, vol. 5, no. 7, 2009, doi:10.1371/journal.pcbi.1000443.
30. J. Dodge et al., "Detecting Visual Text," Proc. 2012 Conf. North Am. Chapter of the Assoc. Computational Linguistics: Human Language Technologies (NAACL), 2012, pp. 762-772.
31. B. Yao et al., "I2T: Image Parsing to Text Description," Proc. IEEE, vol. 98, no. 8, 2010, pp. 1485-1508.
32. I. Langkilde and K. Knight, "Generation That Exploits Corpus-Based Statistical Knowledge," Proc. 17th Int'l Conf. Computational Linguistics (ACL-COLING), vol. 1, 1998, pp. 704-710.
33. C. Pollard and I.A. Sag, Head-Driven Phrase Structure Grammar, Univ. of Chicago Press, 1994.
34. K. Papineni et al., "Bleu: A Method for Automatic Evaluation of Machine Translation," Proc. 40th Ann. Meeting Assoc. Computational Linguistics (ACL), 2002, pp. 311-318.
35. S. Banerjee and A. Lavie, "Meteor: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments," Proc. ACLWorkshop Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005, pp. 65-72.