Semantic model vectors for complex video event recognition

IEEE Transactions on Multimedia

Volumn 14, Issue 1, 2012, Pages 88-101

  Merler, Michele ^a   Huang, Bert ^b   Xie, Lexing ^c   Hua, Gang ^d   Natsev, Apostol ^e  

^a Fu Foundation School of Engineering and Applied Science   (United States)

^b University of Maryland   (United States)

^c AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^d Stevens Institute of Technology   (United States)

^e IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Complex video events; event recognition; high level descriptor

Indexed keywords

CLIP LEVEL; DESCRIPTORS; EMPIRICAL STUDIES; EVENT DETECTION; EVENT RECOGNITION; FEATURE FUSION; INTERMEDIATE LEVEL; SEMANTIC MODEL; SEMANTIC REPRESENTATION; SPATIO-TEMPORAL; STATIC FEATURES; SVM MODEL; TRECVID; VIDEO EVENTS; WEB IMAGES; YOUTUBE;

IMAGE RETRIEVAL; SEMANTICS; VECTORS;

SEMANTIC WEB;

EID: 84856141623     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2011.2168948     Document Type: Article

References (65)

1. "Cisco Visual Networking Index: Forecast and Methodology, 2009-2014," 2010. [Online]. Available: http://www.cisco.com/en/US/ solutions/collateral/ns341/ns525/ns537/ns705/ns827/whitepaper-c11-481360-ns827- Networking-Solutions-White Paper.html.
2. "Great Scott! Over 35 Hours of Video Uploaded Every Minute to Youtube," The official YouTube blog, 2010. [Online]. Available: http://youtube-global.blogspot.com/2010/11/great-scott-over-35-hours-of-video. html.
3. L. Ballan, M. Bertini, A. D. Bimbo, and G. Serra, "Semantic annotation of soccer videos by visual instance clustering and spatial/temporal reasoning in ontologies," Multimedia Tools Appl., pp. 313-337, Jun. 2010.
4. L. Ballan, M. Bertini, A. D. Bimbo, and G. Serra, "Video event classification using string Kernels," Multimedia Tools Appl., vol. 48, no. 1, pp. 69-87, May 2010.
5. O. Boiman and M. Irani, "Detecting irregularities in images and in video," Int. J. Comput. Vis. (IJCV), vol. 74, pp. 17-31, 2007. (Pubitemid 46673101)
6. F. Bremond, N. Maillot, M. Thonnat, and V.-T. Vu, "Ontologies for video events," Research Report Number 5189, 2004.
7. M. Campbell, A. Haubold, M. Liu, A. Natsev, J. R. Smith, J. Tesic, L. Xie, R. Yan, and J. Yang, "IBM research TRECVID-2007 video retrieval system," in Proc. NIST TRECVID Workshop, 2007.
8. P. Dollár, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recognition via sparse spatio-temporal features," in Proc. 2nd Joint IEEE Int. Workshop Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS), 2005, pp. 65-72. (Pubitemid 46178025)
9. L. Duan, D. Xu, W.-H. Tsang, and J. Luo, "Visual event recognition in videos by learning from web data," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
10. S. Ebadollahi, L. Xie, S. Chang, and J. Smith, "Visual event detection using multi-dimensional concept dynamics," in Proc. Int. Conf. Multimedia and Expo (ICME), 2006, pp. 881-884. (Pubitemid 46679858)
11. C. Galleguillos and S. Belongie, "Context based object categorization: A critical survey," Comput. Vis. Image Understand., 2010.
12. A. Gupta and L. Davis, "Objects in action: An approach for combining action understanding and object perception," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1-8.
13. A. Hakeem and M. Shah, "Learning, detection and representation of multi-agent events in videos," Artif. Intell., vol. 171, Jun. 2007.
14. Y. Hu, L. Cao, F. Lv, S. Yan, Y. Gong, and T. S. Huang, "Action detection in complex scenes with spatial and temporal ambiguities," in Proc. Int. Conf. Computer Vision (ICCV), 2009, pp. 128-135.
15. Y. Huang, K. Huang, T. Tan, and D. Tao, "A novel visual organization based on topological perception," in Proc. Asian Conf. Computer Vision, 2009, pp. 180-189.
16. Y. Huang, K. Huang, L. Wang, D. Tao, T. Tan, and X. Li, "Enhanced biologically inspired model," in Proc. Computer Vision and Pattern Recognition, 2008.
17. N. Ikizler-Cinbis, R. G. Cinbis, and S. Sclaroff, "Learning actions from the web," in Proc. Int. Conf. Computer Vision (ICCV), 2009.
18. N. Ikizler-Cinbis and S. Sclaroff, "Object, scene and actions: Combining multiple features for human action recognition," in Proc. Eur. Conf. Computer Vision (ECCV), 2010.
19. W. Jiang, C. Cotton, S.-F. Chang, D. Ellis, and A. C. Loui, "Short-term audio-visual atoms for generic video concept classification," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2009, pp. 5-14.
20. Y.-G. Jiang, G. Ye, S.-F. Chang, D. Ellis, and A. C. Loui, "Consumer video understanding: A benchmark database and an evaluation of human and machine performance," in Proc. ACM Int. Conf. Multimedia Retrieval (ICMR), 2011.
21. D. Kuettel, M. D. Breitenstein, L. V. Gool, and V. Ferrari, "What's going on? Discovering spatio-temporal dependencies in dynamic scenes," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
22. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-8.
23. I. Laptev, "On space-time interest points," Int. J. Comput. Vis., vol. 64, no. 2, pp. 107-123, 2005. (Pubitemid 41314645)
24. L. Li and L. Fei-Fei, "What, where and who? classifying events by scene and object recognition," in Proc. Int. Conf. in Computer Vision (ICCV), 2007, pp. 1-8.
25. W. Li, Z. Zhang, and Z. Liu, "Action recognition based on a bag of 3d points," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
26. J. Liu, J. Luo, and M. Shah, "Recognizing realistic actions from videos in the wild," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 1996-2003.
27. A. C. Loui, J. Luo, S.-F. Chang, D. Ellis, W. Jiang, L. Kennedy, K. Lee, and A. Yanagawa, "Kodak's consumer video benchmark data set: Concept definition and annotation," in Proc. ACM SIGMM Int. Workshop Multimedia Information Retrieval, Sep. 2007.
28. D. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
29. M. Marszalek, I. Laptev, and C. Schmid, "Actions in context," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 2929-2936.
30. A. Natsev, M. R. Naphade, and J. R. Smith, "Semantic representation, search and mining of multimedia content," in Proc. ACM Int. Conf. Knowledge Discovery and Data Mining (KDD'04), Seattle, WA, Aug. 2004, pp. 641-646. (Pubitemid 40114984)
31. A. Natsev, M. Hill, J. R. Smith, L. Xie, Rong, Y. S. Bao, M. Merler, and Y. Zhang, "IBM research TRECVID-2009 video retrieval system," in Proc. NIST TRECVID Workshop, 2009.
32. J. C. Niebles, B. Han, A. Ferencz, and L. Fei-Fei, "Extracting moving people from internet videos," in Proc. Eur. Conf. Computer Vision (ECCV), 2008, pp. 527-540.
33. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," Int. J. Comput. Vis., vol. 42, no. 3, pp. 145-175, 2001. (Pubitemid 32680801)
34. K. Prabhakar, S. Oh, P. Wang, G. D. Abowd, and J. M. Rehg, "Temporal causality for the analysis of visual events," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
35. A. Quattoni and A. Torralba, "Recognizing indoor scenes," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 413-420.
36. P. M. Roth and M. Winter, Survey of Appearance-Based Methods for Object Recognition, 2008, Tech/ Rep. ICG-TR-01/08.
37. R. W. Poppe, "A survey on vision-based human action recognition," Image Vis. Comput., vol. 28, pp. 976-990, Jun. 2010.
38. M. Ryoo and J. Aggarwal, "Hierarchical recognition of human activities interacting with objects," in Proc. 2nd Int. Workshop Semantic Learning Applications in Multimedia (SLAM) at CVPR, 2007, pp. 1-8.
39. M. Ryoo and J. Aggarwal, "Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities," in Proc. Int. Conf. Computer Vision (ICCV), 2009, pp. 1593-1600.
40. G. Schindler, L. Zitnick, and M. Brown, "Internet video category recognition," Internet Vis., pp. 1-7, 2008.
41. C. Schuldt, I. Laptev, and B. Caputo, "Recognizing human actions: A local SVM approach," in Proc. 17th Int. Conf. Pattern Recognition (ICPR'04), 2004, vol. 3, pp. 32-36.
42. S. Sen and J. Ma, "Contextualized eventdriven prediction with ontology-based similarity," in Proc. AAAI Spring Symp. Intelligent Event Processing, 2009.
43. A. F. Smeaton, P. Over, and W. Kraaij, "High level feature detection from video in trecvid: A 5-year retrospective of achievements," in Proc. Multimedia Content Analysis, 2009, pp. 151-174.
44. J. Smith, M. Naphade, and A. Natsev, "Multimedia semantic indexing using model vectors," in Proc. IEEE Int. Conf. Multimedia Expo. (ICME), 2003, pp. 445-448.
45. D. Song and D. Tao, "Biologically inspired feature manifold for scene classification," IEEE Trans. Image Process., vol. 19, no. 1, pp. 174-184, Jan. 2010.
46. J. Sun, X. Wu, S. Yan, L.-F. Cheong, T.-S. Chua, and J. Li, "Hierarchical spatio-temporal context modeling for action recognition," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 2004-2011.
47. L. Torresani, M. Szummer, and A. Fitzgibbon, "Efficient object category recognition using classemes," in Proc. Eur. Conf. Computer Vision (ECCV), Sep. 2010, pp. 776-789.
48. K. E. A. van de Sande, T. Gevers, and C. G. M. Snoek, "Evaluating color descriptors for object and scene recognition," IEEE Trans. Pattern Anal. Mach. Intell., to be published.
49. J. C. van Gemert, C. G. M. Snoek, C. J. Veenman, A. W. M. Smeulders, and J.-M. Geusebroek, "Comparing compact codebooks for visual categorization," Computer Vis. Image Understand., to be published.
50. F. Wang, Y. Jiang, and C. Ngo, "Video event detection using motion relativity and visual relatedness," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2008, pp. 239-248.
51. H. Wang, M. M. Ullah, A. Klaser, I. Laptev, and C. Schmid, "Evaluation of local spatio-temporal features for action recognition," in Proc. British Machine Vision Conf. (BMVC), 2009.
52. J.Wang, L. Duan, L. Xu, H. Lu, and J. S. Jin, "Tv ad video categorization with probabilistic latent concept learning," in Proc. Multimedia Information Retrieval, 2007, pp. 217-226.
53. T.Wang, J. Li, Q. Diao, W. Hu, Y. Zhang, and Dulong, "Semantic event detection using conditional random fields," in Proc. Computer Vision and Pattern Recognition Workshop (CVPR), 2006, pp. 109-109.
54. G. Willems, T. Tuytelaars, and L. J. V. Gool, "An efficient dense and scale-invariant saptio-temporal interest point detector," in Proc. Eur. Conf. Computer Vision (ECCV), 2008, pp. 650-663.
55. J.Wu, A. Osuntogun, T. Choudhury, M. Philipose, and J. M. Rehg, "A scalable approach to activity recognition based on object use," in Proc. Int. Conf. Computer Vision (ICCV), 2007, pp. 1-8.
56. J. Xiao, J. Hays, K. Ehinger, A. Oliva, and A. Torralba, "SUN database: Large-scale scene recognition from abbey to zoo," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
57. D. Xu and S.-F. Chang, "Video event recognition using Kernel methods with multilevel temporal alignment," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 11, pp. 1985-1997, Nov. 2008.
58. R. Yan, J. Tesic, and J. R. Smith, "Model-shared subspace boosting for multi-label classification," in Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), 2007, pp. 834-843. (Pubitemid 350229268)
59. A. Yanagawa, S.-F. Chang, L. Kennedy, and W. Hsu, Columbia University's Baseline Detectors for 374 Lscom Semantic Visual Concepts, Columbia Univ., 2007, Tech. Rep.
60. B.Yao and L. Fei-Fei, "Grouplet:A structured image representation for recognizing human and object interactions," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
61. J. Yin, D. H. Hu, and Q. Yang, "Spatio-temporal event detection using dynamic conditional random fields," in Proc. 21st Int. Joint Conf. Artificial Intelligence (IJCAI), 2009, pp. 1321-1326.
62. L. Zelnik-Manor and M. Irani, "Statistical analysis of dynamic actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, pp. 1530-1535, 2006. (Pubitemid 44132659)
63. D. Zhang, D. Gatica-Perez, S. Bengio, and I. McCowan, "Semi-supervised adapted HMMs for unusual event detection," in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2005, pp. 611-618. (Pubitemid 43897250)
64. X. Zhou, X. Zhuang, S. Yan, S. F. Chang, M. H. Johnson, and T. S. Huang, "Sift-bag Kernel for video event analysis," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2008, pp. 229-238.
65. G. Zhu, M. Yang, K. Yu, W. Xu, "Detecting video events based on action recognition in complex scenes using spatio-temporal descriptors," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2009, pp. 165-174.