Deep Fusion of Multiple Semantic Cues for Complex Event Recognition

IEEE Transactions on Image Processing

Volumn 25, Issue 3, 2016, Pages 1033-1046

  Zhang, Xishan ^a,b   Zhang, Hanwang ^c   Zhang, Yongdong ^a   Yang, Yang ^d   Wang, Meng ^e   Luan, Huanbo ^f   Li, Jintao ^a   Chua, Tat Seng ^c  

^a INSTITUTE OF COMPUTING TECHNOLOGY   (China)

^b UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

^c NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^d UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA   (China)

^e HEFEI UNIVERSITY OF TECHNOLOGY   (China)

^f TSINGHUA UNIVERSITY   (China)

Author keywords

deep learning; fusion; multimedia event recognition

Indexed keywords

FUSION REACTIONS; LEARNING SYSTEMS;

COMPLEX EVENTS; CROSS MODALITY; DEEP ARCHITECTURES; DEEP LEARNING; EVENT RECOGNITION; FEATURE ABSTRACTION; FEATURE FUSION METHOD; SEMANTIC FEATURES;

SEMANTICS;

EID: 84962692397     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2015.2511585     Document Type: Article

References (48)

1. A. Gaidon, Z. Harchaoui, and C. Schmid, "Actom sequence models for efficient action detection, " in Proc. CVPR, 2011, pp. 3201-3208.
2. J. Liu et al., "Video event recognition using concept attributes, " in Proc. WACV, 2013, pp. 339-346.
3. J. Chen, Y. Cui, G. Ye, D. Liu, and S.-F. Chang, "Event-driven semantic concept discovery by exploiting weakly tagged Internet images, " in Proc. ICMR, 2014, p. 1.
4. C. Sun and R. Nevatia, "ACTIVE: Activity concept transitions in video event classification, " in Proc. ICCV, 2013, pp. 913-920.
5. C. Sun et al., "ISOMER: Informative segment observations for multimedia event recounting, " in Proc. ICMR, 2014, p. 241.
6. C. H. Papadimitriou, H. Tamaki, P. Raghavan, and S. Vempala, "Latent semantic indexing: A probabilistic analysis, " in Proc. 17th ACM SIGACT-SIGMOD-SIGART Symp. Principles Database Syst., 1998, pp. 159-168.
7. R. Salakhutdinov and G. Hinton, "Semantic hashing, " Int. J. Approx. Reasoning, vol. 50, no. 7, pp. 969-978, Jul. 2009.
8. L.-J. Li, H. Su, L. Fei-Fei, and E. P. Xing, "Object bank: A highlevel image representation for scene classification & semantic feature sparsification, " in Proc. NIPS, 2010, pp. 1378-1386.
9. S. Sadanand and J. J. Corso, "Action bank: A high-level representation of activity in video, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2012, pp. 1234-1241.
10. H. Hotelling, "Relations between two sets of variates, " Biometrika, vol. 28, nos. 3-4, pp. 321-377, 1936.
11. J. D. R. Farquhar, D. R. Hardoon, H. Meng, J. S. Shawe-Taylor, and S. Szedmak, "Two view learning: SVM-2K, theory and practice, " in Proc. Adv. Neural Inf. Process. Syst., 2005, pp. 355-362.
12. Y. Bengio, "Learning deep architectures for AI, " Found. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, 2009.
13. A. Tamrakar et al., "Evaluation of low-level features and their combinations for complex event detection in open source videos, " in Proc. CVPR, 2012, pp. 3681-3688.
14. N. Ikizler-Cinbis and S. Sclaroff, "Object, scene and actions: Combining multiple features for human action recognition, " in Proc. ECCV, 2010, pp. 494-507.
15. D. Oneata, J. Verbeek, and C. Schmid, "Action and event recognition with fisher vectors on a compact feature set, " in Proc. ICCV, 2013, pp. 1817-1824.
16. C. Sun and R. Nevatia, "Large-scale Web video event classification by use of fisher vectors, " in Proc. WACV, 2013, pp. 15-22.
17. P. Natarajan et al., "Multimodal feature fusion for robust event detection in Web videos, " in Proc. CVPR, 2012, pp. 1298-1305.
18. E. Younessian, T. Mitamura, and A. Hauptmann, "Multimodal knowledge-based analysis in multimedia event detection, " in Proc. ICMR, 2012, Art. ID 51.
19. Y. Yang and M. Shah, "Complex events detection using data-driven concepts, " in Proc. ECCV, 2012, pp. 722-735.
20. S. Wu, S. Bondugula, F. Luisier, X. Zhuang, and P. Natarajan, "Zeroshot event detection using multi-modal fusion of weakly supervised concepts, " in Proc. CVPR, 2014, pp. 2665-2672.
21. L. Jiang, A. G. Hauptmann, and G. Xiang, "Leveraging high-level and low-level features for multimedia event detection, " in Proc. MM, 2012, pp. 449-458.
22. H. Izadinia and M. Shah, "Recognizing complex events using large margin joint low-level event model, " in Proc. ECCV, 2012, pp. 430-444.
23. L. Torresani, M. Szummer, and A. Fitzgibbon, "Efficient object category recognition using classemes, " in Proc. ECCV, 2010, pp. 776-789.
24. M. Mazloom, E. Gavves, K. E. A. van de Sande, and C. G. M. Snoek, "Searching informative concept banks for video event detection, " in Proc. 3rd ACM Conf. Int. Conf. Multimedia Retr., 2013, pp. 255-262.
25. G. K. Myers et al., "Evaluating multimedia features and fusion for example-based event detection, " Mach. Vis. Appl., vol. 25, no. 1, pp. 17-32, 2014.
26. C. G. Snoek, M. Worring, and A. W. M. Smeulders, "Early versus late fusion in semantic video analysis, " in Proc. MM, 2005, pp. 399-402.
27. D. Liu, K.-T. Lai, G. Ye, M.-S. Chen, and S.-F. Chang, "Sample-specific late fusion for visual category recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2013, pp. 803-810.
28. G. Ye, D. Liu, I.-H. Jhuo, and S.-F. Chang, "Robust late fusion with rank minimization, " in Proc. CVPR, 2012, pp. 3021-3028.
29. Y. Yang, J. Song, Z. Huang, Z. Ma, N. Sebe, and A. G. Hauptmann, "Multi-feature fusion via hierarchical regression for multimedia analysis, " IEEE Trans. Multimedia, vol. 15, no. 3, pp. 572-581, Apr. 2013.
30. Z. Wu, Y.-G. Jiang, J. Wang, J. Pu, and X. Xue, "Exploring interfeature and inter-class relationships with deep neural networks for video classification, " in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 167-176.
31. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep learning, " in Proc. ICML, 2011, pp. 1-8.
32. N. Srivastava and R. Salakhutdinov, "Multimodal learning with deep Boltzmann machines, " in Proc. NIPS, 2012, pp. 2949-2980.
33. Y. Tang. (2013). "Deep learning using support vector machines. " [Online]. Available: http: //arxiv. org/abs/1306. 0239
34. C.-W. Hsu and C.-J. Lin, "A comparison of methods for multiclass support vector machines, " IEEE Trans. Neural Netw., vol. 13, no. 2, pp. 415-425, Mar. 2002.
35. A. Over et al., "TRECVID 2010-An overview of the goals, tasks, data, evaluation mechanisms, and metrics, " in Proc. TRECVid, 2011, pp. 10-12.
36. Y.-G. Jiang, "SUPER: Towards real-time event recognition in Internet videos, " in Proc. ICMR, 2012, Art. ID 7.
37. V. Ramanathan, P. Liang, and L. Fei-Fei, "Video event understanding using natural language descriptions, " in Proc. ICCV, 2013, pp. 905-912.
38. J. Donahue et al. (2013). "DeCAF: A deep convolutional activation feature for generic visual recognition. " [Online]. Available: http: //arxiv. org/abs/1310. 1531
39. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "SUN database: Large-scale scene recognition from abbey to zoo, " in Proc. CVPR, 2010, pp. 3485-3492.
40. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 1. Jun. 2005, pp. 886-893.
41. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope, " Int. J. Comput. Vis., vol. 42, no. 3, pp. 145-175, 2001.
42. T. Ojala, M. Pietikäinen, and T. Mäenpää, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002.
43. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2. Jun. 2006, pp. 2169-2178.
44. H. Wang and C. Schmid, "Action recognition with improved trajectories, " in Proc. ICCV, 2013, pp. 3551-3558.
45. I. Laptev, "On space-time interest points, " Int. J. Comput. Vis., vol. 64, no. 2, pp. 107-123, 2005.
46. T. Tieleman, "Training restricted boltzmann machines using approximations to the likelihood gradient, " in Proc. ICML, 2008.
47. Z. Xu, Y. Yang, I. Tsang, N. Sebe, and A. G. Hauptmann, "Feature weighting via optimal thresholding for video analysis, " in Proc. ICCV, 2013, pp. 3440-3447.
48. I.-H. Jhuo et al., "Discovering joint audio-visual codewords for video event detection, " Mach. Vis. Appl., vol. 25, no. 1, pp. 33-47, 2014.