Modeling spatial-Temporal clues in a hybrid deep learning framework for video classification

MM 2015 - Proceedings of the 2015 ACM Multimedia Conference

Volumn , Issue , 2015, Pages 461-470

  Wu, Zuxuan ^a   Wang, Xi ^a   Jiang, Yu Gang ^a   Ye, Hao ^a   Xue, Xiangyang ^a  

^a FUDAN UNIVERSITY   (China)

Author keywords

CNN; Deep Learning; Fusion.; LSTM; Video Classification

Indexed keywords

BENCHMARKING; FUSION REACTIONS; IMAGE CLASSIFICATION; MOTION ESTIMATION; NEURAL NETWORKS; SEMANTICS; VIDEO STREAMING;

COMPETITIVE PERFORMANCE; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; LONG SHORT TERM MEMORY; LSTM; SPATIAL INFORMATIONS; SPATIAL TEMPORALS; VIDEO CLASSIFICATION;

CLASSIFICATION (OF INFORMATION);

EID: 84962921420     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2733373.2806222     Document Type: Conference Paper

References (54)

1. R. Arandjelovic and A. Zisserman. Three things everyone should know to improve object retrieval. In CVPR, 2012.
2. F. R. Bach, G. R. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the smo algorithm. In ICML, 2004.
3. Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade. Springer, 2012.
4. J. Chung, C. Gülcehre, K. Cho, and Y. Bengio. Gated feedback recurrent neural networks. CoRR, 2015.
5. G. E. Dahl, D. Yu, L. Deng, and A. Acero. Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition. IEEE TASLP, 2012.
6. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-Term recurrent convolutional networks for visual recognition and description. CoRR, 2014.
7. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
8. A. Graves, A. Mohamed, and G. E. Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, 2013.
9. A. Graves and J. Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 2005.
10. C. Harris and M. J. Stephens. A combined corner and edge detector. In Alvey Vision Conference, 1988.
11. M. Jain, J. van Gemert, and C. G. M. Snoek. University of amsterdam at thumos challenge 2014. In ECCV THUMOS Challenge Workshop, 2014.
12. I.-H. Jhuo, G. Ye, S. Gao, D. Liu, Y.-G. Jiang, D. T. Lee, and S.-F. Chang. Discovering joint audio-visual codewords for video event detection. Machine Vision and Applications, 2014.
13. S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neural networks for human action recognition. In ICML, 2010.
14. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM Multimedia, 2014.
15. W. Jiang, C. Cotton, S.-F. Chang, D. Ellis, and A. Loui. Short-Term audio-visual atoms for generic video concept classification. In ACM Multimedia, 2009.
16. Y.-G. Jiang, S. Bhattacharya, S.-F. Chang, and M. Shah. High-level event recognition in unconstrained videos. IJMIR, 2013.
17. Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Action recognition with a large number of classes. http://crcv.ucf.edu/THUMOS14/, 2014.
18. Y.-G. Jiang, G. Ye, S.-F. Chang, D. Ellis, and A. C. Loui. Consumer video understanding: A benchmark database and an evaluation of human and machine performance. In ICMR, 2011.
19. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
20. A. Klaser, M. Marsza lek, and C. Schmid. A spatio-Temporal descriptor based on 3d-gradients. In BMVC, 2008.
21. M. Kloft, U. Brefeld, S. Sonnenburg, and A. Zien. Lp-norm multiple kernel learning. The Journal of Machine Learning Research, 2011.
22. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
23. Z. Lan, M. Lin, X. Li, A. G. Hauptmann, and B. Raj. Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition. CoRR, 2014.
24. I. Laptev. On space-Time interest points. IJCV, 2007.
25. W. Li, Z. Zhang, and Z. Liu. Expandable data-driven graphical modeling of human actions based on salient postures. IEEE TCSVT, 2008.
26. D. Liu, K.-T. Lai, G. Ye, M.-S. Chen, and S.-F. Chang. Sample-specific late fusion for visual category recognition. In CVPR, 2013.
27. D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
28. A. J. Ma and P. C. Yuen. Reduced analytic dependency modeling: Robust fusion for visual recognition. IJCV, 2014.
29. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Ng. Multimodal deep learning. In ICML, 2011.
30. D. Oneata, J. Verbeek, C. Schmid, et al. Action and event recognition with fisher vectors on a compact feature set. In ICCV, 2013.
31. M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling: A baseline for generative models of natural videos. CoRR, 2014.
32. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-The-shelf: An astounding baseline for recognition. CoRR, 2014.
33. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.
34. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 2014.
35. K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, 2012.
36. N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using LSTMs. CoRR, 2015.
37. N. Srivastava and R. Salakhutdinov. Multimodal learning with deep boltzmann machines. In NIPS, 2012.
38. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going Deeper with Convolutions. CoRR, 2014.
39. K. Tang, L. Fei-Fei, and D. Koller. Learning latent temporal structure for complex event detection. In CVPR, 2012.
40. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. C3d: Generic features for video analysis. CoRR, 2014.
41. D. L. Vail, M. M. Veloso, and J. D. Lafferty. Conditional random fields for activity recognition. In AAAMS, 2007.
42. A. Vedaldi, V. Gulshan, M. Varma, and A. Zisserman. Multiple kernels for object detection. In ICCV, 2009.
43. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. J. Mooney, and K. Saenko. Translating videos to natural language using deep recurrent neural networks. CoRR, 2014.
44. H. Wang and C. Schmid. Action recognition with improved trajectories. In ICCV, 2013.
45. H. Wang, M. M. Ullah, A. Klaser, I. Laptev, and C. Schmid. Evaluation of local spatio-Temporal features for action recognition. In BMVC, 2009.
46. Y. Wang and G. Mori. Max-margin hidden conditional random fields for human action recognition. In CVPR, 2009.
47. Z. Wu, Y.-G. Jiang, J. Wang, J. Pu, and X. Xue. Exploring inter-feature and inter-class relationships with deep neural networks for video classification. In ACM Multimedia, 2014.
48. Z. Xu, Y. Yang, I. Tsang, N. Sebe, and A. Hauptmann. Feature weighting via optimal thresholding for video analysis. In ICCV, 2013.
49. H. Yang, M. R. Lyu, and I. King. Efficient online learning for multitask feature selection. ACM SIGKDD, 2013.
50. G. Ye, D. Liu, I.-H. Jhuo, and S.-F. Chang. Robust late fusion with rank minimization. In CVPR, 2012.
51. H. Ye, Z. Wu, R.-W. Zhao, X. Wang, Y.-G. Jiang, and X. Xue. Evaluating two-stream cnn for video classification. In ICMR, 2015.
52. Z. Zeng and Q. Ji. Knowledge based activity recognition with dynamic Bayesian network. In ECCV, 2010.
53. S. Zha, F. Luisier, W. Andrews, N. Srivastava, and R. Salakhutdinov. Exploiting image-Trained cnn architectures for unconstrained video classification. CoRR, 2015.
54. J. Zhang, M. Marsza lek, S. Lazebnik, and C. Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. IJCV, 2007.