3D-based Deep Convolutional Neural Network for action recognition with depth sequences

Image and Vision Computing

Volumn 55, Issue , 2016, Pages 93-100

  Liu, Zhi ^a   Zhang, Chenyang ^b   Tian, Yingli ^b  

^a CHONGQING UNIVERSITY OF TECHNOLOGY   (China)

^b City College of New York   (United States)

Author keywords

3D convolution; Action recognition; Convolutional neural network; Deep learning; Depth sequences

Indexed keywords

ALGORITHMS; CONVOLUTION; IMAGE RETRIEVAL; MOTION ESTIMATION; MUSCULOSKELETAL SYSTEM; NEURAL NETWORKS; SUPPORT VECTOR MACHINES;

ACTION RECOGNITION; CLASSIFICATION ACCURACY; CLASSIFICATION RESULTS; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; DEPTH SEQUENCES; SPATIO TEMPORAL FEATURES; STATE-OF-THE-ART METHODS;

CLASSIFICATION (OF INFORMATION);

EID: 84965007465     PISSN: 02628856     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.imavis.2016.04.004     Document Type: Article

References (53)

1. [1] Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A., Real-time human pose recognition in parts from single depth images. Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 2011, 1297–1304.
2. [2] Yang, X., Tian, Y., Super normal vector for activity recognition using depth sequences. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 2014, 804–811.
3. [3] Peng, X., Qiao, Y., Peng, Q., Motion boundary based sampling and 3D co-occurrence descriptors for action recognition. Image Vis. Comput. 32:9 (2014), 616–628.
4. [4] Roshtkhari, M.J., Levine, M.D., Human activity recognition in videos using a single example. Image Vis. Comput. 31:11 (2013), 864–876.
5. [5] O'Hara, S., Lui, Y.M., Draper, Bruce A., Using a product manifold distance for unsupervised action recognition. Image Vis. Comput. 30:3 (2012), 206–216.
6. [6] Yang, X., Zhang, C., Tian, Y., Recognizing actions using depth motion maps-based histograms of oriented gradients. Proceedings of the 20th ACM International Conference on Multimedia, MM ’12, 2012, ACM, 1057–1060 New York, NY, USA.
7. [7] Oreifej, O., Liu, Z., HON4D: histogram of oriented 4D normals for activity recognition from depth sequences. Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, 2013, 716–723.
8. [8] Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B., Learning realistic human actions from movies. Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2008, 1–8.
9. [9] Davis, J.W., Bobick, A.F., The representation and recognition of human movement using temporal templates. Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on, IEEE, 1997, 928–934.
10. [10] Wang, J., Liu, Z., Wu, Y., Yuan, J., Mining actionlet ensemble for action recognition with depth cameras. Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2012, 1290–1297.
11. [11] Vemulapalli, Arrate, F., Chellappa, R., Human action recognition by representing 3D skeletons as points in a lie group. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 2014, 588–595.
12. [12] Li, W., Zhang, Z., Liu, Z., Action recognition based on a bag of 3D points. Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, 2010, 9–14.
13. [13] Hinton, G., Osindero, S., Teh, Y.-W., A fast learning algorithm for deep belief nets. Neural Comput. 18:7 (2006), 1527–1554.
14. [14] Shuiwang Ji, V., Xu, W., Yang, M., Yu, K., 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35:1 (2013), 221–231.
15. [15] Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y., Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 2011, 3361–3368.
16. [16] Lin Sun, S., Jia, K., Chan, T.-H., Fang, Y., Wang, G., Yan, S., DL-SFA: deeply-learned slow feature analysis for action recognition. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 2014, 2625–2632.
17. [17] Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 1097–1105.
18. [18] Ye, M., Zhang, Q., Wang, L., Zhu, J., Yang, R., Gall, J., A survey on human motion analysis from depth data. Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, 2013, Springer, 149–187.
19. [19] Valle, A., Starostenko, O., Recognition of human walking/running actions based on neural network. Electrical Engineering, Computing Science and Automatic Control (CCE), 2013 10th International Conference on, 2013, 239–244.
20. [20] Xia, L., Chen, C.-C., Aggarwal, J.K., View invariant human action recognition using histograms of 3D joints. Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on, 2012, 20–27.
21. [21] Lowe, D.G., Object recognition from local scale-invariant features. Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, vol. 2, 1999, 1150–1157 Ieee.
22. [22] Zhu, Y., Chen, W., Guo, G., Evaluating spatiotemporal interest point features for depth-based action recognition. Image Vis. Comput. 32:8 (2014), 453–464.
23. [23] Thi, T.H., Cheng, L., Zhang, J., Wang, L., Satoh, S., Structured learning of local features for human action classification and localization. Image Vis. Comput. 30:1 (2012), 1–14.
24. [24] Bay, H., Ess, A., Tuytelaars, T., Van Gool, L., Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110:3 (2008), 346–359.
25. [25] Wang, Heng., Schmid, Cordelia., Action recognition with improved trajectories. Computer Vision (ICCV), 2013 IEEE International Conference on, IEEE, 2013, 3551–3558.
26. [26] Shao, L., Liu, L., Yu, M., Kernelized multiview projection for robust action recognition. Int. J. Comput. Vis., 2015, 1–15.
27. [27] Han, J., Shao, L., Xu, D., Shotton, J., Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE Trans. Cybern. 43:5 (2013), 1318–1334.
28. [28] Zanfir, M., Leordeanu, M., Sminchisescu, C., The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection. Computer Vision (ICCV), 2013 IEEE International Conference on, 2013, 2752–2759.
29. [29] Ni, B., Pei, Y., Moulin, P., Yan, S., Multilevel depth and image fusion for human activity detection. IEEE Trans. Cybern. 43:5 (2013), 1383–1394.
30. [30] Ye, G., Liu, Y., Deng, Y., Hasler, N., Ji, X., Dai, Q., Theobalt, Christian., Free-viewpoint video of human actors using multiple handheld kinects. IEEE Trans. Cybern. 43:5 (2013), 1370–1382.
31. [31] M. Yu, L. Liu, L. Shao, Structure-preserving binary representations for RGB-D action recognition, IEEE Trans. Pattern Anal. Mach. Intell.
32. [32] Murray, R.M., Li, Z., Sastry, S.S., Sastry, S.S., A Mathematical Introduction to Robotic Manipulation. 1994, CRC Press.
33. [33] Zhang, C., Tian, Y., Edge enhanced depth motion map for dynamic hand gesture recognition. Computer Vision and Pattern Recognition Workshops (CVPRW), 2013 IEEE Conference on, 2013, 500–505.
34. [34] Xia, L., Aggarwal, J.K., Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, 2013, 2834–2841.
35. [35] Wang, J., Liu, Z., Chorowski, J., Chen, Z., Wu, Y., Robust 3d action recognition with random occupancy patterns. Computer Vision-ECCV 2012, 2012, Springer, 872–885.
36. [36] Klaser, A., Marszałek, M., Schmid, C., A spatio-temporal descriptor based on 3d-gradients. BMVC 2008-19th British Machine Vision Conference, 2008, British Machine Vision Association 275–1.
37. [37] LeCun, Y., Boser, B.., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., Backpropagation applied to handwritten zip code recognition. Neural Comput. 1:4 (1989), 541–551.
38. [38] Sermanet, Kavukcuoglu, K., Chintala, S., LeCun, Y., Pedestrian detection with unsupervised multi-stage feature learning. Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, 2013, 3626–3633.
39. [39] Burger, H.C., Schuler, C.J., Harmeling, S., Image denoising: can plain neural networks compete with BM3D?. Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2012, 2392–2399.
40. [40] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, T. Darrell, Decaf: a deep convolutional activation feature for generic visual recognition, arXiv preprint arXiv:1310.1531.
41. [41] Jin, L., Gao, S., Li, Z., Tang, J., Hand-crafted features or machine learnt features? Together they improve RGB-D object recognition. Multimedia (ISM), 2014 IEEE International Symposium on, IEEE, 2014, 311–319.
42. [42] Liu, L., Shao, L., Learning discriminative representations from RGB-D video data. Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, 2013, AAAI Press, 1493–1500.
43. [43] L. Liu, L. Shao, X. Li, K. Lu, Learning spatio-temporal representations for action recognition: a genetic programming approach, IEEE Trans. Cybern.
44. [44] Wu, D., Shao, L., Leveraging hierarchical parametric networks for skeletal joints based action segmentation and recognition. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE, 2014, 724–731.
45. [45] Molchanov, P., Gupta, S., Kim, K., Kautz, J., Hand gesture recognition with 3D convolutional neural networks. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Work., 2015.
46. [46] Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, Li., Large-scale video classification with convolutional neural networks. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 2014, 1725–1732.
47. [47] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M., Learning spatiotemporal features with 3D convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, 2015, 4489–4497.
48. [48] Hunter, D., Yu, H., Pukish, M.S. III, Kolbusz, J., Wilamowski, B.M., Selection of proper neural network sizes and architectures, a comparative study. IEEE Trans. Ind. Inf. 8:2 (2012), 228–240.
49. [49] Collobert, R., Kavukcuoglu, K., Farabet, C., Torch7: A Matlab-like environment for machine learning. BigLearn, NIPS Workshop, 2011.
50. [50] Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., Lin, C.-J., LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9 (2008), 1871–1874.
51. [51] Devanne, M., Wannous, H., Berretti, S., Pala, P., Daoudi, M., Del Bimbo, A., Space-time pose representation for 3D human action recognition. New Trends in Image Analysis and Processing-ICIAP 2013, 2013, Springer, 456–464.
52. [52] Chrungoo, A., Manimaran, S.S., Ravindran, B., Activity recognition for natural human robot interaction. Social Robotics, 2014, Springer, 84–94.
53. [53] Oquab, M., Bottou, L., Laptev, I., Sivic, J., Learning and transferring mid-level image representations using convolutional neural networks. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE, 2014, 1717–1724.