Human action recognition in RGB-D videos using motion sequence information and deep learning

Pattern Recognition

Volumn 72, Issue , 2017, Pages 504-516

  Ijjina, Earnest Paul ^a   Chalavadi, Krishna Mohan ^a  

^a INDIAN INSTITUTE OF TECHNOLOGY HYDERABAD   (India)

Author keywords

Deep learning; Extreme learning machines; Motion information; Multi modal action recognition

Indexed keywords

EDUCATION; IMAGE RECOGNITION; LEARNING SYSTEMS; NEURAL NETWORKS; VIDEO STREAMING;

ACTION RECOGNITION; CONVOLUTIONAL NEURAL NETWORK; DISCRIMINATIVE FEATURES; EXTREME LEARNING MACHINE; HUMAN ACTIONS; HUMAN-ACTION RECOGNITION; MOTION INFORMATION; MOTION SEQUENCES;

DEEP LEARNING;

EID: 85023621585     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2017.07.013     Document Type: Article

References (52)

1. Han, J., Shao, L., Xu, D., Shotton, J., Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans. Cybern. 43:5 (2013), 1318–1334.
2. Zhang, J., Li, W., Ogunbona, P.O., Wang, P., Tang, C., Rgb-d-based action recognition datasets: a survey. CoRR, abs/1601.05511, 2016.
3. Firman, M., RGBD datasets: past, present and future. CoRR, abs/1604.00999, 2016.
4. Shahroudy, A., Liu, J., Ng, T., Wang, G., Ntu rgb+d: a large scale dataset for 3d human activity analysis. CoRR, abs/1604.02808, 2016.
5. Chen, C., Jafari, R., Kehtarnavaz, N., Utd-mhad: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. IEEE International Conference on Image Processing (ICIP), 2015, 168–172, 10.1109/ICIP.2015.7350781.
6. Ni, B., Wang, G., Moulin, P., Rgbd-hudaact: a color-depth video database for human daily activity recognition. Fossati, A., Gall, J., Grabner, H., Ren, X., Konolige, K., (eds.) (Trans.) Consumer Depth Cameras for Computer Vision (Advances in Computer Vision and Pattern Recognition), 2013, Springer, 193–208.
7. Cheng, Z., Qin, L., Ye, Y., Huang, Q., Tian, Q., Human daily action analysis with multi-view and color-depth data. Fusiello, A., Murino, V., Cucchiara, R., (eds.) ECCV Workshops (2) Lecture Notes in Computer Science, 7584, 2012, Springer, 52–61.
8. Liu, Y., Qin, L., Cheng, Z., Zhang, Y., Zhang, W., Huang, Q., Da-ccd: A novel action representation by deep architecture of local depth feature. IEEE International Conference on Image Processing (ICIP), 2014, 833–837.
9. Sui, W., Wu, X., Feng, Y., Jia, Y., Heterogeneous discriminant analysis for cross-view action recognition. Neurocomputing 191 (2016), 286–295, 10.1016/j.neucom.2016.01.051.
10. Liu, A.-A., Xu, N., Su, Y.-T., Lin, H., Hao, T., Yang, Z.-X., Single/multi-view human action recognition via regularized multi-task learning. Neurocomputing 151 (2015), 544–553 Part 2.
11. Xu, C., Tao, D., Xu, C., A survey on multi-view learning. CoRR, abs/1304.5634, 2013.
12. Yun, K., Honorio, J., Chattopadhyay, D., Berg, T.L., Samaras, D., Two-person interaction detection using body-pose features and multiple instance learning. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012, 28–35.
13. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., Xie, X., Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. CoRR, abs/1603.07772, 2016.
14. Evangelidis, G., Singh, G., Horaud, R., Skeletal quads: Human action recognition using joint quadruples. International Conference on Pattern Recognition (ICPR), 2014, 4513–4518.
15. Oreifej, O., Liu, Z., Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, 716–723.
16. Wei Shen, K.D., Bai, X., Leyvand, T., Guo, B., Tu, Z., Exemplar-based human action pose correction and tagging. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, 1784–1791.
17. Wei, S., Ke, D., Xiang, B., Tommer, L., Baining, G., Zhuowen, T., Exemplar-based human action pose correction. IEEE Trans. Cybern. 44:7 (2014), 1053–1066.
18. Gupta, R., Chia, A.Y.-S., Rajan, D., Human activities recognition using depth images. ACM International Conference on Multimedia MM ’13, 2013, ACM, New York, NY, USA, 283–292.
19. Du, Y., Wang, W., Wang, L., Hierarchical recurrent neural network for skeleton based action recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 1110–1118.
20. Jiang, Z., Lin, Z., Davis, L.S., Label consistent k-svd: learning a discriminative dictionary for recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35:11 (2013), 2651–2664.
21. Lin, W., Chu, H., Wu, J., Sheng, B., Chen, Z., A heat-map-based algorithm for recognizing group activities in videos. IEEE Trans. Circuits Syst. Video Technol. 23:11 (2013), 1980–1992.
22. Prest, A., Ferrari, V., Schmid, C., Explicit modeling of human-object interactions in realistic videos. IEEE Trans. Pattern Anal. Mach. Intell. 35:4 (2013), 835–848.
23. Ji, S., Xu, W., Yang, M., Yu, K., 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35:1 (2013), 221–231.
24. Xia, L., Chen, C.-C., Aggarwal, J.K., View invariant human action recognition using histograms of 3d joints. Workshop on Human Activity Understanding from 3D Data in conjunction with CVPR (HAU3D), 2012, Rhode Island, USA, 20–27.
25. Megavannan, V., Agarwal, B., Venkatesh Babu, R., Human action recognition using depth maps. International Conference on Signal Processing and Communications (SPCOM), 2012, 1–5.
26. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324, 10.1109/5.726791.
27. Bengio, Y., Courville, A., Vincent, P., Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1798–1828, 10.1109/TPAMI.2013.50.
28. Yosinski, J., Clune, J., Bengio, Y., Lipson, H., How transferable are features in deep neural networks?. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K., (eds.) Advances in Neural Information Processing Systems (NIPS) 27, 2014, Curran Associates, Inc., 3320–3328.
29. Extreme learning machine: theory and applications. Neurocomputing 70:13 (2006), 489–501 Neural Networks Selected Papers from the 7th Brazilian Symposium on Neural Networks (SBRN ’04).
30. Foggia, P., Percannella, G., Saggese, A., Vento, M., Recognizing human actions by a bag of visual words. IEEE International Conference on Systems, Man, and Cybernetics, 2013, 2910–2915, 10.1109/SMC.2013.496.
31. Song, Y., Demirdjian, D., Davis, R., Tracking body and hands for gesture recognition: Natops aircraft handling signals database. IEEE International Conference on Automatic Face and Gesture Recognition (FG 2011), 2011, 500–506.
32. Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R., Actions as space-time shapes. IEEE International Conference on Computer Vision (ICCV), 2, 2005, 1395–1402, 10.1109/ICCV.2005.28.
33. V. Carletti, P. Foggia, G. Percannella, A. Saggese, M. Vento, Recognition of Human Actions from RGB-D Videos Using a Reject Option, Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 436–445. 10.1007/978-3-642-41190-8_47.
34. Brun, L., Percannella, G., Saggese, A., Vento, M., Hack: A system for the recognition of human actions by kernels of visual strings. IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2014, 142–147, 10.1109/AVSS.2014.6918658.
35. Foggia, P., Saggese, A., Strisciuglio, N., Vento, M., Exploiting the deep learning paradigm for recognizing human actions. IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2014, 93–98, 10.1109/AVSS.2014.6918650.
36. Brun, L., Foggia, P., Saggese, A., Vento, M., Recognition of human actions using edit distance on aclet strings. International Conference on Computer Vision Theory and Applications (VISAPP), 2015, 97–103.
37. Song, Y., Demirdjian, D., Davis, R., Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans. Interact. Intell. Syst. 2:1 (2012), 5:1–5:28.
38. Song, Y., Morency, L.-P., Davis, R., Multi-view latent variable discriminative models for action recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, 2120–2127.
39. Lin, L., Wang, K., Zuo, W., Wang, M., Luo, J., Zhang, L., A deep structured model with radius-margin bound for 3d human activity recognition. CoRR, abs/1512.01642, 2015.
40. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J., An end-to-end spatio-temporal attention model for human action recognition from skeleton data. CoRR, abs/1611.06067, 2016.
41. J. Liu, A. Shahroudy, D. Xu, G. Wang, Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition, Springer International Publishing, Cham, pp. 816–833. 10.1007/978-3-319-46487-9_50.
42. Ji, Y., Ye, G., Cheng, H., Interactive body part contrast mining for human interaction recognition. IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2014, 1–6.
43. Edwards, M., Xie, X., Generating local temporal poses from gestures with aligned cluster analysis for human action recognition. Tam, G.K.L., (eds.) UK Computer Vision Student Workshop (BMVW), 2015, BMVA Press, 1.1–1.12.
44. Huynh-The, T., Banos, O., Le, B.V., Bui, D.M., Lee, S., Yoon, Y., Le-Tien, T., Pam-based flexible generative topic model for 3d interactive activity recognition. International Conference on Advanced Technologies for Communications (ATC), 2015, 117–122, 10.1109/ATC.2015.7388302.
45. Ali, S., Basharat, A., Shah, M., Chaotic invariants for human action recognition. IEEE International Conference on Computer Vision (ICCV), 2007, 1–8, 10.1109/ICCV.2007.4409046.
46. Boiman, O., Irani, M., Similarity by composition. Schȵlkopf, B., Platt, J., Hoffman, T., (eds.) Advances in Neural Information Processing Systems (NIPS), 2006, MIT Press, Cambridge, MA, 177–184.
47. Crook, P., Kellokumpu, V., Zhao, G., Pietikainen, M., Human activity recognition using a dynamic texture based method. British Machine Vision Conference (BMVC), 2008, BMVA Press, 88.1–88.10.
48. Wang, Y., Mori, G., Human action recognition by semilatent topic models. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 31:10 (2009), 1762–1774.
49. Huang, G.B., Zhou, H., Ding, X., Zhang, R., Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern.Part B (Cybernetics) 42:2 (2012), 513–529, 10.1109/TSMCB.2011.2168604.
50. Maaten, L.v.d., Hinton, G., Visualizing data using t-sne. J. Mach. Learn. Res. 9:November (2008), 2579–2605.
51. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L., Large-scale video classification with convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, 1725–1732, 10.1109/CVPR.2014.223.
52. Feichtenhofer, C., Pinz, A., Zisserman, A., Convolutional two-stream network fusion for video action recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 1933–1941, 10.1109/CVPR.2016.213.