Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition

Pattern Recognition

Volumn 76, Issue , 2018, Pages 80-94

  Núñez, Juan C ^a   Cabido, Raúl ^a   Pantrigo, Juan J ^a   Montemayor, Antonio S ^a   Vélez, José F ^a  

^a UNIVERSIDAD REY JUAN CARLOS   (Spain)

Author keywords

Convolutional Neural Network; Deep learning; Hand gesture recognition; Human activity recognition; Long Short Term Memory; Real time; Recurrent neural network

Indexed keywords

BRAIN; CONVOLUTION; DEEP LEARNING; LONG SHORT-TERM MEMORY; MUSCULOSKELETAL SYSTEM; NEURAL NETWORKS; PALMPRINT RECOGNITION; RECURRENT NEURAL NETWORKS;

CONVOLUTIONAL NEURAL NETWORK; EXPERIMENTAL TESTING; HAND-GESTURE RECOGNITION; HUMAN ACTIVITY RECOGNITION; LEARNING-BASED APPROACH; REAL TIME; RECURRENT NETWORKS; STATE-OF-THE-ART PERFORMANCE;

GESTURE RECOGNITION;

EID: 85040351573     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2017.10.033     Document Type: Article

References (62)

1. Farooq, A., Won, C.S., A survey of human action recognition approaches that use an RGB-D sensor. IEEE Transactions on Smart Processing and Computing, 4, 2015, 281–290.
2. Wang, P., Li, W., Gao, Z., Zhang, J., Tang, C., Ogunbona, P., Deep convolutional neural networks for action recognition using depth map sequences. Comput. Res. Repository (CoRR), abs/1501.04686, 2015.
3. S. Escalera, X. Baro, J. Gonzalez, M. Bautista, M. Madadi, M. Reyes, V. Ponce, H. Escalante, J. Shotton, I. Guyon, ChaLearn Looking at People Challenge 2014: Dataset and Results, vol. 8925, Lecture Notes in Computer Science, pp. 459–473. 10.1007/978-3-319-16178-5_32.
4. Shahroudy, A., Liu, J., Ng, T.-T., Wang, G., Ntu RGB+D: a large scale dataset for 3d human activity analysis. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 1010–1019, 10.1109/CVPR.2016.115.
5. Zhang, J., Li, W., Ogunbona, P.O., Wang, P., Tang, C., RGB-D-based action recognition datasets: a survey. Pattern Recognit. 60:Supplement C (2016), 86–105 https://doi.org/10.1016/j.patcog.2016.05.019.
6. Presti, L.L., Cascia, M.L., 3D skeleton-based human action classification: a survey. Pattern Recognit. 53:Supplement C (2016), 130–147 https://doi.org/10.1016/j.patcog.2015.11.019.
7. Xia, L., Chen, C., Aggarwal, J.K., View invariant human action recognition using histograms of 3d joints. The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012, 20–27, 10.1109/CVPRW.2012.6239233.
8. Zanfir, M., Leordeanu, M., Sminchisescu, C., The moving pose: an efficient 3d kinematics descriptor for low-latency action recognition and detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013, 2752–2759, 10.1109/ICCV.2013.342.
9. Devanne, M., Wannous, H., Berretti, S., Pala, P., Daoudi, M., Bimbo, A., Space-time pose representation for 3d human action recognition. Proceedings of the ICIAP 2013 International Workshops on New Trends in Image Analysis and Processing (ICIAP), 8158, 2013, Springer-Verlag New York, Inc., New York, NY, USA, 456–464, 10.1007/978-3-642-41190-8_49.
10. A. Chrungoo, S.S. Manimaran, B. Ravindran, Activity Recognition for Natural Human Robot Interaction, vol. 8755, Springer International Publishing, Cham, pp. 84–94. 10.1007/978-3-319-11973-1_9.
11. Vemulapalli, R., Arrate, F., Chellappa, R., Human action recognition by representing 3d skeletons as points in a lie group. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, 588–595, 10.1109/CVPR.2014.82.
12. Evangelidis, G., Singh, G., Horaud, R., Skeletal quads: human action recognition using joint quadruples. International Conference on Pattern Recognition (ICPR), 2014, 3054–3063, 10.1109/ICPR.2014.772.
13. Zhang, H., Parker, L.E., Bio-inspired predictive orientation decomposition of skeleton trajectories for real-time human activity prediction. IEEE International Conference on Robotics and Automation (ICRA), 2015, 3053–3060, 10.1109/ICRA.2015.7139618.
14. Tao, L., Vidal, R., Moving poselets: a discriminative and interpretable skeletal motion representation for action recognition. IEEE International Conference on Computer Vision Workshop (ICCVW), 2015, 303–311, 10.1109/ICCVW.2015.48.
15. Coppola, C., Martinez Mozos, O., Bellotto, N., et al. Applying a 3d qualitative trajectory calculus to human action recognition using depth cameras. IEEE/RSJ IROS Workshop on Assistance and Service Robotics in a Human Environment, 2015, IEEE.
16. Ding, W., Liu, K., Cheng, F., Zhang, J., STFC: spatio-temporal feature chain for skeleton-based human action recognition. J. Vis. Commun. Image Represent. 26 (2015), 329–337, 10.1016/j.jvcir.2014.10.009.
17. Boulbaba, B.A., Su, J., Anuj, S., Action recognition using rate-invariant analysis of skeletal shape trajectories. IEEE Trans. Pattern Anal. Mach. Intell. 38:1 (2016), 1–13.
18. Zhu, G., Zhang, L., Shen, P., Song, J., An online continuous human action recognition algorithm based on the kinect sensor. Sensors 16:2 (2016), 161:1–161:18, 10.3390/s16020161.
19. Cippitelli, E., Gasparrini, S., Gambi, E., Spinsante, S., A human activity recognition system using skeleton data from RGBD sensors. Comput. Intell. Neurosci. 2016 (2016), 4351435:1–4351435:14, 10.1155/2016/4351435.
20. Wang, C., Wang, Y., Yuille, A.L., Mining 3d key-pose-motifs for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 2639–2647, 10.1109/CVPR.2016.289.
21. Wang, C., Flynn, J., Wang, Y., Yuille, A.L., Recognizing actions in 3d using action-snippets and activated simplices. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence AAAI’16, 2016, AAAI Press, 3604–3610.
22. Lillo, I., Niebles, J.C., Soto, A., A hierarchical pose-based approach to complex action understanding using dictionaries of actionlets and motion poselets. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 1981–1990.
23. Du, Y., Wang, W., Wang, L., Hierarchical recurrent neural network for skeleton based action recognition. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 10.1109/CVPR.2015.7298714.
24. Veeriah, V., Zhuang, V., Qi, G., Differential recurrent neural networks for action recognition. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, 4041–4049, 10.1109/ICCV.2015.460.
25. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., D., T., Saenko, K., Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 2625–2634, 10.1109/TPAMI.2016.2599174.
26. Neverova, N., Wolf, C., Lacey, G., Fridman, L., Chandra, D., Barbello, B., Taylor, G., Learning human identity from motion patterns. IEEE Access 4 (2015), 1810–1820, 10.1109/ACCESS.2016.2557846.
27. Neverova, N., Wolf, C., Taylor, G., Nebout, F., Moddrop: adaptive multi-modal gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38:8 (2016), 1692–1706 doi.ieeecomputersociety.org/10.1109/TPAMI.2015.2461544.
28. Lingfei, M., Fan, L., Yanjia, Z., Anjie, H., Human physical activity recognition based on computer vision with deep learning model. Proceedings of the IEEE International Instrumentation and Measurement Technology Conference (I2MTC), 2016, 1211–1217, 10.1109/I2MTC.2016.7520541.
29. Pichao, W., Zhaoyang, L., Yonghong, H., Wanqing, L., Action recognition based on joint trajectory maps using convolutional neural networks. Proceedings of the 2016 ACM on Multimedia Conference (MM ’16), 2016, 102–106, 10.1145/2964284.2967191.
30. Yanghao, L., Cuiling, L., Junliang, X., Wenjun, Z., Chunfeng, Y., Jiaying, L., Online human action detection using joint classification-regression recurrent neural networks. Proceedings of the European Conference on Computer Vision (ECCV) Lecture Notes in Computer Science, 9911, 2016, 203–220, 10.1007/978-3-319-46478-7_13.
31. Liu, J., Shahroudy, A., Xu, D., Wang, G., Spatio-temporal LSTM with trust gates for 3d human action recognition. Proceedings of the European Conference on Computer Vision (ECCV) Lecture Notes in Computer Science, 9907, 2016, 816–833, 10.1007/978-3-319-46487-9_50.
32. Mahasseni, B., Todorovic, S., Regularizing long short term memory with 3d human-skeleton sequences for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 3054–3063, 10.1109/CVPR.2016.333.
33. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., Xie, X., Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence AAAI’16, 2016, AAAI Press, 3697–3703.
34. Hu, J.F., Zheng, W.S., Lai, J.H., Zhang, J., Jointly learning heterogeneous features for rgb-d activity recognition. IEEE Trans. Pattern Anal. Mach. Intell., PP(99), 2017, 10.1109/TPAMI.2016.2640292 1–1.
35. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J., An end-to-end spatio-temporal attention model for human action recognition from skeleton data. Proceedings of the 2017 AAAI Conference on Artificial Intelligence, 2017, 4263–4270.
36. Han, F., Reily, B., Hoff, W., Zhang, H., Space-time representation of people based on 3d skeletal data: a review. Comput. Vis. Image Understanding 158:Supplement C (2017), 85–105 https://doi.org/10.1016/j.cviu.2017.01.011.
37. Ionescu, B., Coquin, D., Lambert, P., Buzuloi, V., Dynamic hand gesture recognition using the skeleton of the hand. EURASIP J. Appl. Signal Process. 13 (2005), 2101–2109, 10.1155/ASP.2005.2101.
38. Reddy, K., Latha, P., Babu, M., Hand gesture recognition using skeleton of hand and distance based metric. Adv. Comput. Inf. Technol. 198 (2011), 346–354, 10.1007/978-3-642-22555-0_36.
39. Wang, C., Chan, S.C., A new hand gesture recognition algorithm based on joint color-depth superpixel earth mover's distance. 4th International Workshop on Cognitive Information Processing (CIP), 2014, 10.1109/CIP.2014.6844497.
40. De Smedt, Q., Wannous, H., Vandeborre, J.P., Skeleton-based dynamic hand gesture recognition. The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2016, 1206–1214, 10.1109/CVPRW.2016.153.
41. Intel, RealSense SDK Developer Guide 6.0. Technical Report, 2015.
42. Lecun, Y., Bengio, Y., Hinton, G., Deep learning. Nature 521:7553 (2015), 436–444, 10.1038/nature14539.
43. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Comput. 9:8 (1997), 1735–1780, 10.1162/neco.1997.9.8.1735.
44. Glorot, X., Bordes, A., Bengio, Y., Deep sparse rectifier neural networks. Gordon, G.J., Dunson, D.B., (eds.) Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS-11), 15, 2011, 315–323 Journal of Machine Learning Research - Workshop and Conference Proceedings.
45. Raina, R., Madhavan, A., Ng, A.Y., Large-scale deep unsupervised learning using graphics processors. Proceedings of the 26th Annual International Conference on Machine Learning ICML ’09, 2009, ACM, New York, NY, USA, 873–880, 10.1145/1553374.1553486.
46. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15:1 (2014), 1929–1958.
47. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., Backpropagation applied to handwritten zip code recognition. Neural Comput. 1:4 (1989), 541–551, 10.1162/neco.1989.1.4.541.
48. Jain, A.K., Mao, J., Mohiuddin, K.M., Artificial neural networks: a tutorial. Computer 29:3 (1996), 31–44, 10.1109/2.485891.
49. Li, W., Zhang, Z., Liu, Z., Action recognition based on a bag of 3D points. The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, 9–14, 10.1109/CVPRW.2010.5543273.
50. Wang, J., Liu, Z., Wu, Y., Yuan, J., Mining actionlet ensemble for action recognition with depth cameras. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, 1290–1297, 10.1109/CVPR.2012.6247813.
51. Faria, D.R., Vieira, M., Premebida, C., Nunes, U., Probabilistic human daily activity recognition towards robot-assisted living. 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2015, 582–587, 10.1109/ROMAN.2015.7333644.
52. Zeiler, M.D., ADADELTA: an adaptive learning rate method. Comput. Res. Repository (CoRR), abs/1212.5701, 2012.
53. Chen, C., Jafari, R., Kehtarnavaz, N., Action recognition from depth sequences using depth motion maps-based local binary patterns. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV),Waikoloa Beach, HI, 2015, 1092–1099, 10.1109/WACV.2015.150.
54. Gowayyed, M.A., Torki, M., Hussein, M.E., El-Saban, M., Histogram of oriented displacements (hod): describing trajectories of human joints for action recognition. Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence (IJCAI) IJCAI ’13, 2013, AAAI Press, 1351–1357.
55. Behnam, R.E.F., Stats-calculus pose descriptor feeding a discrete HMM low-latency detection and recognition system for 3D skeletal actions. Comput. Res. Repository (CoRR), abs/1509.09014, 2015.
56. Anirudh, R., Turaga, P., Su, J., Srivastava, A., Elastic functional coding of human actions: from vector-fields to latent variables. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 3147–3155, 10.1109/CVPR.2015.7298934.
57. Ling, G., Fu, C., Human action recognition using APJ3D and random forests. J. Softw. (JSW) 8:9 (2013), 2238–2245, 10.4304/jsw.8.9.2238-2245.
58. Liu, A., Nie, W., Su, Y., Ma, L., Hao, T., Yang, Z., Coupled hidden conditional random fields for RGB-D human action recognition. Signal Process. 112:C (2015), 74–82, 10.1016/j.sigpro.2014.08.038.
59. Zhu, Y., Chen, W., Guo, G.-D., Fusing spatiotemporal features and joints for 3Daction recognition. The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013, 486–491, 10.1109/CVPRW.2013.78.
60. Jiang, M., Kong, J., Bebis, G., Huo, H., Informative joints based human action recognition using skeleton contexts. Signal Process. 33 (2015), 29–40, 10.1016/j.image.2015.02.004.
61. Theodorakopoulos, I., Kastaniotis, D., Economou, G., Fotopoulos, S., Pose-based human action recognition via sparse representation in dissimilarity space. J. Vis. Commun. Image Represent. 25:1 (2014), 12–23 https://doi.org/10.1016/j.jvcir.2013.03.008.
62. J.Y. Chang, Nonparametric Gesture Labeling from Multi-modal Data, Springer International Publishing, Cham, pp. 503–517. 10.1007/978-3-319-16178-5_35.