Global context-aware attention LSTM networks for 3D action recognition

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

Volumn 2017-January, Issue , 2017, Pages 3671-3680

  Liu, Jun ^a   Wang, Gang ^b   Hu, Ping ^a   Duan, Ling Yu ^c   Kot, Alex C ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b ALIBABA GROUP   (China)

^c PEKING UNIVERSITY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; PATTERN RECOGNITION;

ACTION RECOGNITION; ACTION SEQUENCES; ATTENTION MECHANISMS; ATTENTION PERFORMANCE; CONTEXTUAL INFORMATION; END-TO-END NETWORK; HUMAN-ACTION RECOGNITION; STATE-OF-THE-ART PERFORMANCE;

LONG SHORT-TERM MEMORY;

EID: 85041922534     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2017.391     Document Type: Conference Paper

References (73)

1. J. K. Aggarwal and L. Xia. Human activity recognition from 3d data: A review. PR Letters, 2014.
2. R. Anirudh, P. Turaga, J. Su, and A. Srivastava. Elastic functional coding of human actions: from vector-fields to latent variables. In CVPR, 2015.
3. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
4. R. Chaudhry, F. Ofli, G. Kurillo, R. Bajcsy, and R. Vidal. Bio-inspired dynamic 3d discriminative skeletal features for human action recognition. In CVPRW, 2013.
5. C. Chen, R. Jafari, and N. Kehtarnavaz. Fusion of depth, skeleton, and inertial data for human action recognition. In ICASSP, 2016.
6. H. Chen, G. Wang, J.-H. Xue, and L. He. A novel hierarchical framework for human action recognition. PR, 2016.
7. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based models for speech recognition. In NIPS, 2015.
8. R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In NIPSW, 2011.
9. M. Devanne, H. Wannous, S. Berretti, P. Pala, M. Daoudi, and A. Del Bimbo. 3-d human action recognition by shape analysis of motion trajectories on riemannian manifold. IEEE Transactions on Cybernetics, 2015.
10. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
11. Y. Du, W. Wang, and L. Wang. Hierarchical recurrent neural network for skeleton based action recognition. In CVPR, 2015.
12. G. Evangelidis, G. Singh, and R. Horaud. Skeletal quads: Human action recognition using joint quadruples. In ICPR, 2014.
13. A. Graves. Supervised sequence labelling. In Supervised Sequence Labelling with Recurrent Neural Networks. 2012.
14. F. Han, B. Reily, W. Hoff, and H. Zhang. Space-time representation of people based on 3d skeletal data: a review. arXiv, 2016.
15. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1997.
16. J.-F. Hu, W.-S. Zheng, J. Lai, and J. Zhang. Jointly learning heterogeneous features for rgb-d activity recognition. In CVPR, 2015.
17. M. Ibrahim, S. Muralidharan, Z. Deng, A. Vahdat, and G. Mori. A hierarchical deep temporal model for group activity recognition. In CVPR, 2016.
18. A. Jain, A. R. Zamir, S. Savarese, and A. Saxena. Structuralrnn: Deep learning on spatio-temporal graphs. In CVPR, 2016.
19. Y. Ji, G. Ye, and H. Cheng. Interactive body part contrast mining for human interaction recognition. In ICMEW, 2014.
20. M. Jiang, J. Kong, G. Bebis, and H. Huo. Informative joints based human action recognition using skeleton contexts. Signal Processing: Image Communication, 2015.
21. Q. Ke, M. Bennamoun, S. An, F. Bossaid, and F. Sohel. Spatial, structural and temporal feature learning for human interaction prediction. arXiv, 2016.
22. P. Koniusz, A. Cherian, and F. Porikli. Tensor representations via kernel linearization for action recognition from 3d skeletons. In ECCV, 2016.
23. A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, V. Zhong, R. Paulus, and R. Socher. Ask me anything: Dynamic memory networks for natural language processing. In ICML, 2016.
24. W. Li, L. Wen, M. Choo Chuah, and S. Lyu. Category-blind human action recognition: A practical recognition system. In ICCV, 2015.
25. Y. Li, C. Lan, J. Xing, W. Zeng, C. Yuan, and J. Liu. Online human action detection using joint classification-regression recurrent neural networks. In ECCV, 2016.
26. I. Lillo, J. Carlos Niebles, and A. Soto. A hierarchical posebased approach to complex action understanding using dictionaries of actionlets and motion poselets. In CVPR, 2016.
27. J. Liu, A. Shahroudy, D. Xu, and G. Wang. Spatio-temporal lstm with trust gates for 3d human action recognition. In ECCV, 2016.
28. J. Luo, W. Wang, and H. Qi. Group sparsity and geometry constrained dictionary learning for action recognition from depth maps. In ICCV, 2013.
29. M.-T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015.
30. S. Ma, L. Sigal, and S. Sclaroff. Learning activity progression in lstms for activity detection and early detection. In CVPR, 2016.
31. M. Meng, H. Drira, M. Daoudi, and J. Boonaert. Humanobject interaction recognition by learning the distances between the object and the skeleton joints. In FG, 2015.
32. F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy. Sequence of the most informative joints (smij): A new representation for human skeletal action recognition. JVCIR, 2014.
33. L. L. Presti and M. La Cascia. 3d skeleton-based human action classification: A survey. PR, 2016.
34. H. Rahmani, A. Mahmood, D. Q. Huynh, and A. Mian. Real time action recognition using histograms of depth gradients and random decision forests. In WACV, 2014.
35. A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In CVPR, 2016.
36. A. Shahroudy, T.-T. Ng, Y. Gong, and G. Wang. Deep multimodal feature analysis for action recognition in rgb+ d videos. TPAMI, 2017.
37. A. Shahroudy, T.-T. Ng, Q. Yang, and G. Wang. Multimodal multipart learning for action recognition in depth videos. TPAMI, 2016.
38. A. Shahroudy, G. Wang, and T.-T. Ng. Multi-modal feature fusion for action recognition in rgb-d sequences. In ISCCSP, 2014.
39. S. Sharma, R. Kiros, and R. Salakhutdinov. Action recognition using visual attention. In ICLRW, 2016.
40. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.
41. R. Slama, H. Wannous, M. Daoudi, and A. Srivastava. Accurate 3d action recognition using learning on the grassmann manifold. PR, 2015.
42. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
43. N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using lstms. In ICML, 2015.
44. M. F. Stollenga, J. Masci, F. Gomez, and J. Schmidhuber. Deep networks with internal selective attention through feedback connections. In NIPS, 2014.
45. S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-toend memory networks. In NIPS, 2015.
46. M. Sundermeyer, R. Schlüter, and H. Ney. Lstm neural networks for language modeling. In INTERSPEECH, 2012.
47. L. Tao and R. Vidal. Moving poselets: A discriminative and interpretable skeletal motion representation for action recognition. In ICCVW, 2015.
48. V. Veeriah, N. Zhuang, and G.-J. Qi. Differential recurrent neural networks for action recognition. In ICCV, 2015.
49. R. Vemulapalli, F. Arrate, and R. Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In CVPR, 2014.
50. C. Wang, J. Flynn, Y. Wang, and A. L. Yuille. Recognizing actions in 3d using action-snippets and activated simplices. In AAAI, 2016.
51. C. Wang, Y. Wang, and A. L. Yuille. Mining 3d key-posemotifs for action recognition. In CVPR, 2016.
52. J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for action recognition with depth cameras. In CVPR, 2012.
53. J. Wang, Z. Liu, Y. Wu, and J. Yuan. Learning actionlet ensemble for 3d human action recognition. TPAMI, 2014.
54. J. Wang and Y. Wu. Learning maximum margin temporal warping for action recognition. In ICCV, 2013.
55. P.Wang, W. Li, Z. Gao, Y. Zhang, C. Tang, and P. Ogunbona. Scene flow to action map: A new representation for rgb-d based action recognition with convolutional neural networks. In CVPR, 2017.
56. P.Wang, W. Li, P. Ogunbona, Z. Gao, and H. Zhang. Mining mid-level features for action recognition based on effective skeleton representation. In DICTA, 2014.
57. P. Wang, C. Yuan, W. Hu, B. Li, and Y. Zhang. Graph based skeleton motion representation and similarity measurement for action recognition. In ECCV, 2016.
58. Y. Wang, S. Wang, J. Tang, N. O'Hare, Y. Chang, and B. Li. Hierarchical attention network for action recognition in videos. arXiv, 2016.
59. J.Weng, C.Weng, and J. Yuan. Spatio-temporal naive-bayes nearest-neighbor for skeleton-based action recognition. In CVPR, 2017.
60. J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015.
61. Z. Wu, X. Wang, Y.-G. Jiang, H. Ye, and X. Xue. Modeling spatial-temporal clues in a hybrid deep learning framework for video classification. In ACM MM, 2015.
62. L. Xia, C.-C. Chen, and J. Aggarwal. View invariant human action recognition using histograms of 3d joints. In CVPRW, 2012.
63. C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. In ICML, 2016.
64. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
65. X. Yang and Y. Tian. Effective 3d action recognition using eigenjoints. JVCIR, 2014.
66. L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting temporal structure. In ICCV, 2015.
67. M. Ye, Q. Zhang, L. Wang, J. Zhu, R. Yang, and J. Gall. A survey on human motion analysis from depth data. In Time-of-flight and depth imaging. sensors, algorithms, and applications. 2013.
68. S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei. Endto-end learning of action detection from frame glimpses in videos. In CVPR, 2016.
69. J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
70. K. Yun, J. Honorio, D. Chattopadhyay, T. L. Berg, and D. Samaras. Two-person interaction detection using bodypose features and multiple instance learning. In CVPRW, 2012.
71. M. Zanfir, M. Leordeanu, and C. Sminchisescu. The moving pose: An efficient 3d kinematics descriptor for low-latency action recognition and detection. In ICCV, 2013.
72. J. Zhang, W. Li, P. O. Ogunbona, P.Wang, and C. Tang. Rgbd-based action recognition datasets: A survey. PR, 2016.
73. W. Zhu, C. Lan, J. Xing, W. Zeng, Y. Li, L. Shen, and X. Xie. Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In AAAI, 2016.