An end-to-end spatio-temporal attention model for human action recognition from skeleton data

31st AAAI Conference on Artificial Intelligence, AAAI 2017

Volumn , Issue , 2017, Pages 4263-4270

  Song, Sijie ^a   Lan, Cuiling ^b   Xing, Junliang ^c   Zeng, Wenjun ^b   Liu, Jiaying ^a  

^a PEKING UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

^c INSTITUTE OF AUTOMATION   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; LONG SHORT-TERM MEMORY; MUSCULOSKELETAL SYSTEM; RECURRENT NEURAL NETWORKS;

ATTENTION MODEL; HUMAN-ACTION RECOGNITION; MODEL LEARNING; RECURRENT NEURAL NETWORK (RNNS); SPATIO TEMPORAL; TEMPORAL EVOLUTION; TEMPORAL FEATURES; TRAINING STRATEGY;

DIGITAL STORAGE;

EID: 85030214213     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (38)

1. Ba, J.; Mnih, V.; and Kavukcuoglu, K. 2014. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755.
2. Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
3. Bazzani, L.; Larochelle, H.; and Torresani, L. 2016. Recurrent mixture density network for spatiotemporal visual attention. arXiv preprint arXiv:1603.08199.
4. Carlsson, S., and Sullivan, J. 2001. Action recognition by shape matching to key frames. In Workshop on Models versus Exemplars in Computer Vision.
5. Du, Y.; Wang, W.; and Wang, L. 2015. Hierarchical recurrent neural network for skeleton based action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 1110-1118.
6. Goferman, S.; Zelnik-Manor, L.; and Tal, A. 2012. Contextaware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(10):1915-1926.
7. Graves, A. 2012. Supervised sequence labelling with recurrent neural networks. Volume 385 of Studies in Computational Intelligence.
8. Han, F.; Reily, B.; Hoff, W.; and Zhang, H. 2016. Spacetime representation of people based on 3D skeletal data: A review. arXiv preprint arXiv:1601.01006.
9. Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735-1780.
10. Hochreiter, S.; Bengio, Y.; Frasconi, P.; and Schmidhuber, J. 2001. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. A field guide to dynamical recurrent neural networks. IEEE Press.
11. Hu, J.-F.; Zheng, W.-S.; Lai, J.; and Zhang, J. 2015. Jointly learning heterogeneous features for RGB-D activity recognition. In IEEE Conference on Computer Vision and Pattern Recognition.
12. Ji, Y.; Ye, G.; and Cheng, H. 2014. Interactive body part contrast mining for human interaction recognition. In IEEE International Conference on Multimedia and Expo Workshops.
13. Jiang, M.; Xu, J.; and Zhao, Q. 2014. Saliency in crowd. In European Conference on Computer Vision.
14. Johansson, G. 1973. Visual perception of biological motion and a model for it is analysis. Perception and Psychophysics 14(2):201-211.
15. Kingma, D., and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
16. Liu, L.; Shao, L.; and Rockett, P. 2013. Boosted key-frame selection and correlated pyramidal motion-feature representation for human action recognition. Pattern Recognition 46(7):1810-1818.
17. Poppe, R. 2010. A survey on vision-based human action recognition. Image and Vision Computing 28(6):976-990.
18. Ramanathan, V.; Huang, J.; Abu-El-Haija, S.; Gorban, A.; Murphy, K.; and Li, F.-F. 2015. Detecting events and key actors in multi-person videos. arXiv preprint arXiv:1511.02917.
19. Raptis, M., and Sigal, L. 2013. Poselet key-framing: A model for human activity recognition. In IEEE Conference on Computer Vision and Pattern Recognition.
20. Shahroudy, A.; Liu, J.; Ng, T.-T.; and Wang, G. 2016. NTU RGB+D: A large scale dataset for 3D human activity analysis. In IEEE Conference on Computer Vision and Pattern Recognition.
21. Sharma, S.; Kiros, R.; and Salakhutdinov, R. 2015. Action recognition using visual attention. arXiv preprint arXiv:1511.04119.
22. Shotton, J.; Fitzgibbon, A.; Cook, M.; Sharp, T.; Finocchio, M.; Moore, R.; Kipman, A.; and Blake, A. 2011. Realtime human pose recognition in parts from a single depth image. In IEEE Conference on Computer Vision and Pattern Recognition.
23. Simonyan, K., and Zisserman, A. 2014. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems.
24. Sminchisescu, C.; Kanaujia, A.; and Metaxas, D. 2006. Conditional models for contextual human motion recognition. Computer Vision and Image Understanding 104(2):210-220.
25. Vemulapalli, R.; Arrate, F.; and Chellappa, R. 2016. R3DG features: Relative 3D geometry-based skeletal representations for human action recognition. Computer Vision and Image Understanding.
26. Wang, J.; Liu, Z.; and Yuan, J. 2012. Mining actionlet ensemble for action recognition with depth cameras. In IEEE Conference on Computer Vision and Pattern Recognition.
27. Weinland, D.; Ronfard, R.; and Boyerc, E. 2011. A survey of vision-based methods for action representation, segmentation and recognition. Computer Vision and Image Understanding 115(2):224-241.
28. Wu, Z.; Wang, X.; Jiang, Y.-G.; Ye, H.; and Xue, X. 2015. Modeling spatial-temporal clues in a hybrid deep learning framework for video classification. In ACM International Conference on Multimedia.
29. Xia, L.; Chen, C.-C.; and Aggarwal, J. K. 2012. View invariant human action recognition using histograms of 3D joints. In IEEE Conference on Computer Vision and Pattern Recognition Workshops.
30. Xu, K.; Ba, J.; Kiros, R.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference for Machine Learning.
31. Yeung, S.; Russakovsky, O.; Jin, N.; Andriluka, M.; Mori, G.; and Li, F.-F. 2015. Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint arXiv:1507.05738.
32. Yu, G.; Liu, Z.; and Yuan, J. 2014. Discriminative orderlet mining for real-time recognition of human-object interaction. In Asian Conference on Computer Vision. Springer.
33. Yu, Y.; Mann, G. K.; and Gosine, R. G. 2010. An objectbased visual attention model for robotic applications. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 40(5):1398-1412.
34. Yun, K.; Honorio, J.; Chattopadhyay, D.; Berg, T. L.; and Samaras, D. 2012. Two-person interaction detection using body-pose features and multiple instance learning. In IEEE Conference on Computer Vision and Pattern Recognition Workshops.
35. Zaremba, W.; Sutskever, I.; and Vinyals, O. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.
36. Zhang, Z. 2012. Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2):4-10.
37. Zhao, Z., and Elgammal, A. 2008. Information theoretic key frame selection for action recognition. In British Machine Vision Conference.
38. Zhu, W.; Lan, C.; Xing, J.; Zeng, W.; Li, Y.; Shen, L.; and Xie, X. 2016. Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In AAAI Conference on Artificial Intelligence.