Scene flow to action map: A new representation for RGB-D based action recognition with convolutional neural networks

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

Volumn 2017-January, Issue , 2017, Pages 416-425

  Wang, Pichao ^a   Li, Wanqing ^a   Gao, Zhimin ^a   Zhang, Yuyao ^a   Tang, Chang ^b   Ogunbona, Philip ^a  

^a UNIVERSITY OF WOLLONGONG   (Australia)

^b CHINA UNIVERSITY OF GEOSCIENCES   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; CONVOLUTION; EXTRACTION; NEURAL NETWORKS; VECTOR SPACES;

ACTION RECOGNITION; CAMERA PARAMETER; CONVOLUTIONAL NEURAL NETWORK; LATER FUSIONS; LONG TERM DYNAMICS; SELF-CALIBRATION METHOD; SPATIO-TEMPORAL DYNAMICS; TRANSFORM KERNEL;

PATTERN RECOGNITION;

EID: 85043297155     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2017.52     Document Type: Conference Paper

References (64)

1. H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould. Dynamic image networks for action recognition. In CVPR, 2016.
2. A. F. Bobick and J. W. Davis. The recognition of human movement using temporal templates. IEEE Transactions on pattern analysis and machine intelligence, 23(3):257-267, 2001.
3. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, pages 2625-2634, 2015.
4. Y. Du, W. Wang, and L. Wang. Hierarchical recurrent neural network for skeleton based action recognition. In CVPR, pages 1110-1118, 2015.
5. B. Fernando, S. Gavves, O. Mogrovejo, J. Antonio, A. Ghodrati, and T. Tuytelaars. Rank pooling for action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016.
6. S. Hadfield and R. Bowden. Scene particles: Unregularized particle-based scene flow estimation. IEEE transactions on pattern analysis and machine intelligence, 36(3):564-576, 2014.
7. R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
8. M. Hornacek, A. Fitzgibbon, and C. Rother. Sphereflow: 6 DoF scene flow from RGB-D pairs. In CVPR, pages 3526- 3533, 2014.
9. Y. Hou, Z. Li, P. Wang, and W. Li. Skeleton optical spectra based action recognition using convolutional neural networks. IEEE Transactions on Circuits and Systems for Video Technology, 2016.
10. J.-F. Hu, W.-S. Zheng, J. Lai, and J. Zhang. Jointly learning heterogeneous features for RGB-D activity recognition. In CVPR, pages 5344-5352, 2015.
11. M. Jaimez, M. Souiai, J. Gonzalez-Jimenez, and D. Cremers. A primal-dual framework for real-time dense RGB-D scene flow. In ICRA, pages 98-104, 2015.
12. S. Ji, W. Xu, M. Yang, and K. Yu. 3D convolutional neural networks for human action recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(1):221- 231, 2013.
13. C. Jia and Y. Fu. Low-rank tensor subspace learning for rgb-d action recognition. IEEE Transactions on Image Processing, 25(10):4641-4652, 2016.
14. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proc. ACM international conference on Multimedia (ACM MM), pages 675-678, 2014.
15. C. Kanzow, N. Yamashita, and M. Fukushima. Withdrawn: Levenberg-marquardt methods with strong local convergence properties for solving nonlinear equations with convex constraints. Journal of Computational and Applied Mathematics, 173(2):321-343, 2005.
16. Y. Kong and Y. Fu. Bilinear heterogeneous information machine for RGB-D action recognition. In CVPR, pages 1054- 1062, 2015.
17. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Proc. Annual Conference on Neural Information Processing Systems (NIPS), pages 1106-1114, 2012.
18. Z. Lan, M. Lin, X. Li, A. G. Hauptmann, and B. Raj. Beyond gaussian pyramid: Multi-skip feature stacking for action recognition. In CVPR, pages 204-212, 2015.
19. C. Li, Y. Hou, P.Wang, andW. Li. Joint distance maps based action recognition with convolutional neural network. IEEE Signal Processing Letters, 2017.
20. W. Li, Z. Zhang, and Z. Liu. Action recognition based on a bag of 3D points. In CVPRW, pages 9-14, 2010.
21. A.-A. Liu, N. Xu, W.-Z. Nie, Y.-T. Su, Y. Wong, and M. Kankanhalli. Benchmarking a multimodal and multiview and interactive dataset for human action recognition. IEEE Transactions on cybernetics, 2016.
22. J. Liu, A. Shahroudy, D. Xu, and G. Wang. Spatio-temporal LSTM with trust gates for 3D human action recognition. In Proc. European Conference on Computer Vision, pages 816- 833, 2016.
23. J. Liu and G. Wang. Global context-aware attention lstm networks for 3d action recognition. In CVPR, 2017.
24. C. Lu, J. Jia, and C.-K. Tang. Range-sample depth feature for action recognition. In CVPR, pages 772-779, 2014.
25. J. Man and B. Bhanu. Individual recognition using gait energy image. IEEE transactions on pattern analysis and machine intelligence, 28(2):316-322, 2006.
26. M. Menze and A. Geiger. Object scene flow for autonomous vehicles. In CVPR, pages 3061-3070, 2015.
27. B. Ni, P. Moulin, and S. Yan. Pose adaptive motion feature pooling for human action analysis. International Journal of Computer Vision, 111(2):229-248, 2015.
28. S. Nie, Z. Wang, and Q. Ji. A generative restricted boltzmann machine based method for high-dimensional motion data modeling. Computer Vision and Image Understanding, pages 14-22, 2015.
29. O. Oreifej and Z. Liu. HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences. In CVPR, pages 716-723, 2013.
30. X. Peng, L. Wang, X. Wang, and Y. Qiao. Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice. Computer Vision and Image Understanding, 2016.
31. X. Peng, C. Zou, Y. Qiao, and Q. Peng. Action recognition with stacked fisher vectors. In ECCV, pages 581-595, 2014.
32. J. Quiroga, T. Brox, F. Devernay, and J. Crowley. Dense semi-rigid scene flow estimation from RGBD images. In ECCV, pages 567-582. 2014.
33. H. Rahmani and A. Mian. Learning a non-linear knowledge transfer model for cross-view action recognition. In CVPR, pages 2458-2466, 2015.
34. A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang. NTU RGB+ D: A large scale dataset for 3D human activity analysis. In CVPR, 2016.
35. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pages 568-576, 2014.
36. A. J. Smola and B. Schölkopf. A tutorial on support vector regression. Statistics and computing, 14(3):199-222, 2004.
37. D. Sun, E. B. Sudderth, and H. Pfister. Layered RGBD scene flow estimation. In CVPR, pages 548-556, 2015.
38. Y. Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10, 000 classes. In CVPR, 2014.
39. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In ICCV, pages 4489-4497, 2015.
40. A. Vedaldi and K. Lenc. Matconvnet - convolutional neural networks for matlab. In Proceeding of the ACM Int. Conf. on Multimedia, 2015.
41. S. Vedula, P. Rander, R. Collins, and T. Kanade. Threedimensional scene flow. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27(3):475-480, 2005.
42. V. Veeriah, N. Zhuang, and G.-J. Qi. Differential recurrent neural networks for action recognition. In ICCV, pages 4041-4049, 2015.
43. R. Vemulapalli, F. Arrate, and R. Chellappa. Human action recognition by representing 3D skeletons as points in a lie group. In CVPR, pages 588-595, 2014.
44. C. Vogel, K. Schindler, and S. Roth. Piecewise rigid scene flow. In ICCV, pages 1377-1384, 2013.
45. J. Wan, G. Guo, and S. Z. Li. Explore efficient local features from RGB-D data for one-shot learning gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8):1626-1639, Aug 2016.
46. J.Wan, S. Z. Li, Y. Zhao, S. Zhou, I. Guyon, and S. Escalera. Chalearn looking at people RGB-D isolated and continuous datasets for gesture recognition. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1-9, 2016.
47. J. Wan, Q. Ruan, W. Li, G. An, and R. Zhao. 3d smosift: three-dimensional sparse motion scale invariant feature transform for activity recognition from rgb-d videos. Journal of Electronic Imaging, 23(2), 2014.
48. H. Wang and C. Schmid. Action recognition with improved trajectories. In ICCV, pages 3551-3558, 2013.
49. J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for action recognition with depth cameras. In CVPR, pages 1290-1297, 2012.
50. L. Wang, Y. Qiao, and X. Tang. Action recognition with trajectory-pooled deep-convolutional descriptors. In CVPR, pages 4305-4314, 2015.
51. P. Wang, W. Li, Z. Gao, C. Tang, J. Zhang, and P. O. Ogunbona. Convnets-based action recognition from depth maps through virtual cameras and pseudocoloring. In ACM MM, pages 1119-1122, 2015.
52. P. Wang, W. Li, Z. Gao, J. Zhang, C. Tang, and P. Ogunbona. Action recognition from depth maps using deep convolutional neural networks. Human-Machine Systems, IEEE Transactions on, 46(4):498-509, 2016.
53. P. Wang, W. Li, S. Liu, Z. Gao, C. Tang, and P. Ogunbona. Large-scale isolated gesture recognition using convolutional neural networks. In Proceedings of ICPRW, 2016.
54. P. Wang, W. Li, S. Liu, Y. Zhang, Z. Gao, and P. Ogunbona. Large-scale continuous gesture recognition using convolutional neural networks. In Proceedings of ICPRW, 2016.
55. P.Wang, W. Li, P. Ogunbona, Z. Gao, and H. Zhang. Mining mid-level features for action recognition based on effective skeleton representation. In DICTA, 2014.
56. P. Wang, Z. Li, Y. Hou, and W. Li. Action recognition based on joint trajectory maps using convolutional neural networks. In ACM MM, pages 102-106, 2016.
57. A. Wedel, T. Brox, T. Vaudrey, C. Rabe, U. Franke, and D. Cremers. Stereoscopic scene flow computation for 3D motion understanding. International Journal of Computer Vision, 95(1):29-51, 2011.
58. C. Wu, J. Zhang, S. Savarese, and A. Saxena. Watch-npatch: Unsupervised understanding of actions and relations. In CVPR, pages 4362-4370, 2015.
59. X. Yang and Y. Tian. Super normal vector for activity recognition using depth sequences. In CVPR, pages 804-811, 2014.
60. X. Yang, C. Zhang, and Y. Tian. Recognizing actions using depth motion maps-based histograms of oriented gradients. In ACM MM, pages 1057-1060, 2012.
61. M. Yu, L. Liu, and L. Shao. Structure-preserving binary representations for rgb-d action recognition. IEEE transactions on pattern analysis and machine intelligence, 38(8):1651- 1664, 2016.
62. J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, pages 4694-4702, 2015.
63. J. Zhang, W. Li, P. O. Ogunbona, P. Wang, and C. Tang. RGB-D-based action recognition datasets: A survey. Pattern Recognition, 60:86-105, 2016.
64. Y. Zhang and C. Kambhamettu. On 3d scene flow and structure estimation. In CVPR, pages 778-785, 2001.