Jointly Learning Heterogeneous Features for RGB-D Activity Recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 11, 2017, Pages 2186-2200

  Hu, Jian Fang ^a,b   Zheng, Wei Shi ^a   Lai, Jianhuang ^a,c   Zhang, Jianguo ^d  

^a SUN YAT SEN UNIVERSITY   (China)

^b NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

^c Guangdong Province Key Laboratory of Information Security   (China)

^d UNIVERSITY OF DUNDEE   (United Kingdom)

Author keywords

action recognition; Heterogeneous features learning; RGB D activity recognition

Indexed keywords

INFERENCE ENGINES; ITERATIVE METHODS; OPTIMIZATION; PATTERN RECOGNITION;

ACTION RECOGNITION; ACTIVITY RECOGNITION; HETEROGENEOUS FEATURES; HIDDEN STRUCTURES; HUMAN-OBJECT INTERACTION; ITERATIVE OPTIMIZATION ALGORITHMS; MULTITASK LEARNING; SPECIFIC COMPONENT;

IMAGE RECOGNITION;

EID: 85032276758     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2640292     Document Type: Article

References (59)

1. Y. Amit, M. Fink, N. Srebro, and S. Ullman, "Uncovering shared structures in multiclass classification," in Proc. Int. Conf. Mach. Learn., 2007, pp. 17-24.
2. R. K. Ando and T. Zhang, "A framework for learning predictive structures from multiple tasks and unlabeled data," J. Mach. Learn. Res., vol. 6, pp. 1817-1853, 2005.
3. Z. Cai, L. Wang, and X. P. Y. Qiao, "Multi-view super vector for action recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 596-603.
4. L. Cao, J. Luo, F. Liang, and T. S. Huang, "Heterogeneous feature machines for visual recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1095-1102.
5. A. A. Chaaraoui, J. R. Padilla-López, and F. Flórez-Revuelta, "Fusion of skeletal and silhouette-based features for human action recognition with RGB-D devices," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2013, pp. 91-97.
6. J. Chen, L. Tang, J. Liu, and J. Ye, "A convex formulation for learning a shared predictive structure from multiple tasks," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 5, pp. 1025-1038, May 2013.
7. Y. Du, W. Wang, and L. Wang, "Hierarchical recurrent neural network for skeleton based action recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1110-1118.
8. T. Guha and R. K. Ward, "Learning sparse representations for human action recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 8, pp. 1576-1588, Aug. 2012.
9. S. Han, X. Liao, and L. Carin, "Cross-domain multitask learning with latent probit models," in Proc. 29th Int. Conf. Mach. Learning, 2012, pp. 1463-1470.
10. J. Hu, W. Zheng, J. Lai, S. Gong, and T. Xiang, "Exemplar-based recognition of human-object interactions," IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 4, pp. 647-660, Apr. 2016.
11. J.-F. Hu, W.-S. Zheng, J. Lai, and J. Zhang, "Jointly learning heterogeneous features for RGB-D activity recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 5344-5352.
12. J.-F. Hu, W.-S. Zheng, L. Ma, G. Wang, and J. Lai, "Real-time RGBD activity prediction by soft regression," in Proc. Eur. Conf. Comput. Vis., 2016, pp. 280-296.
13. M. E. Hussein, M. Torki, M. A. Gowayyed, and M. El-Saban, "Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations," in Proc. Int. Joint Conf. Artif. Intell., 2013, pp. 2466-2472.
14. Y. Kong and Y. Fu, "Bilinear heterogeneous information machine for RGB-D action recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1054-1062.
15. H. S. Koppula, R. Gupta, and A. Saxena, "Learning human activities and object affordances from RGB-D videos," Int. J. Robot. Res., vol. 32, no. 8, pp. 951-970, 2013.
16. J. Lei, X. Ren, and D. Fox, "Fine-grained kitchen activity recognition using RGB-D," in Proc. ACM Conf. Ubiquitous Comput., 2012, pp. 208-211.
17. W. Li, L. Duan, D. Xu, and I. W. Tsang, "Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 6, pp. 1134-1148, Jun. 2014.
18. W. Li, Z. Zhang, and Z. Liu, "Action recognition based on a bag of 3D points," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. Workshops, 2010, pp. 9-14.
19. I. Lillo, A. Soto, and J. C. Niebles, "Discriminative hierarchical modeling of spatio-temporally composable human activities," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2014, pp. 812-819.
20. L. Liu and L. Shao, "Learning discriminative representations from RGB-D video data," in Proc. Int. Joint Conf. Artif. Intell., 2013, pp. 1493-1500.
21. C. Lu, J. Jia, and C.-K. Tang, "Range-sample depth feature for action recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2014, pp. 772-779.
22. J. Luo, W. Wang, and H. Qi, "Group sparsity and geometry constrained dictionary learning for action recognition from depth maps," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1809-1816.
23. F. Lv and R. Nevatia, "Recognition and segmentation of 3D human action using hmm and multi-class adaboost," in Proc. Eur. Conf. Comput. Vis., 2006, pp. 359-372.
24. R. Messing, C. Pal, and H. Kautz, "Activity recognition using the velocity histories of tracked keypoints," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 104-111.
25. M. Müller and T. Röder, "Motion templates for automatic classification and retrieval of motion capture data," in Proc. ACM SIGGRAPH/Eurographics Symp. Comput. Animation, 2006, pp. 137-146.
26. B. Ni, P. Moulin, and S. Yan, "Order-preserving sparse coding for sequence classification," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 173-187.
27. F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, "Sequence of the most informative joints (smij): A new representation for human skeletal action recognition," J. Visual Commun. Image Representation, vol. 25, no. 1, pp. 24-38, 2014.
28. O. Oreifej and Z. Liu, "Hon4d: Histogram of oriented 4D normals for activity recognition from depth sequences," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2013, pp. 716-723.
29. S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345-1359, Oct. 2010.
30. A. Shahroudy, G. Wang, and T.-T. Ng, "Multi-modal feature fusion for action recognition in RGB-D sequences," in Proc. Int. Symp. Control Commun. Signal Process., May 2014, pp. 1-4.
31. J. Shotton, et al., "Real-time human pose recognition in parts from single depth images," Commun. ACM, vol. 56, no. 1, pp. 116-124, 2013.
32. Y. Song, L.-P. Morency, and R. Davis, "Multi-view latent variable discriminative models for action recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2120-2127.
33. J. Sung, C. Ponce, B. Selman, and A. Saxena, "Human activity detection from RGBD images," Plan Activity Intent Recognit., vol. 64, pp. 47-55, 2011.
34. R. Vemulapalli, F. Arrate, and R. Chellappa, "Human action recognition by representing 3D skeletons as points in a lie group," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2013, pp. 588-595.
35. A. Wang, J. Cai, J. Lu, and T.-J. Cham, "MMSS: Multi-modal sharable and specific feature learning for RGB-D object recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1125-1133.
36. H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, "Action recognition by dense trajectories," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2011, pp. 3169-3176.
37. H. Wang, C. Yuan, W. Hu, and C. Sun, "Supervised class-specific dictionary learning for sparse modeling in action recognition," Pattern Recognit., vol. 45, no. 11, pp. 3902-3911, 2012.
38. J. Wang, Z. Liu, J. Chorowski, Z. Chen, and Y. Wu, "Robust 3D action recognition with random occupancy patterns," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 872-885.
39. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Learning actionlet ensemble for 3D human action recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 5, pp. 914-927, May 2014.
40. S. Wang, Y. Yang, Z. Ma, X. Li, C. Pang, and A. G. Hauptmann, "Action recognition by exploring data distribution and feature correlation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 1370-1377.
41. P. Wei, Y. Zhao, N. Zheng, and S.-C. Zhu, "Modeling 4D humanobject interactions for event and object recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3272-3279.
42. Z. Wen and W. Yin, "A feasible method for optimization with orthogonality constraints," Math. Program., vol. 142, no. 1/2, pp. 397-434, 2013.
43. L. Xia and J. Aggarwal, "Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2013, pp. 2834-2841.
44. L. Xia, C.-C. Chen, and J. Aggarwal, "View invariant human action recognition using histograms of 3D joints," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. Workshops, 2012, pp. 20-27.
45. Y. Yan, E. Ricci, R. Subramanian, G. Liu, and N. Sebe, "Multitask linear discriminant analysis for view invariant action recognition," IEEE Trans. Image Process., vol. 23, no. 12, pp. 5599-5611, Dec. 2014.
46. X. Yang, S. Kim, and E. P. Xing, "Heterogeneous multitask learning with joint sparsity constraints," in Proc. Advances Neural Inf. Process. Syst., 2009, pp. 2151-2159.
47. X. Yang and Y. Tian, "Eigenjoints-based action recognition using naive-Bayes-nearest-neighbor," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. Workshops, 2012, pp. 14-19.
48. X. Yang and Y. Tian, "Super normal vector for activity recognition using depth sequences," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2014, pp. 804-811.
49. B. Yao and L. Fei-Fei, "Action recognition with exemplar based 2.5D graph matching," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 173-186.
50. B. Yao and L. Fei-Fei, "Recognizing human-object interactions in still images by modeling the mutual context of objects and human poses," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 9, pp. 1691-1703, Sep. 2012.
51. M. Yu, L. Liu, and L. Shao, "Structure-preserving binary representations for RGB-D action recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp. 1651-1664, Aug. 2016.
52. M. Zanfir, M. Leordeanu, and C. Sminchisescu, "The moving pose: An efficient 3D kinematics descriptor for low-latency action recognition and detection," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2752-2759.
53. Y. Zhang and D.-Y. Yeung, "Multi-task learning in heterogeneous feature spaces," in Proc. Conf. Artif. Intell., 2011, pp. 574-579.
54. Y. Zhang and D.-Y. Yeung, "A convex formulation for learning task relationships in multi-task learning," in Proc. Twenty-Sixth Conf. Uncertainty Artif. Intell., 2010, pp. 733-742.
55. Y. Zhao, Z. Liu, L. Yang, and H. Cheng, "Combing rgb and depth map features for human activity recognition," in Proc. IEEE Asia-Pacific Signal Inf. Process. Assoc. Annu. Summit Conf., 2012, pp. 1-4.
56. W.-S. Zheng, S. Gong, and T. Xiang, "Towards open-world person re-identification by one-shot group-based verification," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 3, pp. 591-606, Mar. 2016.
57. Q. Zhou, G. Wang, K. Jia, and Q. Zhao, "Learning to share latent tasks for action recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2264-2271.
58. Y. Zhu, W. Chen, and G. Guo, "Fusing spatiotemporal features and joints for 3d action recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. Workshops, 2013, pp. 486-491.
59. Y. Zhu, W. Chen, and G. Guo, "Evaluating spatiotemporal interest point features for depth-based action recognition," Image Vis. Comput., vol. 32, no. 8, pp. 453-464, 2014.