Multimodal Multipart Learning for Action Recognition in Depth Videos

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 10, 2016, Pages 2123-2129

  Shahroudy, Amir ^a,b   Ng, Tian Tsong ^b   Yang, Qingxiong ^c   Wang, Gang ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

^c CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Action recognition; group feature selection; joint sparse regression; kinect; mixed norms; structured sparsity

Indexed keywords

IMAGE RECOGNITION; LEARNING SYSTEMS; MOTION ESTIMATION;

ACTION RECOGNITION; KINECT; MIXED-NORM; SPARSE REGRESSION; STRUCTURED SPARSITIES;

FEATURE EXTRACTION;

ALGORITHM; AUTOMATED PATTERN RECOGNITION; HUMAN; HUMAN ACTIVITIES; LEARNING;

ALGORITHMS; HUMAN ACTIVITIES; HUMANS; LEARNING; PATTERN RECOGNITION, AUTOMATED;

EID: 84986267489     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2015.2505295     Document Type: Article

References (51)

1. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1297-1304.
2. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Learning actionlet ensemble for 3D human action recognition, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 5, pp. 914-927, May 2014.
3. O. Oreifej and Z. Liu, "HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 716-723.
4. B. Rao and K. Kreutz-Delgado, "An affine scalingmethodology for best basis selection, " IEEE Trans. Signal Process., vol. 47, no. 1, pp. 187-200, Jan. 1999.
5. W. Li, Z. Zhang, and Z. Liu, "Action recognition based on a bag of 3D points, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog. Workshops, 2010, pp. 9-14.
6. X. Yang and Y. Tian, "Effective 3D action recognition using eigenjoints, " J. Visual Commun. Image Representation, vol. 25, no. 1, pp. 2-11, 2014.
7. L. Xia, C.-C. Chen, and J. Aggarwal, "View invariant human action recognition using histograms of 3D joints, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog. Workshops, 2012, pp. 20-27.
8. G. Evangelidis, G. Singh, and R. Horaud, "Skeletal quads: Human action recognition using joint quadruples, " in Proc. 22nd Int. Conf. Pattern Recog., 2014, pp. 4513-4518.
9. J. Wang, Z. Liu, J. Chorowski, Z. Chen, and Y. Wu, "Robust 3D action recognition with random occupancy patterns, " in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 872-885.
10. H. Zou and T. Hastie, "Regularization and variable selection via the elastic net, " J. Roy. Statistical Soc. : Ser. B (Statistical Methodol.), vol. 67, no. 2, pp. 301-320, 2005.
11. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2005, pp. 886-893.
12. I. Laptev and T. Lindeberg, "Space-time interest points, " in Proc. IEEE Int. Conf. Comput. Vis., 2003, pp. 432-439.
13. B. Ni, G. Wang, and P. Moulin, "RGBD-HuDaAct: A color-depth video database for human daily activity recognition, " in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2011, pp. 193-208.
14. A. Bobick and J. Davis, "The recognition of human movement using temporal templates, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 3, pp. 257-267, Mar. 2001.
15. L. Xia and J. Aggarwal, "Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 2834-2841.
16. A. Klaeser, M. Marszalek, and C. Schmid, "A spatio-temporal descriptor based on 3D-gradients, " in Proc. Brit. Mach. Vis. Conf., 2008, pp. 1-10.
17. H. Rahmani, A. Mahmood, D. Huynh, and A. Mian, "Action classification with locality-constrained linear coding, " in Proc. 22nd Int. Conf. Pattern Recog., 2014, pp. 3511-3516.
18. P. Wang, W. Li, Z. Gao, J. Zhang, C. Tang, and P. O. Ogunbona, "Action recognition from depth maps using deep convolutional neural networks, " IEEE Trans. Human Mach. Syst., vol. PP, no. 99, pp. 1-12, 2015.
19. A. Shahroudy, G. Wang, and T.-T. Ng, "Multi-modal feature fusion for action recognition in RGB-D sequences, " in Proc. 6th Int. Symp. Commun., Control Signal Process., 2014, pp. 1-4.
20. A. Wang, J. Lu, J. Cai, T.-J. Cham, and G. Wang, "Large-margin multimodal deep learning for RGB-D object recognition, " IEEE Trans. Multimedia, vol. 17, no. 11, pp. 1887-1898, Nov. 2015.
21. A. Wang, J. Lu, J. Cai, G. Wang, and T.-J. Cham, "Unsupervised joint feature learning and encoding for RGB-D scene labeling, " IEEE Trans. Image Process., vol. 24, no. 11, pp. 4459-4473, Nov. 2015.
22. J. Wang and Y. Wu, "Learning maximum margin temporal warping for action recognition, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2688-2695.
23. M. Yuan and Y. Lin, "Model selection and estimation in regression with grouped variables, " J. Roy. Statistical Soc. : Ser. B (Statistical Methodol.), vol. 68, pp. 49-67, 2006.
24. R. Tibshirani, "Regression shrinkage and selection via the lasso, " J. Roy. Statistical Soc. Ser. B (Methodol.), vol. 73, no. 3, pp. 273-282, 1996.
25. P. Zhao, G. Rocha, and B. Yu, "The composite absolute penalties family for grouped and hierarchical variable selection, " Ann. Statist., vol. 37, no. 6A, pp. 3468-3497, 2009.
26. M. Szafranski, Y. Grandvalet, and P. Morizet-Mahoudeaux, "Hierarchical penalization, " in Advances in Neural Information Processing Systems (NIPS). Cambridge, MA, USA: MIT Press, 2008.
27. M. Kowalski, "Sparse regression using mixed norms, " Appl. Computat. Harmonic Anal., vol. 27, no. 3, pp. 303-324, 2009.
28. M. Kowalski and B. Torresani, "Structured sparsity: From mixed norms to structured shrinkage, " in Proc. Signal Process. Adaptive Sparse Struct. Represent., 2009.
29. H. Wang, F. Nie, H. Huang, and C. Ding, "Heterogeneous visual features fusion via sparse multimodal machine, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 3097-3102.
30. H. Wang, F. Nie, and H. Huang, "Multi-view clustering and feature learning via structured sparsity, " in Proc. 30th Int. Conf. Mach. Learn., 2013, pp. 352-360.
31. H. Wang, F. Nie, and H. Huang, "Robust and discriminative self-taught learning, " in Proc. Int. Conf. Mach. Learn., 2013, pp. 298-306.
32. H. Wang, F. Nie, W. Cai, and H. Huang, "Semi-supervised robust dictionary learning via efficient l2;0-norms minimization, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1145-1152.
33. R. Caruana, "Multitask learning, " Mach. Learn., vol. 28, no. 1, pp. 41-75, 1997.
34. Q. Zhou, G. Wang, K. Jia, and Q. Zhao, "Learning to share latent tasks for action recognition, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2264-2271.
35. L. Wang, N. T. Pham, T.-T. Ng, G. Wang, K. L. Chan, and K. Leman, "Learning deep features for multiple object tracking by using a multi-task learning strategy, " in Proc. IEEE Int. Conf. Image Process., 2014, pp. 838-842.
36. A. Abdulnabi, G. Wang, J. Lu, and K. Jia, "Multi-task CNN model for attribute prediction, " IEEE Trans. Multimedia, vol. 17, no. 11, pp. 1949-1959, Nov. 2015.
37. H. Liu, M. Palatucci, and J. Zhang, "Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery, " in Proc. 26th Int. Conf. Mach. Learn., 2009, p. 82.
38. G. Obozinski, B. Taskar, and M. I. Jordan, "Joint covariate selection and joint subspace selection for multiple classification problems, " Statist. Comput., vol. 20, no. 2, pp. 231-252, 2010.
39. X.-T. Yuan, X. Liu, and S. Yan, "Visual classification with multitask joint sparse representation, " IEEE Trans Image Process., vol. 21, no. 10, pp. 4349-4360, Oct. 2012.
40. T. Zhang, B. Ghanem, S. Liu, and N. Ahuja, "Robust visual tracking via multi-task sparse learning, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2042-2049.
41. A. Rakotomamonjy, R. Flamary, G. Gasso, and S. Canu, "lp - lq penalty for sparse linear and sparse multiple kernel multitask learning, " IEEE Trans. Neural Netw., vol. 22, no. 8, pp. 1307-1320, Aug. 2011.
42. M. Kowalski and B. Torrsani, "Sparsity and persistence: Mixed norms provide simple signal models with dependent coefficients, " Signal, Image Video Process., vol. 3, no. 3, pp. 251-264, 2009.
43. M. Schmidt. (2005). Minfunc [Online]. Available: http://www. di. ens. fr/~mschmidt/Software/minFunc. html
44. J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, "Locality-constrained linear coding for image classification, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 3360-3367.
45. G. Yu, Z. Liu, and J. Yuan, "Discriminative orderlet mining for real-time recognition of human-object interaction, " in Proc. Asian Conf. Comput. Vis., 2014, pp. 50-65.
46. S. Althloothi, M. H. Mahoor, X. Zhang, and R. M. Voyles, "Human activity recognition using multi-features and multiple kernel learning, " Pattern Recog., vol. 47, no. 5, pp. 1800-1812, 2014.
47. X. Yang, C. Zhang, and Y. Tian, "Recognizing actions using depth motion maps-based histograms of oriented gradients, " in Proc. 20th ACM Int. Conf. Multimedia, 2012, pp. 1057-1060.
48. R. Vemulapalli, F. Arrate, and R. Chellappa, "Human action recognition by representing 3D skeletons as points in a lie group, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 1057-1060.
49. E. Ohn-Bar and M. Trivedi, "Joint angles similarities and hog2 for action recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog. Workshops, 2013, pp. 465-470.
50. J. Luo, W. Wang, and H. Qi, "Group sparsity and geometry constrained dictionary learning for action recognition from depth maps, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1809-1816.
51. H. Rahmani, A. Mahmood, D. Q Huynh, and A. Mian, "HOPC: Histogram of oriented principal components of 3D pointclouds for action recognition, " in Proc. Eur. Conf. Comput. Vis., 2014, pp. 742-757.