CoDe4D: Color-depth local spatio-temporal features for human activity recognition from RGB-D videos

IEEE Transactions on Circuits and Systems for Video Technology

Volumn 26, Issue 3, 2016, Pages 541-555

  Zhang, Hao ^a   Parker, Lynne E ^b  

^a Department of Metallurgical and Materials Engineering   (United States)

^b University of Tennessee   (United States)

Author keywords

3D machine vision; activity recognition; Color depth feature; human representation; RGB D sensor

Indexed keywords

ARTIFICIAL INTELLIGENCE; BENCHMARKING; CAMERAS; COLOR; COMPUTER VISION; FACE RECOGNITION; FEATURE EXTRACTION; HUMAN COMPUTER INTERACTION; IMAGE RECOGNITION; LEARNING SYSTEMS; PATTERN RECOGNITION; ROBOTS; SECURITY SYSTEMS;

3D MACHINE VISION; ACTIVITY RECOGNITION; COLOR DEPTH; HUMAN REPRESENTATION; RGB-D SENSORS;

HUMAN ROBOT INTERACTION;

EID: 84964371812     PISSN: 10518215     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSVT.2014.2376139     Document Type: Article

References (50)

1. P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea, "Machine recognition of human activities: A survey, " IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 11, pp. 1473-1488, Nov. 2008.
2. J. K. Aggarwal and M. S. Ryoo, "Human activity analysis: A review, " ACM Comput. Surv., vol. 43, no. 3, pp. 16:1-16:43, Apr. 2011.
3. P. V. K. Borges, N. Conci, and A. Cavallaro, "Video-based human behavior understanding: A survey, " IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 11, pp. 1993-2008, Nov. 2013.
4. A. Patron-Perez, M. Marszalek, I. Reid, and A. Zisserman, "Structured learning of human interactions in TV shows, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 12, pp. 2441-2453, Dec. 2012.
5. H. Zhang and L. E. Parker, "4-dimensional local spatio-temporal features for human activity recognition, " in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., Sep. 2011, pp. 2044-2049.
6. W. Choi, K. Shahid, and S. Savarese, "Learning context for collective activity recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 3273-3280.
7. T. Lan, Y. Wang, W. Yang, S. N. Robinovitch, and G. Mori, "Discriminative latent models for recognizing contextual group activities, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 8, pp. 1549-1562, Aug. 2012.
8. S. Khamis, V. I. Morariu, and L. S. Davis, "A flow model for joint action recognition and identity maintenance, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp. 1218-1225.
9. L. Xia and J. K. Aggarwal, "Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 2834-2841.
10. A. M. Treisman and G. Gelade, "A feature-integration theory of attention, " Cognit. Psychol., vol. 12, no. 1, pp. 97-136, Jan. 1980.
11. D. G. Lowe, "Distinctive image features from scale-invariant keypoints, " Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, Nov. 2004.
12. H. Wang, M. M. Ullah, A. Klaser, I. Laptev, and C. Schmid, "Evaluation of local spatio-temporal features for action recognition, " in Proc. Brit. Mach. Vis. Conf., Sep. 2009, pp. 127-137.
13. P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recognition via sparse spatio-temporal features, " in Proc. IEEE Int. Workshop Vis. Surveill. Perform. Eval. Tracking Surveill., Oct. 2005, pp. 65-72.
14. I. Laptev, "On space-time interest points, " Int. J. Comput. Vis., vol. 64, nos. 2-3, pp. 107-123, Sep. 2005.
15. H. Zhang, W. Zhou, C. Reardon, and L. E. Parker, "Simplex-based 3D spatio-temporal feature description for action recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 2059-2066.
16. B. Chakraborty, M. B. Holte, T. B. Moeslund, and J. Gonzàlez, "Selective spatio-temporal interest points, " Comput. Vis. Image Understand., vol. 116, no. 3, pp. 396-410, Mar. 2012.
17. H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, "Dense trajectories and motion boundary descriptors for action recognition, " Int. J. Comput. Vis., vol. 103, no. 1, pp. 60-79, May 2013.
18. H. Wang and C. Schmid, "Action recognition with improved trajectories, " in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 3551-3558.
19. E. Yu and J. K. Aggarwal, "Human action recognition with extremities as semantic posture representation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshop, Jun. 2009, pp. 1-8.
20. Z. Zeng and Q. Ji, "Knowledge based activity recognition with dynamic Bayesian network, " in Proc. Eur. Conf. Comput. Vis., 2010, pp. 532-546.
21. A. K. R. Chowdhury and R. Chellappa, "A factorization approach for activity recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshop, Jun. 2003, p. 41.
22. O. Brdiczka, M. Langet, J. Maisonnasse, and J. L. Crowley, "Detecting human behavior models from multimodal observation in a smart home, " IEEE Trans. Autom. Sci. Eng., vol. 6, no. 4, pp. 588-597, Oct. 2009.
23. V. Veeraraghavan, A. K. Roy-Chowdhury, and R. Chellappa, "Matching shape sequences in video with applications in human movement analysis, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 12, pp. 1896-1909, Dec. 2005.
24. P. Yan, S. M. Khan, and M. Shah, "Learning 4D action feature models for arbitrary view action recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-7.
25. M. Singh, A. Basu, and M. K. Mandal, "Human activity recognition based on silhouette directionality, " IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 9, pp. 1280-1292, Sep. 2008.
26. J. Y. Sung, C. Ponce, B. Selman, and A. Saxena, "Human activity detection from RGBD images, " in Proc. AAAIWorkshop Pattern, Activity Intent Recognit., 2011, pp. 47-55.
27. J. Ben-Arie, Z. Wang, P. Pandit, and S. Rajaram, "Human activity recognition using multidimensional indexing, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 8, pp. 1091-1104, Aug. 2002.
28. S. Knoop, S. Vacek, and R. Dillmann, "Sensor fusion for 3D human body tracking with an articulated 3D body model, " in Proc. IEEE Int. Conf. Robot. Automat., May 2006, pp. 1686-1691.
29. L. Schwarz, D. Mateus, V. Castaneda, and N. Navab, "Manifold learning for ToF-based human body tracking and activity recognition, " in Proc. Brit. Mach. Vis. Conf., 2010, pp. 80.1-80.11.
30. H. Zhang, C. Reardon, and L. E. Parker, "Real-time multiple human perception with color-depth cameras on a mobile robot, " IEEE Trans. Cybern., vol. 43, no. 5, pp. 1429-1441, Oct. 2013.
31. G. Willems, T. Tuytelaars, and L. Van Gool, "An efficient dense and scale-invariant spatio-temporal interest point detector, " in Proc. Eur. Conf. Comput. Vis., 2008, pp. 650-663.
32. K.-Y. Wong and R. Cipolla, "Extracting spatiotemporal interest points using global information, " in Proc. IEEE Int. Conf. Comput. Vis., Oct. 2007, pp. 1-8.
33. Z. Cheng, L. Qin, Y. Ye, Q. Huang, and Q. Tian, "Human daily action analysis with multi-view and color-depth data, " in Proc. Eur. Conf. Comput. Vis. Workshop, 2012, pp. 52-61.
34. B. Ni, G. Wang, and P. Moulin, "RGBD-HuDaAct: A color-depth video database for human daily activity recognition, " in Proc. IEEE Int. Conf. Comput. Vis. Workshop, Nov. 2011, pp. 1147-1153.
35. P. Scovanner, S. Ali, and M. Shah, "A 3-dimensional SIFT descriptor and its application to action recognition, " in Proc. IEEE Int. Conf. Multimedia Expo, Sep. 2007, pp. 357-360.
36. A. Klaser, M. Marszalek, and C. Schmid, "A spatio-temporal descriptor based on 3D-gradients, " in Proc. Brit. Mach. Vis. Conf., 2008, pp. 995-1004.
37. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2005, pp. 886-893.
38. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1-8.
39. L. Wang, Y. Qiao, and X. Tang, "Motionlets: Mid-level 3D parts for human motion recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 2674-2681.
40. L. Liu and L. Shao, "Learning discriminative representations from RGB-D video data, " in Proc. Int. Joint Conf. Artif. Intell., 2013, pp. 1493-1500.
41. F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, "Berkeley MHAD: A comprehensive multimodal human action database, " in Proc. IEEE Winter Conf. Appl. Comput. Vis., Jan. 2013, pp. 53-60.
42. O. Oreifej and Z. Liu, "HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 716-723.
43. X. Yang and Y. Tian, "Super normal vector for activity recognition using depth sequences, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 804-811.
44. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 3rd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, Aug. 2007.
45. V. Bloom, D. Makris, and V. Argyriou, "G3D: A gaming action dataset and real time action recognition evaluation framework, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshop, Jun. 2012, pp. 7-12.
46. S. R. Fanello, I. Gori, G. Metta, and F. Odone, "Keep it simple and sparse: Real-time action recognition, " J. Mach. Learn. Res., vol. 14, pp. 2617-2640, Sep. 2013.
47. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp. 1290-1297.
48. J. C. Niebles, H. Wang, and L. Fei-Fei, "Unsupervised learning of human action categories using spatial-temporal words, " Int. J. Comput. Vis., vol. 79, no. 3, pp. 299-318, Sep. 2008.
49. A. Vedaldi and A. Zisserman, "Efficient additive kernels via explicit feature maps, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 3, pp. 480-492, Mar. 2012.
50. D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent Dirichlet allocation, " J. Mach. Learn. Res., vol. 3, pp. 993-1022, Mar. 2003.