Semantic human activity recognition: A literature review

Pattern Recognition

Volumn 48, Issue 8, 2015, Pages 2329-2345

  Ziaeefard, Maryam ^a   Bergevin, Robert ^a  

^a UNIVERSITÉ LAVAL   (Canada)

Author keywords

Attribute; Human activity recognition; Human object interaction; Pose; Poselet; Scene; Survey

Indexed keywords

PATTERN RECOGNITION; SURVEYING;

ATTRIBUTE; HUMAN ACTIVITY RECOGNITION; HUMAN-OBJECT INTERACTION; POSE; POSELET; SCENE;

SEMANTICS;

EID: 84928273573     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2015.03.006     Document Type: Article

References (118)

1. C. Cedras, and M. Shah Motion-based recognition a survey Image Vis. Comput. 13 1995 129 155
2. J. Aggarwal, and Q. Cai Human motion analysis a review Comput. Vis. Image Underst. 73 1999 428 440
3. D. Gavrila The visual analysis of human movement a survey Comput. Vis. Image Underst. 73 1999 82 98
4. P. Turaga, R. Chellappa, V. Subrahmanian, and O. Udrea Machine recognition of human activities a survey IEEE Trans. Circuits Syst. Video Technol. 18 2008 1473 1488
5. R. Poppe A survey on vision-based human action recognition Image Vis. Comput. J. 28 2010 976 990
6. J.K. Aggarwal, and M.S. Ryoo Human activity analysis a review ACM Comput. Surv. 43 2011 1 43
7. G. Guo, and A. Lai A survey on still image based human action recognition Pattern Recognit. 47 2014 3343 3361
8. P.E. Downing, Y. Jiang, M. Shuman, and N. Kanwisher A cortical area selective for visual processing of the human body Science 293 2001 2470 2473
9. M.V. Peelen, and P.E. Downing Selectivity for the human body in the fusiform gyrus J. Neurophysiol. 93 2005 603 608
10. R.F. Schwarzlose, C.I. Baker, and N.G. Kanwisher Separate face and body selectivity on the fusiform gyrus J. Neurosci. 25 2005 11055 11059
11. K. Allan Linguistic Meaning, vol. 1 1986 Routledge & Kegan Paul New York
12. V. Gallese, L. Fadiga, L. Fogassi, and G. Rizzolatti Action recognition in the premotor cortex Brain J. 2 1996 593 609
13. A.G. Huth, S. Nishimoto, A.T. Vu, and J.L. Gallant A continuous semantic space describes the representation of thousands of object and action categories across the human brain Neuron 76 2012 1210 1224
14. S. Park, J.K. Aggarwal, Recognition of human interaction using multiple features in grayscale images, in: Proceedings of the International Conference on Pattern Recognition, 2000, pp. 51-54.
15. N. Ikizler, P. Duygulu, Human action recognition using distribution of oriented rectangular patches, in: Human Motion ICCV, 2007, pp. 271-284.
16. D. Forsyth, M. Fleck, Body plans, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1997, pp. 678-683.
17. Y. Yang, D. Ramanan, Articulated pose estimation with flexible mixtures of parts, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
18. N. Yukita, Iterative action and pose recognition using global-and-pose features and action-specific models, in: IEEE Internat. Conf. on Computer Vision Workshops (ICCVW), 2013, pp. 476-483.
19. M. Andriluka, L. Sigal, M.J. Black, Benchmark datasets for pose estimation and tracking, in: Moesland, Hilton, Krüger, Sigal (Eds.), Visual Analysis of Humans: Looking at People, 2011, pp. 253-274.
20. J.A. Webb, and J.K. Aggarwal Visually interpreting the motion of objects in space IEEE Comput. 1981 40 46
21. F.J. Perales, J. Torres, A system for human motion matching between synthetic and real image based on a biomechanic graphical model, in: Proceedings of the IEEE Computer Society Workshop on Motion of Non-Rigid and Articulated Objects, 1994, pp. 83-88.
22. A. Shio, J. Sklansky, Segmentation of people in motion, in: Proceedings of the IEEE Workshop on Visual Motion, 1991, pp. 325-332.
23. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake, Real-time human pose recognition in parts from single depth images, in: IEEE International Conference on Computer Vision (ICCV), 2011, pp. 1297-1304.
24. A. Yao, J. Gall, G. Fanelli, L.V. Gool, Does human action recognition benefit from pose estimation? in: Proceedings of the British Machine Vision Conference (BMVC), vol. 67, 2011, pp. 1-11.
25. L. Bourdev, J. Malik, Poselets: body part detectors trained using 3D human pose annotations, in: IEEE International Conference on Computer Vision (ICCV), 2009, pp. 1365-1372.
26. L. Bourdev, S. Maji, T. Brox, J. Malik, Detecting people using mutually consistent poselet activations, in: European Conference on Computer Vision (ECCV), 2010, pp. 168-181.
27. S. Maji, L. Bourdev, J. Malik, Action recognition from a distributed representation of pose and appearance, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 3177-3184.
28. S. Maji, J. Malik, Object detection using a max-margin hough transform, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 1038-1045.
29. Y. Wang, D. Tran, Z. Liao, Learning hierarchical poselets for human parsing, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 1705-1712.
30. B. Holt, E.J. Ong, H. Cooper, R. Bowden, Putting the pieces together: connected poselets for human pose estimation, in: International Conference on Computer Vision Workshops (ICCV Workshops), 2011, pp. 1196-1201.
31. L. Breiman Random forests Mach. Learn. 45 2001 5 32
32. L. Pishchulin, M. Andriluka, P. Gehler, B. Schiele, Poselet conditioned pictorial structures, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 588-595.
33. A. Klaser, M. Marszek, I. Laptev, C. Schmid, Will Person Detection Help Bag-of-Features Action Recognition?, Research Report, No. RR-7373, 2010.
34. L. Vaina, and M. Jaulent Object structure and action requirements a compatibility model for functional recognition Int. J. Intel. Syst. 6 1991 313 336
35. A. Farhadi, I. Endres, D. Hoiem, D. Forsyth, Describing objects by their attributes, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 1778-1785.
36. L. Bourdev, S. Maji, J. Malik, Describing people: a poselet-based approach to attribute classification, in: IEEE International Conference on Computer Vision (ICCV), 2011, pp. 1543-1550.
37. J. Liu, B. Kuipers, S. Savarese, Recognizing human actions by attributes, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 3337-3344.
38. C.H. Lampert, H. Nickisch, S. Harmeling, Learning to detect unseen object classes by between-class attribute transfer, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 951-958.
39. T.B. Moeslund, and E. Granum A survey of computer vision-based human motion capture Comput. Vis. Image Underst. 81 2001 231 268
40. P.F. Felzenszwalb, and D.P. Huttenlocher Pictorial structures for object recognition Int. J. Comput. Vis. 61 2005 55 79
41. D. Ramanan, Learning to parse images of articulated bodies, in: Neural Information Processing Systems (NIPS), 2006, pp. 1129-1136.
42. G. Shakhnarovich, P. Viola, T. Darrell, Fast pose estimation with parameter sensitive hashing, in: IEEE International Conference on Computer Vision (ICCV), 2001, pp. 750-757.
43. J. Sullivan, S. Carlsson, Recognizing and tracking human action, in: Seventh European Conference on Computer Vision (ECCV), 2002, pp. 629-644.
44. J. Starck, A. Hilton, Spherical matching for temporal correspondence of non-rigid surfaces, in: IEEE International Conference on Computer Vision (ICCV), 2005, pp. 1387-1394.
45. R. Kehl, M. Bray, L. VanGool, Full body tracking from multiple views using stochastic sampling, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005, pp. 129-136.
46. M.B. Holte, C. Tran, M.M. Trivedi, and T.B. Moeslund Human pose estimation and activity recognition from multi-view videos comparative explorations of recent developments IEEE J. Select. Top. Signal Proc. 6 2011 538 552
47. K. Raja, I. Laptev, P. Perez, L. Oisel, Joint pose estimation and action recognition in image graphs, in: 18th IEEE International Conference on Image Processing (ICIP), 2011, pp. 25-28.
48. S. Mukherjee, S.K. Biswas, and D.P. Mukherjee Recognizing interactions between human performers by 'Dominating Pose Doublet' J. Mach. Vis. Appl. 25 2014 1033 1052
49. B. Yao, L. Fei-Fei, Action recognition with exemplar based 2.5D graph matching, in: European Conference on Computer Vision (ECCV), 2012, pp. 173-186.
50. C. Wang, Y. Wang, A.L. Yuille, An approach to pose-based action recognition, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 915-922.
51. L. Meng, L. Qing, P. Yang, J. Miao, X. Chen, D.N. Metaxas, Activity recognition based on semantic spatial relation, in: IEEE International Conference on Pattern Recognition (ICPR), 2012, pp. 609-612.
52. A. Eweiwi, S. Cheema, C. Thurau, C. Bauckhage, Temporal key poses for human action recognition, in: The IEEE International Conference on Computer Vision Workshops (ICCVW), 2011, pp. 1310-1317.
53. A. Bobick, and J. Davis The recognition of human movement using temporal templates IEEE Trans. Pattern Recognit. Mach. Intel. 23 2001 257 267
54. A.A. Chaaraoui, P.C. Prez, and F.F. Revuelta Silhouette-based human action recognition using sequences of key poses Pattern Recognit. Lett. 2013 1799 1807
55. S. Cheema, A. Eweiwi, C. Thurau, C. Bauckhage, Action recognition by learning discriminative key poses, The IEEE International Conference on Computer Vision Workshops (ICCVW), 2011, pp. 1302-1309.
56. A. Vahdat, B. Gao, M. Ranjbar, G. Mori, A discriminative key pose sequence model for recognizing human interactions, in: IEEE International Workshop on Visual Surveillance, 2011, pp. 1729-1736.
57. F. Lv, R. Nevatia, Single view human action recognition using key pose matching and viterbi path searching, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1-8.
58. S. Park, and J.K. Aggarwal A hierarchical Bayesian network for event recognition of human actions and interactions J. Multimed. Syst. 10 2004 164 179
59. F.S. Khan, J. van de Weijer, R.M. Anwer, M. Felsberg, and C. Gatta Semantic pyramids for gender and action recognition IEEE Trans. Image Process. 23 2014 3633 3645
60. W. Yang, Y. Wang, G. Mori, Recognizing human actions from still images with latent poses, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 2030-2037.
61. Y. Zheng, Y.J. Zhang, X. Li, B.D. Liu, Action recognition in sill images using a combination of human pose and context information, in: 19th IEEE International Conference on Image Processing (ICIP), 2012, pp. 785-788.
62. M. Raptis, L. Sigal, Poselet key-framing: a model for human activity recognition, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 2650-2657.
63. M. Nabi, A.D. Bue, V. Murino, Temporal poselets for collective activity detection and recognition, The IEEE International Conference on Computer Vision Workshops (ICCVW), 2013, pp. 500-507.
64. J. Wang, X. Nie, Y. Xia, Y. Wu, Mining discriminative 3D poselet for cross-view action recognition, in: Applications of Computer Vision (WACV), 2014, pp. 634-639.
65. C.Y. Chen, K. Grauman, Watching unlabeled video helps learn new human actions from very few labeled snapshots, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 572-579.
66. M. Marszalek, I. Laptev, C. Schmid, Actions in context, in: IEEE Conference on Computer, Vision and Pattern Recognition (CVPR), 2009, pp. 2929-2936.
67. Y. Zhang, W. Qu, D. Wang, Action-scene model for human action recognition from videos, in: Second AASRI Conference on Computational Intelligence and Bioinformatics, vol. 6, 2014, pp. 111-117.
68. J. Liu, H. Xiang, Y. Shi, D. Yu, Action recognition with trajectory and scene, in: International Conference on Digital Home (ICDH), 2012, pp. 63-68.
69. M.M. Ullah, S.N. Parizi, I. Laptev, Improving bag-of-features action recognition with non-local cues, in: Proceedings of the British Machine Vision Conference (BMVC), 2010, pp. 1-11.
70. S. Jones, L. Shao, Unsupervised spectral dual assignment clustering of human actions in context, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 604-611.
71. A. Gupta, L.S. Davis, Objects in action: an approach for combining action understanding and object perception, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1-8.
72. A. Gupta, A. Kembhavi, L.S. Davis, Observing human-object interactions: using spatial and functional compatibility for recognition, IEEE Trans. Pattern Anal. Mach. Intel. 31 (10) (2009) 1775-1789.
73. J. Wu, A. Osuntogun, T. Choudhury, M. Philipose, J. Rehg, A scalable approach to activity recognition based on object use, in: Proceedings. IEEE International Conference on Computer Vision (ICCV), 2007, pp. 1-8.
74. R. Filipovych, E. Ribeiro, Recognizing primitive interactions by exploring actor-object states, in: Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.
75. Y. Kuniyoshi, M. Shimozaki, A self-organizing neural model for context based action recognition, in: IEEE Engineering in Medicine and Biology Society Conference on Neural Engineering, 2003, pp. 442-445.
76. V. Delaitre, J. Sivic, I. Laptev, Learning person-object interactions for action recognition in still images, in: Neural Information Processing Systems (NIPS), 2011, pp. 1503-1511.
77. C. Desai, D. Ramanan, C. Fowlkes, Discriminative models for static human-object interactions, in: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, pp. 9-16.
78. B. Yao, L. Fei-Fei, Grouplet: A structured image representation for recognizing human and object interactions, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 9-16.
79. B. Yao, L. Fei-Fei, Modeling mutual context of object and human pose in human-object interaction activities, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 17-24.
80. L.J. Li, L. Fei-Fei, What, where and who? Classifying events by scene and object recognition, in: IEEE International Conference on Computer Vision (ICCV), 2007, pp 1-8.
81. N. Ikizler-Cinbis, S, Sclaroff, Object, scene and actions: combining multiple features for human action recognition, in: European Conference on Computer Vision (ECCV), vol. 6311, 2010, pp. 494-507.
82. D. Han, L. Bo, C. Sminchisescu, Selection and context for action recognition, in: IEEE International Conference on Computer Vision (ICCV), 2009, pp.1933-1940.
83. G. Sharma, F. Jurie, C. Schmid, Expanded parts model for human attribute and action recognition in still images, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 652-659.
84. W. Li, N. Vasconcelos, Recognizing activities by attribute dynamics, in: Advances in Neural Information Processing Systems (NIPS), 2012, pp. 1115-1123.
85. Q. Qiu, Z. Jiang, R. Chellappa, Sparse dictionary-based representation and recognition of action attributes, in: IEEE International Conference on Computer Vision (ICCV), 2011, pp. 707-714.
86. Z. Zhang, C. Wang, B. Xiao, W. Zhou, S. Liu, Attribute regularization based human action recognition, in: IEEE Transactions on Information Forensics and Security, 2013, pp. 1600-1609.
87. M. Rohrbach, M. Regneri, M. Andriluka, S. Amin, M. Pinkal, B. Schiele, Script data for attribute-based recognition of composite activities, in: European Conference on Computer Vision (ECCV), 2012, pp. 144-157.
88. B. Yao, X. Jiang, A. Khosla, A.L. Lin, L. Guibas, L. Fei-Fei, Human action recognition by learning bases of action attributes and parts, in: IEEE International Conference on Computer Vision (ICCV), 2011, pp. 1331-1338.
89. M.S. Ryoo, J.K. Aggarwal, Hierarchical recognition of human activities interacting with objects, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-8.
90. M.S. Ryoo, J.K. Aggarwal, UT-Interaction Dataset. ICPR Contest on Semantic Description of Human Activities (SDHA), 2010.
91. V. Delaitre, I. Laptev, J. Sivic, Recognizing human actions in still images: a study of bag-of-features and part-based representations, in: Proceedings of the British Machine Vision Conference (BMVC), 2010, pp. 1-11.
92. M. Everingham, L.V. Gool, C.K.I. Williams, J. Winn, and A. Zisserman The Pascal visual object classes (VOC) challenge Int. J. Comput. Vis. 88 2010 303 338
93. A. Yao, J. Gall, L. Van Gool, A hough transform-based voting framework for action recognition, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 2061-2068.
94. M.S. Ryoo, J.K. Aggarwal, Spatio-temporal relationship match: video structure comparison for recognition of complex human activities, in: IEEE International Conference on Computer Vision (ICCV), 2009, pp. 1593-1600.
95. P. Matikainen, M. Hebert, R. Sukthankar, Representing pairwise spatial and temporal relations for action recognition, in: European Conference on Computer Vision (ECCV), 2010, pp. 508-521.
96. Y. Kong, Y. Jia, Y. Fu, Learning human interaction by interactive phrases, in: European Conference on Computer Vision (ECCV), 2012, pp. 300-313.
97. S. Lazebnik, C. Schmid, J. Ponce, Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006, pp. 2169-2178.
98. F.S. Khan, J. van de Weijer, A. Bagdanov, M. Felsberg, Scale coding bag-of-words for action recognition, in: IEEE International Conference on Pattern Recognition (ICPR), 2014.
99. F.S. Khan, R.M. Anwer, J. van de Weijer, A. Bagdanov, A. Lopez, and M. Felsberg Coloring action recognition in still images Int. J. Comput. Vis. 105 2013 205 221
100. S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, K. Saenko, YouTube2Text: recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition, in: The IEEE International Conference on Computer Vision (ICCV), 2013, pp. 2712-2719.
101. T.S. Motwani, R.J. Mooney, Improving video activity recognition using object recognition and text mining, in: Proceedings of the 20th European Conference on Artificial Intelligence (ECAI), 2012, pp. 600-605.
102. S. Chen, J. Liu, H. Wang, J.C. Augusto, A hierarchical human activity recognition framework based on automated reasoning, in: Proceedings of the 2013 IEEE International Conference on Systems, Man, and Cybernetics, 2013, pp. 3495-3499.
103. G.J. Wang, and H.J. Zhou Introduction to Mathematical Logic and Resolution Principle 2nd ed. 2009 Alpha Science International Limited Oxford
104. S. Chen, K. Clawson, M. Jing, J. Liu, H. Wang, B. Scotney, Uncertainty reasoning based formal framework for big video data understanding, in: Proceedings of IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technologies, 2014, pp. 487-494.
105. X. Liu, K. Clawson, H. Wang, B. Scotney, J. Liu, Complex event recognition with uncertainty reasoning, in: Proceedings of the International Conference on Machine Learning and Cybernetics, 2013, pp. 1823-1828.
106. M.S. Ryoo, J.K. Aggarwal, Semantic representation and recognition of continued and recursive human activities. Int. J. Comput. Vis. (IJCV) 82(1)(2009) 1-24.
107. K. Ramirez-Amaro, E.S. Kim, J. Kim, B.T. Zhang, M. Beetz,G. Cheng, Enhancing human action recognition through spatio-temporal feature learning and semantic rules, in: IEEE-RAS International Conference, 2013.
108. M. Palatucci, D. Pomerleau, G. Hinton, T. Mitchell, Zero-shot learning with semantic output codes, in: Neural Information Processing Systems (NIPS), 2009, pp. 1410-1418.
109. H.T. Cheng, F.T. Sun, M.L. Griss, P. Davis, J. Li, D. You, NuActiv: recognizing unseen new activities using semantic attribute-based learning, in: Proceedings of the ACM International Conference on Mobile Systems, Applications, and Services (MobiSys), 2013, pp. 361-374.
110. M.S. Ryoo, Human activity prediction: early recognition of ongoing activities from streaming videos, in: The IEEE International Conference on Computer Vision (ICCV), 2011, pp. 1036-1043.
111. K. Li, Y. Fu, ARMA-HMM: a new approach for early recognition of human activity, in: IEEE Conference on Pattern Recognition (ICPR), 2012, pp. 1779-1782.
112. M. Hoai, F. Torre, Max-margin early event detectors, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 2863-2870.
113. Y. Cao, D. Barrett, A. Barbu, S. Narayanaswamy, Y. Haonan, A. Michaux, L. Yuewei, S. Dickinson, J.M. Siskind, W. Song, Recognizing human activities from partially observed videos, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 2658-2665.
114. D. Munoz, J.A. Bagnell, M. Hebert, Stacked hierarchical labeling, in: European Conference on Computer Vision (ECCV), 2010, pp. 57-70.
115. D. Munoz, J.A. Bagnell, M. Hebert, Co-inference machines for multimodal scene analysis, in: European Conference on Computer Vision (ECCV), 2012, pp. 668-681.
116. C. Baker, R. Saxe, and J. Tenenbaum Action understanding as inverse planning Cognition 13 2009 329 349
117. K.M. Kitani, B.D. Ziebart, J.A. Bagnell, M. Hebert, Activity forecasting, in: European Conference on Computer Vision (ECCV), vol. 7575, 2012, pp. 201-214.
118. L. Wu, J. Zhang, F. Yan, A poselet based key frame searching approach in sports training videos, in: Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012, pp. 1-4.