Human pose estimation and activity recognition from multi-view videos: Comparative explorations of recent developments

IEEE Journal on Selected Topics in Signal Processing

Volumn 6, Issue 5, 2012, Pages 538-552

  Holte, Michael B ^a   Tran, Cuong ^b   Trivedi, Mohan M ^b   Moeslund, Thomas B ^a  

^a AALBORG UNIVERSITY   (Denmark)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

3 D; comparative study; human action recognition; human pose estimation; i3DPost; INRIA Xmas Motion Acquisition Sequences (IXMAS); maker less; multi view; survey; view invariance; vision based; volumetric reconstruction

Indexed keywords

3-D; COMPARATIVE STUDIES; HUMAN POSE ESTIMATIONS; HUMAN-ACTION RECOGNITION; I3DPOST; MAKER-LESS; MOTION ACQUISITION; MULTI-VIEWS; VIEW-INVARIANCE; VISION BASED; VOLUMETRIC RECONSTRUCTION;

ANIMATION; GESTURE RECOGNITION; IMAGE RECOGNITION; SECURITY SYSTEMS; SURVEYS; THREE DIMENSIONAL; THREE DIMENSIONAL COMPUTER GRAPHICS;

MOTION ESTIMATION;

EID: 84865384447     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2012.2196975     Document Type: Article

References (102)

1. H. Lee and J. H. Kim, "An HMM-based threshold model approach for gesture recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, pp. 961-973, 1999. (Pubitemid 129304130)
2. A. Postawa, M. Kleinsorge, J. Krueger, and G. Seliger, "Automated image based recognition of manual work steps in the remanufacturing of alternators," Adv. Sustain. Manuf., vol. 5, pp. 209-214, 2011.
3. C. Rougier, J.Meunier, A. St-Arnaud, and J. Rousseau, "Fall detection from human shape and motion history using video surveillance," in Proc. 21st Int. Conf. Adv. Inf. Netw. Applicat. Workshops, 2007.
4. A. Alonso, R. Rosa, L. Val, M. Jimenez, and S. Franco, "A robot controlled by blinking for ambient assisted living," in Proc. 10th Int.Work- Conf. Artif. Neural Netw.: II: Distrib. Comput., Artif. Intell., Bioinf., Soft Comput., Ambient Assist. Living, 2009.
5. C. Tran and M. Trivedi, "Introducing XMOB: Extremity movement observation framework for upper body pose tracking in 3D," in Proc. IEEE Int. Symp. Multimedia, 2009, pp. 446-447.
6. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images," in Proc. Computer Comput. Vis. Pattern Recognit., 2011.
7. M. Trivedi and S. Cheng, "Holistic sensing and active displays for intelligent driver support systems," IEEE Comput. Mag., vol. 40, no. 5, pp. 60-68, May 2007.
8. E. Murphy-Chutorian and M. Trivedi, "Head pose estimation and augmented reality tracking: An integrated system and evaluation for monitoring driver awareness," IEEE Trans. Intell. Transport. Syst., vol. 11, no. 2, pp. 300-311, Jun. 2010.
9. C. Tran and M. Trivedi, "Driver assistance for 'keeping hands on the wheel and eyes on the road," in Proc. IEEE Int. Conf. Vehicular Electronics and Safety, 2009, pp. 97-101.
10. C. Tran, A. Doshi, and M. Trivedi, "Pedal errors prediction by driver foot gesture analysis: A vision-based inquiry," in Proc. IEEE Intell. Veh. Symp., 2011, pp. 577-582.
11. M. Trivedi, S. Cheng, E. Childers, and S. Krotosky, "Occupant posture analysis with stereo and thermal infrared video: Algorithms and experimental evaluation," IEEE Trans. Veh. Technol., Special Iss. In-Veh. Vis. Syst., vol. 53, no. 6, pp. 1698-1712, Nov. 2004.
12. S. Cheng and M. Trivedi, "Turn-intent analysis using body pose for intelligent driver assistance," IEEE Pervas. Comput., vol. 5, no. 4, pp. 28-37, Oct.-Dec. 2006. (Pubitemid 44717469)
13. J. Geigel and M. Schweppe, "Motion capture for realtime control of virtual actors in live, distributed, theatrical performances," in Proc. FG'11, 2011, pp. 774-779.
14. A. Alatan, Y. Yemez, U. Güdükbay, X. Zabulis, K. Müller, C. Erdem, C. Weigel, and A. Smolic, "Scene representation technologies for 3dtv - A survey," EEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1587-1605, Nov. 2007. (Pubitemid 350027674)
15. J. Radmer and J. Krueger, "Depth data-based capture of human movement for biomechanical application in clinical rehabilitation use," in Proc. 5th Int. Symp. Health Inform. Bioinformat., 2010, pp. 144-148.
16. L. Muendermann, S. Corazza, A. Chaudhari, T. Andriacchi, A. Sundaresan, and R. Chellappa, "Measuring human movement for biomechanical applications using markerless motion capture," in Proc. SPIE Three-Dimensional Image Capture Applicat., 2006.
17. Y. Chen, L. Smith, S. Hongwei, A. Pereira, and T. Smith, "Active information selection: Visual attention through the hands," IEEE Trans. Auton. Mental Develop., vol. 1, no. 2, pp. 141-151, Aug. 2009.
18. M. Trivedi, K. Huang, and I. Mikic, "Dynamic context capture and distributed video arrays for intelligent spaces," IEEE Trans. Syst., Man, Cybern., A, vol. 35, no. 1, pp. 145-163, Jan. 2005. (Pubitemid 40085391)
19. E. Murphy-Chutorian and M. Trivedi, "3D tracking and dynamic analysis of human head movements and attentional targets," in Proc. IEEE/ACM Int. Conf. Distrib. Smart Cameras, 2008, pp. 1-8.
20. A. Waibel, R. Stiefelhagen, R. Carlson, J. Casas, J. Kleindienst, L. Lamel, O. Lanz, D. Mostefa, M. Omologo, F. Pianesi, L. Polymenakos, G. Potamianos, J. Soldatos, G. Sutschet, and J. Terken, "Computers in the human interaction loop," in Handbook of Ambient Intelligence and Smart Environments. New York: Springer, 2010.
21. S. Park and M. Trivedi, "Understanding human interactions with track and body synergies (TBS) captured from multiple views," Comput. Vis. Image Understand., vol. 111, no. 1, pp. 2-20, 2008.
22. A. Utasi and C. Benedek, "A 3-D marked point process model for multi-view people detection," in Proc. Computer Comput. Vis. Pattern Recognit., 2011, pp. 3385-3392.
23. J. Assfalg, M. Bertini, C. Colombo, A. Bimbo, and W. Nunziati, "Semantic annotation of soccer videos: Automatic highlights identification," Comput. Vis. Image Understand., vol. 92, no. 2-3, pp. 285-305, 2003.
24. J. Kilner, J.-Y. Guillemaut, and A. Hilton, "3D action matching with key-pose detection," in Proc. Int. Conf. Comput. Vis. Workshops, 2009, pp. 1-8.
25. N. Werghi, "Segmentation and modeling of full human body shape from 3-D scan data: A survey," IEEE Trans. Syst., Man, Cyber. C, vol. 37, no. 6, pp. 1122-1136, Nov. 2007. (Pubitemid 350130706)
26. D. Fofi, T. Sliwa, and Y. Voisin, "A comparative survey on invisible structured light," Proc. SPIE, vol. 5303, pp. 90-98, 2004.
27. A. Kolb, E. Barth, R. Koch, and R. Larsen, "Time-of-flight sensors in computer graphics," in Proc. Eurographics - State of the Art Reports, 2009.
28. E. Stoykova, A. Alatan, P. Benzie, N. Grammalidis, S. Malasitis, J. Ostermann, S. Piekh, V. Sainov, C. Theobalt, T. Thevar, and X. Zabulis, "3-d time-varying scene capture technologies: A survey," EEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1568-1586, Nov. 2007.
29. N. Gkalelis, H. Kim, A. Hilton, N. Nikolaidis, and I. Pitas, "The i3Dpost multi-view and 3d human action/interaction database," in Proc. CVMP, 2009, pp. 159-168.
30. X. Ji and H. Liu, "Advances in view-invariant human motion analysis: A review," Trans. Syst., Man, Cybern. C, vol. 40, no. 1, pp. 13-24, Jan. 2010.
31. T. Moeslund and E. Granum, "A survey of computer vision-based human motion capture," Comput. Vis. Image Understand., vol. 81, no. 3, pp. 231-268, 2001. (Pubitemid 32415976)
32. T. Moeslund, A. Hilton, and V. Krüger, "A survey of advances in visionbased human motion capture and analysis," Comput. Vis. Image Understand., vol. 104, no. 2-3, pp. 90-126, 2006. (Pubitemid 44573873)
33. R. Poppe, "Vision-based human motion analysis: An overview," Comput. Vis. Image Understand., vol. 108, no. 1-2, pp. 4-18, 2007. (Pubitemid 47333391)
34. R. Poppe, "A survey on vision-based human action recognition," Image Vis. Comput., vol. 28, no. 6, pp. 976-990, 2010.
35. D. Weinland, R. Ronfard, and E. Boyer, "A survey of vision-based methods for action representation, segmentation and recognition," INRIA Rep. vol. RR-7212, pp. 54-111, 2010 [Online]. Available: http://hal.inria.fr/ inria-00459653/PDF/RR-7212.pdf
36. D. Weinland, R. Ronfard, and E. Boyer, "Free viewpoint action recognition using motion history volumes," Comput. Vis. Image Understand., vol. 104, no. 2, pp. 249-257, 2006. (Pubitemid 44573869)
37. S. Y. Cheng and M. M. Trivedi, "Articulated human body pose inference from voxel data using a kinematically constrained Gaussian mixture model," in Proc. Computer Comput. Vis. Pattern Recognit. Workshops, 2007.
38. L. Sigal and M. Black, "Guest editorial: State of the art in image and video based human pose and motion estimation," Int. J. Comput. Vis., vol. 87, pp. 1-3, 2010.
39. MuHAVi Dataset Instructions, [Online]. Available: http://dipersec. king. ac.uk/MuHAVi- MAS
40. B.-W. Hwang, S. Kim, and S.-W. Lee, "A fullbody gesture database for automatic gesture recognition," in Proc. FG'06, 2006 [Online]. Available: http://gesturedb.korea.ac.kr
41. L. Sigal and M. Black, "Humaneva: Synchronized video and motion capture dataset for evaluation of articulated human motion," 2006, Tech. Rep.
42. J. Starck and A. Hilton, "Surface capture for performance based animation," IEEE Comput. Graphics Applicat., vol. 27, no. 3, pp. 21-31, May-Jun. 2007. (Pubitemid 46767722)
43. M. Holte, T. Moeslund, N. Nikolaidis, and I. Pitas, "3D human action recognition for multi-view camera systems," in Proc. 3DIMPVT, 2011.
44. I. Mikic, M. M. Trivedi, E. Hunter, and P. Cosman, "Human body model acquisition and tracking using voxel data," Int. J. Comput. Vis., vol. 53, no. 3, pp. 199-223, 2003.
45. A. Sundaresan and R. Chellappa, "Model driven segmentation of articulating humans in Laplacian eigenspace," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 10, pp. 1771-1785, Oct. 2008.
46. Q. Delamarre and O. Faugeras, "3D articulated models and multiview tracking with physical forces," Comput. Vis. Image Understand., vol. 81, no. 3, pp. 328-357, 2001. (Pubitemid 32415980)
47. G. Cheung, S. Baker, and T. Kanade, "Shape-from-silhouette of articulated objects and its use for human body kinematic estimation and motion capture," in Proc. Comput. Comput. Vis. Pattern Recognit., 2003.
48. J. Ziegler, K. Nickel, and R. Stiefelhagen, "Tracking of the articulated upper body on multi-view stereo image sequences," in Proc. Comput. Vis. Pattern Recognit., 2006.
49. F. Caillette, A. Galata, and T. Howard, "Real-time 3-D human body tracking using learnt models of behaviour," Comput. Vis. Image Understand., vol. 109, pp. 112-125, 2008.
50. O. Bernier, P. Cheung-Mon-Chana, and A. Bougueta, "Fast nonparametric belief propagation for real-time stereo articulated body tracking," Comput. Vis. Image Understand., vol. 113, no. 1, pp. 29-47, 2009.
51. J. Gall, B. Rosenhahn, T. Brox, and H. Seidel, "Optimization and filtering for human motion capture: A multi-layer framework," Int. J. Comput. Vis., vol. 87, no. 1-2, pp. 75-92, 2010.
52. S. Corazza, L. Mundermann, E. Gambaretto, G. Ferrigno, and T. Andriacchi, "Markerless motion capture through visual hull, articulated ICP and subject specific model generation," Int. J. Comput. Vis., vol. 87, no. 1-2, pp. 156-169, 2010.
53. R. Li, T. Tian, S. Sclaroff, and M. Yang, "3d human motion tracking with a coordinated mixture of factor analyzers," Int. J. Comput. Vis., vol. 87, no. 1-2, 2010.
54. M. Hofmann and D. Gavrila, "Multi-view 3D human pose estimation in complex environment," Int. J. Comput. Vis., 2011.
55. S. Y. Cheng and M. M. Trivedi, "Multimodal voxelization and kinematically constrained Gaussian mixture model for full hand pose estimation: An integrated systems approach," in Proc. Int. Conf. Comput. Vis. Syst., 2006.
56. G. Cheung and T. Kanade, "A real-time system for robust 3d voxel reconstruction of human motions," in Proc. Comput. Vis. Pattern Recognit., 2000.
57. Z. Husz and A. Wallace, "Evaluation of a hierarchical partitioned particle filter with action primitives," in Proc. Comput. Vis. Pattern Recognit. Workshop, 2007.
58. D. Knossow, R. Ronfard, and R. Horaud, "Human motion tracking with a kinematic parameterization of extremal contours," Int. J. Comput. Vis., vol. 79, no. 3, pp. 247-269, 2008.
59. R. Poppe, "Evaluating example-based pose estimation: Experiments on the humaneva sets," in Proc. Comput. Vis. Pattern Recognit. Workshops, 2007.
60. G. Slabaugh, B. Culbertson, and T. Malzbender, "A survey of methods for volumetric scene reconstruction for photographs," in Proc. Int. Conf. Vol. Graphics, 2001.
61. C. Tran and M. Trivedi, "Hand modeling and tracking from voxel data: An integrated framework with automatic initialization," in Proc. IEEE Int. Conf. Pattern Recognit., 2008.
62. A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly, "Vision- based hand pose estimation: A review," Comput. Vis. Image Understand., vol. 108, no. 1-2, 2007.
63. E. Murphy-Chutorian and M. Trivedi, "Head pose estimation in computer vision: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 4, pp. 607-626, Apr. 2009.
64. K. Huang and M. Trivedi, "3D shape context based gesture analysis integrated with tracking using omni video array," in Proc. Comput. Comput. Vis. Pattern Recognit. Workshops, 2005.
65. M. Ahmad and S.-W. Lee, "HMM-based human action recognition using multiview image sequences," in Proc. Int. Conf. Pattern Recognit., 2006.
66. C. Canton-Ferrer, J. Casas, and M. Pardás, "Human model and motion based 3d action recognition in multiple view scenarios," in Proc. EUSIPCO, 2006.
67. M. Pierobon, M. Marcon, A. Sarti, and S. Tubaro, "3-D body posture tracking for human action template matching," in Proc. Int. Conf. Acoust., Speech, Signal Process., 2006, pp. 501-504.
68. F. Lv and R. Nevatia, "Single view human action recognition using key pose matching and viterbi path searching," in Proc. Comput. Vis. Pattern Recognit., 2007.
69. D. Weinland, R. Ronfard, and E. Boyer, "Action recognition from arbitrary views using 3d exemplars," in Proc. Int. Conf. Comput. Vis., 2007.
70. S. Cherla, K. Kulkarni, A. Kale, and V. Ramasubramanian, "Towards fast, view-invariant human action recognition," in Computer Comput. Vis. Pattern Recognit. Workshops, 2008.
71. A. Farhadi and M. Tabrizi, "Learning to recognize activities from the wrong view point," in Proc. Eur. Conf. Comput. Vis., 2008.
72. I. Junejo, E. Dexter, I. Laptev, and P. Pérez, "Cross-view action recognition from temporal self-similarities," in Proc. Eur. Conf. Comput. Vis., 2008.
73. J. Liu, S. Ali, and M. Shah, "Recognizing human actions using multiple features," in Computer Comput. Vis. Pattern Recognit., 2008.
74. J. Liu and M. Shah, "Learning human actions via information maximization," in Computer Comput. Vis. Pattern Recognit., 2008.
75. R. Souvenir and J. Babbs, "Learning the viewpoint manifold for action recognition," in Computer Comput. Vis. Pattern Recognit., 2008.
76. D. Tran and A. Sorokin, "Human activity recognition with metric learning," in Proc. Eur. Conf. Comput. Vis., 2008.
77. P. Turaga, A. Veeraraghavan, and R. Chellappa, "Statistical analysis on Stiefel and Grassmann manifolds with applications in computer vision," in Computer Comput. Vis. Pattern Recognit., 2008.
78. S. Vitaladevuni, V. Kellokumpu, and L. Davis, "Action recognition using ballistic dynamics," in Computer Comput. Vis. Pattern Recognit., 2008.
79. P. Yan, S. Khan, and M. Shah, "Learning 4d action feature models for arbitrary view action recognition," in Computer Comput. Vis. Pattern Recognit., 2008.
80. N. Gkalelis, N. Nikolaidis, and I. Pitas, "Viewindepedent human movement recognition from multi-view video exploiting a circular invariant posture representation," in ICME, 2009.
81. K. Reddy, J. Liu, and M. Shah, "Incremental action recognition using feature-tree," in Proc. Int. Conf. Comput. Vis., 2009.
82. P. Huang, A. Hilton, and J. Starck, "Shape similarity for 3D video sequences of people," Int. J. Comput. Vis., vol. 89, pp. 362-381, 2010.
83. A. Iosifidis, N. Nikolaidis, and I. Pitas, "Movement recognition exploiting multi-view information," in Proc. Multimedia Signal Process., 2010.
84. D. Weinland, M. Özuysal, and P. Fua, "Making action recognition robust to occlusions and viewpoint changes," in Proc. Eur. Conf. Comput. Vis., 2010.
85. A. Haq, I. Gondal, and M. Murshed, "On dynamic scene geometry for view-invariant action matching," in Proc. Comput. Vis. Pattern Recognit., 2011.
86. I. Junejo, E. Dexter, I. Laptev, and P. Pérez, "View-independent action recognition from temporal self-similarities," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 1, pp. 172-185, Jan. 2011.
87. J. Liu, M. Shah, B. Kuipers, and S. Savarese, "Cross-view action recognition via view knowledge transfer," in Proc. Comput. Comput. Vis. Pattern Recognit., 2011.
88. S. Pehlivan and P. Duygulu, "A new pose-based representation for recognizing actions from multiple cameras," Comput. Vis. Image Understand., vol. 115, pp. 140-151, 2011.
89. Y. Song, D. Demirdjian, and R. Davis, "Multi-signal gesture recognition using temporal smoothing hidden conditional random fields," in Proc. FG'11, 2011.
90. P. Matikainen, P. Pillai, L. Mummert, R. Sukthankar, and M. Hebert, "Prop-free pointing detection in dynamic cluttered environments," in Proc. FG'11, 2011.
91. P. Fihl and T. B. Moeslund, "Invariant gait continuum based on the duty-factor," Signal Image Video Process., vol. 3, no. 4, pp. 391-402, 2008.
92. A. Johnson and M. Hebert, "Using spin images for efficient object recognition in cluttered 3d scenes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 5, pp. 433-449, May 1999.
93. R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin, "Shape distributions," ACM Trans. Graph., vol. 21, pp. 807-832, 2002.
94. M. Ankerst, G. Kastenmüller, H.-P. Kriegel, and T. Seidl, "3D shape histograms for similarity search and classification in spatial databases," in Proc. Int Symp. Spatial Datbases, 1999.
95. S. Belongie, J. Malik, and J. Puzicha, "Shape matching and object recognition using shape contexts," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 4, pp. 509-522, Apr. 2002. (Pubitemid 34448769)
96. M. Körtgen, M. Novotni, and R. Klein, "3D shape matching with 3D shape contexts," in Proc. Central Eur. Seminar Comput. Graph., 2003.
97. M. Kazhdan, T. Funkhouser, and S. Rusinkiewicz, "Rotation invariant spherical harmonic representation of 3D shape descriptors," in Proc. SIGGRAPH, 2003.
98. P. Huang and A. Hilton, "Shape-colour histograms for matching 3D video sequences," in Proc. 3DIM, 2009.
99. I. Cohen and H. Li, "Inference of human postures by classification of 3D human body shape," in Proc. Int.Workshop Anal. Modeling of Faces and Gestures, 2003.
100. C. Tran and M. Trivedi, "Human body modeling and tracking using volumetric representation: Selected recent studies and possibilities for extensions," in Proc. ACM Workshops, 2008.
101. R. Gross and J. Shi, "The CMU Motion of Body (MoBo) Database," 2001, Tech. Rep.
102. M. Trivedi, "Human movement capture and analysis in intelligent environments," Mach. Vis. Applicat., vol. 14, no. 4, pp. 215-217, 2003. (Pubitemid 40877358)