Human Pose Estimation from Video and IMUs

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 8, 2016, Pages 1533-1547

  Von Marcard, Timo ^a   Pons Moll, Gerard ^b   Rosenhahn, Bodo ^a  

^a LEIBNIZ UNIVERSITY HANNOVER   (Germany)

^b MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

Author keywords

animation; Human pose estimation; IMU; inertial sensors; motion capture; multisensor fusion

Indexed keywords

ANIMATION; INERTIAL NAVIGATION SYSTEMS; MOTION ANALYSIS; SENSOR DATA FUSION; VIDEO RECORDING;

ACCURATE ESTIMATION; COMPLEMENTARY DATA; CONTINUOUS OPERATION; HUMAN MOTION CAPTURE; HUMAN POSE ESTIMATIONS; INERTIAL SENSOR; MOTION CAPTURE; POSITION INFORMATION;

MOTION ESTIMATION;

ALGORITHM; BODY POSITION; HUMAN; MOTION; VIDEORECORDING;

ALGORITHMS; HUMANS; MOTION; POSTURE; VIDEO RECORDING;

EID: 84978427623     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2522398     Document Type: Article

References (58)

1. T. Moeslund, A. Hilton, and V. Krüger, "A survey of advances in vision-based human motion capture and analysis," Comput. Vis. Image Understanding, vol. 104, no. 2, pp. 90-126, 2006.
2. A. Baak, B. Rosenhahn, M. Müller, and H. Seidel, "Stabilizing motion tracking using retrieved motion priors," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1428-1435.
3. A. O. Balan, L. Sigal, M. J. Black, J. E. Davis, and H. W. Haussecker, "Detailed human shape and pose from images," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
4. L. Herda, R. Urtasun, and P. Fua, "Implicit surface joint limits to constrain video-based motion capture," in Proc. Eur. Conf. Comput. Vis., 2004, vol. 3022, pp. 405-418.
5. S. Ioffe and D. Forsyth, "Human tracking with mixtures of trees," in Proc. IEEE Int. Conf. Comput. Vis., 2001, vol. 1, pp. 690-695.
6. H. Sidenbladh, M. Black, and D. Fleet, "Stochastic tracking of 3d human figures using 2d image motion," in Proc. Eur. Conf. Comput. Vis., 2000, vol. 1843, pp. 702-718.
7. G. Shakhnarovich, P. Viola, and T. Darrell, "Fast pose estimation with parameter-sensitive hashing," in Proc. IEEE Int. Conf. Comput. Vis., 2003, pp. 750-757.
8. J. Gall, A. Yao, and L. Van Gool, "2D action recognition serves 3D human pose estimation," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 425-438.
9. A. Elgammal and C. Lee, "Inferring 3D body pose from silhouettes using activity manifoldlearning," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2004, vol. 2, pp. II-681-II-688.
10. C. Sminchisescu, A. Kanaujia, Z. Li, and D. Metaxas, "Discriminative density propagation for 3D human motion estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2005, vol. 1, p. 390.
11. L. Bo and C. Sminchisescu, "Twin Gaussian processes for structured prediction," Int. J. Comput. Vis., vol. 87, pp. 28-52, 2010.
12. C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, "Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 7, pp. 1325-1339, 2014.
13. G. Pons-Moll, D. J. Fleet, and B. Rosenhahn, "Posebits for monocular human pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2014, pp. 2345-2352.
14. R. Urtasun, D. J. Fleet, and P. Fua, "3D people tracking with Gaussian process dynamical models," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2006, pp. 238-245.
15. J. Wang, D. Fleet, and A. Hertzmann, "Gaussian process dynamical models for human motion," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 2, pp. 283-298, Feb. 2008.
16. D. Brubaker M. A., Fleet and A. Hertzmann, "Physics-based person tracking using the anthropomorphic walker," Int. J. Comput. Vis., vol. 87, no. 1-2, pp. 140-155, 2010.
17. P. Guan, A. Weiss, A. Balan, and M. Black, "Estimating human shape and pose from a single image," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1381-1388.
18. L. Sigal, M. Vondrak, and O. Jenkins, "Physical simulation for probabilistic motion tracking," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
19. E. de Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun, "Performance capture from sparse multi-view video," ACM Trans. Graph. (TOG), vol. 27, no. 3, pp. 1-98, 2008.
20. J. Gall, C. Stoll, E. de Aguiar, C. Theobalt, B. Rosenhahn, and H. Seidel, "Motion capture using joint skeleton tracking and surface estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1746-1753.
21. D. Roetenberg, "Inertial and magnetic sensing of human motion," Univ. Twente, 2006.
22. G. Pons-Moll, A. Baak, T. Helten, M. Müller, H.-P. Seidel, and B. Rosenhahn, "Multisensor-fusion for 3D full-body human motion capture," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 663-670.
23. (2016). Multimodal human motion database TNT15 [Online]. Available: http://www.tnt.uni-hannover.de/project/TNT15/
24. L. Sigal, A. Balan, and M. Black, "Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion," Int. J. Comput. Vis., vol. 87, no. 1, pp. 4-27, 2010.
25. M. Andriluka, S. Roth, and B. Schiele, "Monocular 3d pose estimation and tracking by detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 623-630.
26. V. Belagiannis, X. Wang, B. Schiele, P. Fua, S. Ilic, and N. Navab, "Multiple human pose estimation with temporally consistent 3D pictorial structures," in Proc. Eur. Conf. Comput. Vis., ChaLearn Looking People Workshop, 2014, pp. 742-754.
27. J. Gall, B. Rosenhahn, T. Brox, and H. Seidel, "Optimization and filtering for human motion capture," Int. J. Comput. Vis., vol. 87, pp. 75-92, 2010.
28. W. Zhang, L. Shang, and A. Chan, "A robust likelihood function for 3d human pose tracking," IEEE Trans. Image Process., vol. 23, no. 12, pp. 5374-5389, Dec. 2014.
29. E. De Aguiar, C. Theobalt, C. Stoll, and H.-P. Seidel, "Marker-less deformable mesh tracking for human shape and motion capture," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
30. C. Wu, B. Wilburn, Y. Matsushita, and C. Theobalt, "High-quality shape from multi-view stereo and shading under general illumination," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 969-976.
31. C.-H. Huang, E. Boyer, N. Navab, and S. Ilic, "Human shape and pose tracking using keyframes," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 3446-3453.
32. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1297-1304.
33. J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon, "The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 103-110.
34. G. Pons-Moll, J. Taylor, J. Shotton, A. Hertzmann, and A. Fitzgibbon, "Metric regression forests for human pose estimation," in Proc. Brit. Mach. Vis. Conf., 2013, pp. 4.1-4.11.
35. V. Ganapath, C. Plagemann, S. Thrun, and D. Koller, "Real time motion capture using a time-of-flight camera," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 755-762.
36. V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun, Real-Time Human Pose Tracking from Range Data. New York, NY, USA: Springer, 2012.
37. A. Baak, M. Müller, G. Bharaj, H.-P. Seidel, and C. Theobalt, "A data-driven approach for real-time full body pose reconstruction from a depth camera," in Consumer Depth Cameras for Computer Vision. New York, NY, USA: Springer, 2013, pp. 71-98.
38. T. Helten, M. Müller, H.-P. Seidel, and C. Theobalt, "Real-time body tracking with one depth camera and inertial sensors," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1105-1112.
39. F. Bogo, M. J. Black, M. Loper, and J. Romero, "Detailed fullbody reconstructions of moving people from monocular RGB-D sequences," in Proc. Int. Conf. Comput. Vis., Dec. 2015, pp. 2300-2308.
40. L. Chen, H. Wei, and J. Ferryman, "A survey of human motion analysis using depth imagery," Pattern Recog. Lett., vol. 34, no. 15, pp. 1995-2006, Nov. 2013.
41. H. Dejnabadi, B. Jolles, E. Casanova, P. Fua, and K. Aminian, "Estimation and visualization of sagittal kinematics of lower limbsorientation using body-fixed sensors," IEEE Trans. Biomed. Eng., vol. 53, no. 7, pp. 1382-1393, Jul. 2006.
42. Y. Tao, H. Hu, and H. Zhou, "Integration of vision and inertial sensors for 3D arm motion tracking in home-based rehabilitation," Int. J. Robot. Res., vol. 26, no. 6, p. 607, 2007.
43. R. Slyper and J. Hodgins, "Action capture with accelerometers," in Proc. ACM SIGGRAPH/Eurographics, 2008, pp. 193-199.
44. J. Ziegler, H. Kretzschmar, C. Stachniss, G. Grisetti, and W. Burgard, "Accurate human motion capture in large areas by combining IMU- and laser-based people tracking," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2011, pp. 86-91.
45. G. Pons-Moll, A. Baak, J. Gall, L. Leal-Taixé, M. Müller, H.-P. Seidel, and B. Rosenhahn, "Outdoor human motion capture using inverse kinematics and von mises-fisher sampling," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 1243-1250.
46. R. Murray, Z. Li, and S. Sastry, A Mathematical Introduction to Robotic Manipulation. Baton Rouge, LA, USA: CRC Press, 1994.
47. G. Pons-Moll and B. Rosenhahn, "Model-based pose estimation," in Visual Analysis of Humans: Looking at People. New York, NY, USA: Springer, 2011, pp. 139-170.
48. G. Pons-Moll and B. Rosenhahn, "Ball joints for marker-less human motion capture," in Proc. Workshop Appl. Comput. Vis., 2009, pp. 1-8.
49. B. Rosenhahn and T. Brox, "Scaled motion dynamics for markerless motion capture," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
50. C. Bregler, J. Malik, and K. Pullen, "Twist based acquisition and tracking of animal and human kinematics," Int. J. Comput. Vis., vol. 56, no. 3, pp. 179-194, 2004.
51. C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, "Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 7, pp. 1325-1339, Jul. 2014.
52. (2015). Multimodal human motion database MPI08 [Online]. Available: http://www.tnt.uni-hannover.de/project/MPI08- Database/
53. C. R. Wren, A. Azarbayejani, T. Darrell, and A. P. Pentland, "Pfinder: Real-time tracking of the human body," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 780-785, Jul. 1997.
54. (2015). Xsens Motion Technologies [Online]. Available: http:// www.xsens.com/
55. A. Baak, T. Helten, M. Müller, G. Pons-Moll, B. Rosenhahn, and H. Seidel, "Analyzing and evaluating markerless motion tracking using inertial sensors," in Proc. 3rd Int. Workshop Human Motion Conjunction with ECCV, 2010, vol. 6553, pp. 137-150.
56. M. Loper, N. Mahmood, and M. J. Black, "MoSh: Motion and shape capture from sparse markers," ACM Trans. Graph., vol. 33, no. 6, pp. 220:1-220:13, Nov. 2014.
57. M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, "SMPL: A skinned multi-person linear model," ACM Trans. Graph., vol. 34, no. 6, pp. 248:1-248:16, Oct. 2015.
58. L. Sigal, M. Isard, H. Haussecker, and M. J. Black, "Loose-limbed people: Estimating 3d human pose and motion using nonparametric belief propagation," Int. J. Comput. Vis., vol. 98, no. 1, pp. 15-48, 2012.