Free-viewpoint video of human actors using multiple handheld kinects

IEEE Transactions on Cybernetics

Volumn 43, Issue 5, 2013, Pages 1370-1382

  Ye, Genzhi ^a   Liu, Yebin ^a   Deng, Yue ^a   Hasler, Nils ^b   Ji, Xiangyang ^a   Dai, Qionghai ^a   Theobalt, Christian ^b  

^a TSINGHUA UNIVERSITY   (China)

^b MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

Author keywords

Free viewpoint video; Handheld kinect; Markerless performance capture

Indexed keywords

DYNAMIC BACKGROUND; ENERGY MINIMIZATION PROBLEM; FREE-VIEWPOINT VIDEO; HANDHELDS; JOINTLY OPTIMIZATIONS; PERFORMANCE CAPTURE; SCENE SEGMENTATION; SURFACE GEOMETRIES;

ALGORITHMS; IMAGE SEGMENTATION; TEXTURES;

CAMERAS;

ACTIMETRY; ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER; COMPUTER SIMULATION; EQUIPMENT; HUMAN; IMAGE ENHANCEMENT; METHODOLOGY; RECREATION; THREE DIMENSIONAL IMAGING; TRANSDUCER; VIDEORECORDING; WHOLE BODY IMAGING; DEVICES; PROCEDURES;

ACTIGRAPHY; ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER PERIPHERALS; COMPUTER SIMULATION; HUMANS; IMAGE ENHANCEMENT; IMAGING, THREE-DIMENSIONAL; PATTERN RECOGNITION, AUTOMATED; TRANSDUCERS; VIDEO GAMES; VIDEO RECORDING; WHOLE BODY IMAGING;

EID: 84890357691     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2013.2272321     Document Type: Article

References (59)

1. C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. A. J. Winder, and R. Szeliski, "High-quality video view interpolation using a layered representation," ACM Trans. Graph., vol. 23, no. 3, pp. 600-608, 2004.
2. J. Starck and A. Hilton, "Surface capture for performance based animation," IEEE Comput. Graphics Applicat., vol. 27, no. 3, pp. 21-31, May-Jun. 2007.
3. Y. Liu, Q. Dai, and W. Xu, "A point-cloud-based multiview stereo algorithm for free-viewpoint video," IEEE Trans. Vis. Comput. Graph., vol. 16, no. 3, pp. 407-418, May-Jun. 2010.
4. E. B. Jean-Sebastien Franco, "Exact polyhedral visual hulls," in Proc. BMVC, Sep. 2003.
5. W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, "Image-based visual hulls," in Proc. SIGGRAPH, 2000, pp. 369-374.
6. E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H. Seidel, and S. Thrun, "Performance capture from sparse multiview video," ACM Trans. Graphics, vol. 27, no. 3, p. 98, 2008.
7. C. Cagniart, E. Boyer, and S. Ilic, "Free-form mesh tracking: A patch-based approach," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2010, pp. 1339-1346.
8. D. Vlasic, I. Baran, W. Matusik, and J. Popović, "Articulated mesh animation from multiview silhouettes," ACM Trans. Graph., vol. 27, no. 3, pp. 1-9, 2008.
9. J. Gall, C. Stoll, E. De Aguiar, C. Theobalt, B. Rosenhahn, and H. Seidel, "Motion capture using joint skeleton tracking and surface estimation," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2009, pp. 1746-1753.
10. C. Wu, K. Varanasi, Y. Liu, H.-P. Seidel, and C. Theobalt, "Shading-based dynamic shape refinement from multiview video under general illumination," in Proc. IEEE Int. Conf. Comput. Vision, Nov. 2011, pp. 1108-1115.
11. S. Corazza, L. Mündermann, A. Chaudhari, T. Demattio, C. Cobelli, and T. Andriacchi, "A markerless motion capture system to study musculoskeletal biomechanics: Visual hull and simulated annealing approach," Ann. Biomed. Eng., vol. 34, no. 6, pp. 1019-1029, 2006.
12. C. Stoll, J. Gall, E. de Aguiar, S. Thrun, and C. Theobalt, "Video-based reconstruction of animatable human characters," ACM Trans. Graph., vol. 29, no. 6, p. 139, 2010.
13. F. Xu, Y. Liu, C. Stoll, J. Tompkin, G. Bharaj, Q. Dai, H.-P. Seidel, J. Kautz, and C. Theobalt, "Video-based characters: Creating new human performances from a multiview video database," ACM Trans. Graph., vol. 30, no. 4, p. 32, 2011.
14. G. Li, C. Wu, C. Stoll, Y. Liu, K. Varanasi, Q. Dai, and C. Theobalt, "Capturing relightable human performances under general uncontrolled illumination," Eurographics, Comput. Graph. Forum, vol. 32, no. 2, pp. 275-284, 2013.
15. N. Hasler, B. Rosenhahn, T. Thormählen, M. Wand, J. Gall, and H.-P. Seidel, "Markerless motion capture with unsynchronized moving cameras," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2009, pp. 224-231.
16. M. Ye, X. Wang, R. Yang, L. Ren, and M. Pollefeys, "Accurate 3-D pose estimation from a single depth image," in Proc. IEEE Int. Conf. Comput. Vision, 2011, pp. 731-738.
17. A. Baak, M. Müller, G. Bharaj, H.-P. Seidel, and C. Theobalt, "A data-driven approach for real-time full body pose reconstruction from a depth camera," in Proc. IEEE Int. Conf. Comput. Vision, Nov. 2011, pp. 1092-1099.
18. Y. Liu, C. Stoll, J. Gall, H.-P. Seidel, and C. Theobalt, "Markerless motion capture of interacting characters using multiview image segmentation," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2011, pp. 1249-1256.
19. Y. Liu, J. Gall, C. Stoll, Q. Dai, H.-P. Seidel, and C. Theobalt, "Markerless motion capture of multiple characters using multiview image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. PP, no. 99, p. 1, Feb. 2013.
20. G. Ye, Y. Liu, N. Hasler, X. Ji, Q. Dai, and C. Theobalt, "Performance capture of interacting characters with handheld Kinects," in Proc. ECCV, 2012.
21. C. Theobalt, N. Ahmed, H. P. A. Lensch, M. A. Magnor, and H.-P. Seidel, "Seeing people in different light-joint shape, motion, and reflectance capture," IEEE Trans. Vis. Comput. Graph., vol. 13, no. 4, pp. 663-674, Jul.-Aug. 2007.
22. P. Sand, L. McMillan, and J. Popovic, "Continuous capture of skin deformation," ACM Trans. Graph., vol. 22, no. 3, pp. 578-586, 2003.
23. D. Bradley, T. Popa, A. Sheffer, W. Heidrich, and T. Boubekeur, "Markerless garment capture," ACM Trans. Graph., vol. 27, no. 3, 2008.
24. B. Goldlücke and M. A. Magnor, "Space-time isosurface evolution for temporally coherent 3-D reconstruction," in Proc. IEEE Comput. Vision Pattern Recogn., Jun.-Jul. 2004, pp. 350-355.
25. C. Cagniart, E. Boyer, and S. Ilic, "Probabilistic deformable surface tracking from multiple videos," in Proc. ECCV, 2010, pp. 326-339.
26. L. Ballan, G. J. Brostow, J. Puwein, and M. Pollefeys, "Unstructured video-based rendering: Interactive exploration of casually captured videos," ACM Trans. Graph., vol. 29, no. 4, 2010.
27. J. Deutscher, A. Blake, and I. Reid, "Articulated body motion capture by annealed particle filtering," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2000, pp. 1144-1149.
28. C. Bregler, J. Malik, and K. Pullen, "Twist based acquisition and tracking of animal and human kinematics," Int. J. Comput. Vision, vol. 56, no. 3, pp. 179-194, 2004.
29. L. Sigal and M. Black, "Humaneva: Synchronized video and motion capture dataset for evaluation of articulated human motion," Brown University, Providence, RI, USA, Tech. Rep. CS-06-08, 2006.
30. A. Balan, L. Sigal, M. Black, J. Davis, and H. Haussecker, "Detailed human shape and pose from images," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2007, pp. 1-8.
31. R. Poppe, "Vision-based human motion analysis: An overview," Comput. Vision Image Understand., vol. 108, nos. 1-2, pp. 4-18, Oct.-Nov. 2007.
32. L. Ballan and G. Cortelazzo, "Markerless motion capture of skinned models in a four camera set-up using optical flow and silhouettes," in Proc. 3-DPVT, 2008.
33. D. Wu, Y. Liu, I. Ihrke, Q. Dai, and C. Theobalt, "Performance capture of high-speed motion using staggered multiview recording," Comput. Graph. Forum, vol. 31, no. 7-1, pp. 2019-2028, 2012.
34. C. Stoll, N. Hasler, J. Gall, H.-P. Seidel, and C. Theobalt, "Fast articulated motion tracking using a sums of Gaussians body model," in Proc. IEEE Int. Conf. Comput. Vision, Nov. 2011, pp. 951-958.
35. A. Elhayek, C. Stoll, N. Hasler, K. I. Kim, H.-P. Seidel, and C. Theobalt, "Spatio-temporal motion tracking with unsynchronized cameras," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2012, pp. 1870-1877.
36. A. Kolb, E. Barth, R. Koch, and R. Larsen, "Time-of-flight cameras in computer graphics," Comput. Graph. Forum, vol. 29, no. 1, pp. 141-159, 2010.
37. J. Han, L. Shao, D. Xu, and J. Shotton, "Enhanced computer vision with Microsoft Kinect sensor: A review," IEEE Trans. Cybern., vol. 43, no. 5, pp. 1318-1334, Oct. 2013.
38. S. Knoop, S. Vacek, and R. Dillmann, "Fusion of 2-D and 3-D sensor data for articulated body tracking," Robotics Autonomous Syst., vol. 57, no. 3, pp. 321-329, 2009.
39. Y. Zhu, B. Dariush, and K. Fujimura, "Kinematic self retargeting: A framework for human pose estimation," Comput. Vision Image Understand., vol. 114, no. 12, pp. 1362-1375, 2010.
40. Y. Pekelny and C. Gotsman, "Articulated object reconstruction and markerless motion capture from depth video," CGF, vol. 27, no. 2, pp. 399-408, 2008.
41. V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun, "Real time motion capture using a single time-of-flight camera," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2010, pp. 755-762.
42. J. Shotton, A. W. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from single depth images," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2011, pp. 1297-1304.
43. R. Girshick, A. Shotton, P. Kohli, A. Criminisi, and A. Fitzgibbon, "Efficient regression of general-activity human poses from depth images," in Proc. IEEE Int. Conf. Comput. Vision, Nov. 2011, pp. 415-422.
44. M. Ye, X. Wang, R. Yang, L. Ren, and M. Pollefeys, "Accurate 3-D pose estimation from a single depth image," in Proc. IEEE Int. Conf. Comput. Vision, 2011, pp. 731-738.
45. A. Weiss, D. Hirshberg, and M. J. Black, "Home 3-D body scans from noisy image and range data," in Proc. IEEE Int. Conf. Comput. Vision, Nov. 2011, pp. 1951-1958.
46. K. Cheung, T. Kanade, J.-Y. Bouguet, and M. Holler, "A real time system for robust 3D voxel reconstruction of human motions," in Proc. IEEE Comput. Vision Pattern Recogn., Jun. 2000, pp. 714-720.
47. R. Horaud, M. Niskanen, G. Dewaele, and E. Boyer, "Human motion tracking by registering an articulated surface to 3-D points and normals," TPAMI, vol. 31, no. 1, pp. 158-163, 2009.
48. S. Corazza, L. Mündermann, E. Gambaretto, G. Ferrigno, and T. P. Andriacchi, "Markerless motion capture through visual hull, articulated ICP and subject specific model generation," Int. J. Comput. Vision, vol. 87, pp. 156-169, Mar. 2010 [Online]. Available: http://dx.doi.org/10.1007/s11263- 009-0284-3
49. K. Berger, K. Ruhl, Y. Schroeder, C. Bruemmer, A. Scholz, and M. A. Magnor, "Markerless motion capture using multiple color-depth sensors," in Proc. VMV, 2011, pp. 317-324.
50. J.-Y. Bouguet, "Camera calibration toolbox for Matlab".
51. I. Baran and J. Popovic, "Automatic rigging and animation of 3-D characters," ACM Trans. Graph., vol. 26, no. 3, p. 72, 2007.
52. D. Aiger, N. J. Mitra, and D. Cohen-Or, "4-points congruent sets for robust surface registration," ACM Trans. Graph., vol. 27, no. 3, pp. 1-10, 2008.
53. M. Botsch and O. Sorkine, "On linear variational surface deformation methods," IEEE Trans. Vis. Comput. Graph., vol. 14, no. 1, pp. 213-230, Jan.-Feb. 2008.
54. P. E. Debevec, Y. Yu, and G. Borshukov, "Efficient view-dependent image-based rendering with projective texture-mapping," in Proc. Symp. Rendering Tech., 1998, pp. 105-116.
55. M. Eisemann, B. de Decker, M. A. Magnor, P. Bekaert, E. de Aguiar, N. Ahmed, C. Theobalt, and A. Sellent, "Floating textures," Comput. Graph. Forum, vol. 27, no. 2, pp. 409-418, 2008.
56. Y. Deng, Y. Liu, Q. Dai, Z. Zhang, and Y. Wang, "Noisy depth maps fusion for multiview stereo via matrix completion," IEEE J. Sel. Topics Signal Process., vol. 6, no. 5, pp. 566-582, Sep. 2012.
57. Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, "Low-rank structure learning via nonconvex heuristic recovery," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 3, pp. 383-396, Mar. 2013.
58. J. Carranza, C. Theobalt, M. A. Magnor, and H.-P. Seidel, "Free-viewpoint video of human actors," ACM Trans. Graph., vol. 22, no. 3, pp. 569-577, 2003.
59. S. Vedula, S. Baker, and T. Kanade, "Image-based spatio-temporal modeling and view interpolation of dynamic events," ACM Trans. Graph., vol. 24, no. 2, pp. 240-261, 2005.