Markerless shape and motion capture from multiview video sequences

IEEE Transactions on Circuits and Systems for Video Technology

Volumn 21, Issue 3, 2011, Pages 320-334

  Li, Kun ^a   Dai, Qionghai ^a   Xu, Wenli ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

3 D mesh; deformation; depth map; motion capture; shape recovery

Indexed keywords

3-D MESH; 3D OBJECT; CONVEX OBJECTS; DEPTH MAP; GEOMETRIC DETAILS; MARCHING CUBE; MARKERLESS; MOTION CAPTURE; MOTION-CAPTURE APPROACH; MULTIVIEW VIDEO SEQUENCES; NON-RIGID MOTION; POINT CLOUD; SCENE REPRESENTATION; SHAPE RECOVERY; VARIATIONAL MODELS; VOLUMETRIC DEFORMATION;

DEFORMATION; MERGING; RECOVERY; SHAPE OPTIMIZATION; VIDEO RECORDING;

THREE DIMENSIONAL;

EID: 79953060078     PISSN: 10518215     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSVT.2011.2106251     Document Type: Article

References (59)

1. J. Starck, A. Maki, S. Nobuhara, A. Hilton, and T. Matsuyama, "The multiple-camera 3-D production studio," IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 6, pp. 856-869, Jun. 2009.
2. M. Krinidis, N. Nikolaidis, and I. Pitas, "3-D head pose estimation in monocular video sequences using deformable surfaces and radial basis functions," IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 2, pp. 261-272, Feb. 2009.
3. T. Matsuyama, X. Wu, T. Takai, and T. Wada, "Real-time dynamic 3-D object shape reconstruction and high-fidelity texture mapping for 3-D video," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 3, pp. 357-369, Mar. 2004.
4. F. Huguet and F. Devernay, "A variational method for scene flow estimation from stereo sequences," in Proc. IEEE Int. Conf. Comput. Vision, Oct. 2007, pp. 1-7.
5. J.-P. Pons, R. Keriven, and O. Faugeras, "Multiview stereo reconstruction and scene flow estimation with a global image-based matching score," Int. J. Comput. Vision, vol. 72, no. 2, pp. 179-193, 2007. (Pubitemid 46094759)
6. G. Vogiatzis, C. Hernández, P. H. S. Torr, and R. Cipolla, "Mul-tiview stereo via volumetric graph-cuts and occlusion robust photo-consistency," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 12, pp. 2241-2246, Dec. 2007. (Pubitemid 350124071)
7. Y. Furukawa and J. Ponce, "Accurate, dense, and robust multiview stereopsis," in Proc. IEEE Conf. Comput. Vision Pattern Recog., Jul. 2007, pp. 1-8.
8. M. Goesele, B. Curless, and S. M. Seitz, "Multiview stereo revisited," in Proc. IEEE Conf. Comput. Vision Pattern Recog., vol. 2. Oct. 2006, pp. 2402-2409.
9. J. Starck and A. Hilton, "Surface capture for performance-based animation," IEEE Comput. Graph. Applicat., vol. 27, no. 3, pp. 21-31, May-Jun. 2007. (Pubitemid 46767722)
10. B. Rosenhahn, T. Brox, and H.-P. Seidel, "Scaled motion dynamics for markerless motion capture," in Proc. IEEE Conf. Comput. Vision Pattern Recog., Jun. 2007, pp. 1-8.
11. J. Gall, B. Rosenhahn, and H.-P. Seidel, "Drift-free tracking of rigid and articulated objects," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2008, pp. 1-8.
12. J. Gall, C. Stoll, E. De Aguiar, C. Theobalt, B. Rosenhahn, and H.-P. Seidel, "Motion capture using joint skeleton tracking and surface estimation," in Proc. IEEE Conf. Comput. Vision Pattern Recog., Jun. 2009, pp. 1746-1753.
13. R. Zhang, P. Tsai, J. E. Cryer, and M. Shah, "Shape from shading: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 8, pp. 690-706, Aug. 1999.
14. H. Jin, D. Cremers, D. Wang, A. Yezzi, E. Prados, and S. Soatto, "3-D reconstruction of shaded objects from multiple images under unknown illumination," Int. J. Comput. Vision, vol. 76, no. 3, pp. 245-256, Mar. 2008.
15. J. S. Franco and E. Boyer, "Exact polyhedral visual hulls," in Proc. Br. Mach. Vision Conf., Sep. 2003, pp. 329-338.
16. P. Gargallo and P. Sturm, "Bayesian 3-D modeling from images using multiple depth maps," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., vol. 2. Jun. 2005, pp. 885-891.
17. G. Vogiatzis, P. Torr, S. M. Seitz, and R. Cipolla, "Reconstructing relief surfaces," Image Vision Comput., vol. 26, no. 3, pp. 397-404, 2008. (Pubitemid 350192035)
18. G. Vogiatzis, P. Torr, and R. Cipolla, "Multiview stereo via volumetric graph-cuts," in Proc. IEEE Conf. Comput. Vision Pattern Recog., Jun. 2005, pp. 391-398.
19. O. Faugeras and R. Keriven, "Variational principles, surface evolution, PDEs, level set methods and the stereo problem," IEEE Trans. Image Process., vol. 7, no. 3, pp. 336-344, Mar. 1998. (Pubitemid 128745351)
20. C. Hernández and F. Schmitt, "Silhouette and stereo fusion for 3-D object modeling," Comput. Vision Image Understand., vol. 96, no. 3, pp. 367-392, 2004.
21. K. Kolev, M. Klodt, T. Brox, and D. Cremers, "Continuous global optimization in multiview 3-D reconstruction," Int. J. Comput. Vision, vol. 84, no. 1, pp. 80-96, Aug. 2009.
22. S. I. Park and J. K. Hodgins, "Capturing and animating skin deformation in human motion," ACM Trans. Graph., vol. 25, no. 3, pp. 881-889, 2006. (Pubitemid 44481050)
23. V. Scholz, T. Stich, M. Magnor, M. Keckeisen, and M. Wacker, "Garment motion capture using color-coded patterns," in Proc. Eurographics, vol. 24, no. 3. 2005, pp. 439-448.
24. R. White, K. Crane, and D. A. Forsyth, "Capturing and animating occluded cloth," in Proc. ACM SIGGRAPH, 2007, pp. 34-41.
25. P. Sand, L. McMillan, and J. Popović, "Continuous capture of skin deformation," in Proc. ACM SIGGRAPH, 2003, pp. 578-586.
26. B. Allen, B. Curless, and Z. Popović, "Articulated body deformation from range scan data," in Proc. ACM SIGGRAPH, 2002, pp. 612-619.
27. B. Rosenhahn, U. Kersting, K. Powell, and H.-P. Seidel, "Cloth X-ray: Mocap of people wearing textiles," in Proc. 28th Pattern Recognit. DAGM Symp., vol. 4174. 2006, pp. 495-504. (Pubitemid 44608895)
28. T. Brox, B. Rosenhahn, J. Gall, and D. Cremers, "Combined region and motion-based 3-D tracking of rigid and articulated objects," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 3, pp. 402-415, Mar. 2010.
29. D. Vlasic, I. Baran, W. Matusik, and J. Popović, "Articulated mesh animation from multiview silhouettes," ACM Trans. Graph., vol. 27, no. 3, pp. 1-9, 2008.
30. L. Sigal and M. J. Black, "HumanEva: Synchronized video and motion capture dataset for evaluation of articulated human motion," Dept. Comput. Sci., Brown Univ., Providence, RI, Tech. Rep. CS-06-08, 2006.
31. M. Botsch and O. Sorkine, "On linear variational surface deformation methods," IEEE Trans. Visual. Comput. Graph., vol. 14, no. 1, pp. 213-230, Jan.-Feb. 2008. (Pubitemid 350157892)
32. W. Xu, K. Zhou, Y. Yu, Q. Tan, Q. Peng, and B. Guo, "Gradient domain editing of deforming mesh sequences," in Proc. ACM SIGGRAPH, 2007, pp. 84-93.
33. R. W. Sumner and J. Popović, "Deformation transfer for triangle meshes," in Proc. ACM SIGGRAPH, 2004, pp. 399-405.
34. E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H. P. Seidel, and S. Thrun, "Performance capture from sparse multiview video," in Proc. ACM SIGGRAPH, 2008, pp. 1-10.
35. C. Stoll, E. De Aguiar, C. Theobalt, and H.-P. Seidel, "A volumetric approach to interactive shape editing," Max-Planck-Inst. Inform., Saarbrücken, Germany, Tech. Rep. MPI-I-2007-4-004, Jun. 2007.
36. O. Sorkine and M. Alexa, "As-rigid-as-possible surface modeling," in Proc. Eurograph. Symp. Geom. Process., 2007, pp. 109-116.
37. B. K. P. Horn and B. G. Schunck, "Determining optical flow," Artif. Intell., vol. 17, pp. 185-203, 1981.
38. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, "High accuracy optical flow estimation based on a theory for warping," in Proc. Eur. Conf. Comput. Vision, vol. 3024. 2004, pp. 25-36.
39. Z. Christopher, P. Thomas, and B. Horst, "A duality based approach for realtime TV-L1 optical flow," in Proc. DAGM Symp. Pattern Recognit., 2007, pp. 214-223.
40. A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, and C. Schnörr, "Variational optical flow computation in real-time," IEEE Trans. Image Process., vol. 14, no. 5, pp. 608-615, May 2005. (Pubitemid 40690617)
41. A. Blake and A. Zisserman, Visual Reconstruction. Cambridge, MA: MIT Press, 1987.
42. M. Lefébure and L. D. Cohen, "Image registration, optical flow and local rigidity," J. Math. Imaging Vision, vol. 14, no. 2, pp. 131-147, 2001. (Pubitemid 32546687)
43. W. E. Lorensen, "Marching cubes: A high resolution 3-D surface construction algorithm," in Proc. ACM SIGGRAPH, 1987, pp. 163-169.
44. H. Si and K. Gaertner, "Meshing piecewise linear complexes by constrained delaunay tetrahedralizations," in Proc. Int. Meshing Roundtable, 2005, pp. 147-163.
45. H. Si, "On refinement of constrained delaunay tetrahedralizations, " in Proc. Int. Meshing Roundtable, 2006, pp. 509-528.
46. O. Sorkine, "Laplacian mesh processing," in Proc. Eurograph. State-of-the-Art Rep., Aug. 2005, pp. 1-18.
47. E. J. Candès and B. Recht, "Exact matrix completion via convex optimization," Found. Comput. Math., vol. 9, no. 6, pp. 717-772, 2009.
48. E. J. Candès and Y. Plan, "Matrix completion with noise," Proc. IEEE, vol. 98, no. 6, pp. 925-936, Jun. 2010.
49. 1 norm," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2010, pp. 771-778.
50. S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade, "Three-dimensional scene flow," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp. 475-480, Mar. 2005. (Pubitemid 40377340)
51. M. Fazel, H. Hindi, and S. Boyd, "Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices," in Proc. Am. Control Conf., Jun. 2003, pp. 2156-2162.
52. E. J. Candès and Y. Plan, "A singular value thresholding algorithm for matrix completion," SIAM J. Optimization, vol. 20, no. 4, pp. 1956-1982, 2010.
53. Y. Furukawa. PMVS: Patch-Based Multiview Stereo Software [Online]. Available: http://www.cs.washington.edu/homes/furukawa/research/pmvs/index.html
54. S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, "A comparison and evaluation of multiview stereo reconstruction algorithms," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2006, pp. 519-528.
55. D. Bradley, T. Boubekeur, and W. Heidrich, "Accurate multiview reconstruction using robust binocular stereo and surface meshing," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2008, pp. 1-8.
56. J. Starck, A. Hilton, and G. Miller, "Volumetric stereo with silhouette and feature constraints," in Proc. Br. Mach. Vision Conf., vol. 3. Sep. 2006, pp. 1189-1198.
57. J. P. Pons, R. Keriven, and O. Faugeras, "Modeling dynamic scenes by registering multiview image sequences," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2005, pp. 822-827.
58. A. Zaharescu, E. Boyer, and R. Horaud, "Transformesh: A topology-adaptive mesh-based approach to surface evolution," in Proc. Asian Conf. Comput. Vision, Nov. 2007, pp. 166-175.
59. H. Si, "A quality tetrahedral mesh generator and 3-D delaunay trian-gulator," Weierstrauss Instit. Appl. Anal. Stochastics, Berlin, Germany, Tech. Rep. 9, 2004.