Real-time non-rigid reconstruction using an RGB-D camera

ACM Transactions on Graphics

Volumn 33, Issue 4, 2014, Pages

  Zollhöfer, Michael ^a   Nießner, Matthias ^b   Izadi, Shahram ^c   Rehmann, Christoph ^c   Zach, Christopher ^c   Fisher, Matthew ^b   Wu, Chenglei ^d   Fitzgibbon, Andrew ^c   Loop, Charles ^c   Theobalt, Christian ^d   Stamminger, Marc ^a  

^a UNIVERSITY OF ERLANGEN NUREMBERG   (Germany)

^b STANFORD UNIVERSITY   (United States)

^c MICROSOFT RESEARCH   (United States)

^d MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

Author keywords

3D scanning; Deformation; Depth camera; Non rigid; Shape; Stereo matching; Surface reconstruction

Indexed keywords

CAMERAS; DEFORMATION; HARDWARE; RIGIDITY; STEREO IMAGE PROCESSING; SURFACE RECONSTRUCTION; IMAGE RECONSTRUCTION; INTERACTIVE COMPUTER GRAPHICS; THREE DIMENSIONAL COMPUTER GRAPHICS; TOYS;

3D-SCANNING; DEPTH CAMERA; NON-RIGID; SHAPE; STEREO MATCHING; HARDWARE AND SOFTWARE; HARDWARE SOLUTIONS; NON-RIGID RECONSTRUCTION; REAL- TIME; STEREO-MATCHING; SURFACES RECONSTRUCTION;

INTERACTIVE COMPUTER GRAPHICS; SURFACE RECONSTRUCTION;

EID: 84905746956     PISSN: 07342071     EISSN: 15577333     Source Type: Journal    
DOI: 10.1145/2601097.2601165     Document Type: Conference Paper

References (52)

1. BEELER, T., HAHN, F., BRADLEY, D., BICKEL, B., BEARDSLEY, P., GOTSMAN, C., SUMNER, R. W., AND GROSS, M. 2011. High-quality passive facial performance capture using anchor frames. ACM TOG (Proc. SIGGRAPH) 30, 4, 75.
2. BLANZ, V., AND VETTER, T. 1999. A morphable model for the synthesis of 3D faces. In Proc. SIGGRAPH, 187-194.
3. BLEYER, M., RHEMANN, C., AND ROTHER, C. 2011. Patchmatch stereo: Stereo matching with slanted support windows. In Proc. BMVC, vol. 11, 1-11.
4. BOJSEN-HANSEN, M., LI, H., AND WOJTAN, C. 2012. Tracking surfaces with evolving topology. ACM Trans. Graph. 31, 4, 53.
5. BOTSCH, M., AND SORKINE, O. 2008. On linear variational surface deformation methods. IEEE Trans. Vis. Comp. Graph 14, 1, 213-230.
6. BRADLEY, D., POPA, T., SHEFFER, A., HEIDRICH, W., AND BOUBEKEUR, T. 2008. Markerless garment capture. ACM TOG (Proc. SIGGRAPH) 27, 3, 99.
7. BROWN, B. J., AND RUSINKIEWICZ, S. 2007. Global non-rigid alignment of 3D scans. ACM TOG 26, 3, 21-30.
8. CAGNIART, C., BOYER, E., AND ILIC, S. 2010. Free-form mesh tracking: a patch-based approach. In Proc. CVPR.
9. CAO, C., WENG, Y., LIN, S., AND ZHOU, K. 2013. 3D shape regression for real-time facial animation. ACM TOG 32, 4, 41.
10. CHEN, J., IZADI, S., AND FITZGIBBON, A. 2012. Kinêtre: animating the world with the human body. In Proc. UIST, 435-444.
11. DE AGUIAR, E., STOLL, C., THEOBALT, C., AHMED, N., SEIDEL, H.-P., AND THRUN, S. 2008. Performance capture from sparse multi-view video. ACM TOG (Proc. SIGGRAPH) 27, 1-10.
12. DOU, M., FUCHS, H., AND FRAHM, J.-M. 2013. Scanning and tracking dynamic objects with commodity depth cameras. In Proc. ISMAR, 99-106.
13. GALL, J., STOLL, C., DE AGUIAR, E., THEOBALT, C., ROSENHAHN, B., AND SEIDEL, H.-P. 2009. Motion capture using joint skeleton tracking and surface estimation. In Proc. CVPR, 1746-1753.
14. GARRIDO, P., VALGAERT, L., WU, C., AND THEOBALT, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM TOG (Proc. SIGGRAPH Asia) 32, 6, 158.
15. HELTEN, T., BAAK, A., BHARAJ, G., MULLER, M., SEIDEL, H.-P., AND THEOBALT, C. 2013. Personalization and evaluation of a real-time depth-based full body tracker. In Proc. 3DV, 279-286.
16. HERNÁNDEZ, C., VOGIATZIS, G., BROSTOW, G. J., STENGER, B., AND CIPOLLA, R. 2007. Non-rigid photometric stereo with colored lights. In Proc. ICCV, 1-8.
17. IZADI, S., KIM, D., HILLIGES, O., MOLYNEAUX, D., NEW-COMBE, R., KOHLI, P., SHOTTON, J., HODGES, S., FREEMAN, D., DAVISON, A., AND FITZGIBBON, A. 2011. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proc. UIST, 559-568.
18. KOLB, A., BARTH, E., KOCH, R., AND LARSEN, R. 2009. Time-of-flight sensors in computer graphics. In Proc. Eurographics State-of-the-art Reports, 119-134.
19. LI, H., SUMNER, R. W., AND PAULY, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. In Proc. SGP, Eurographics Association, 1421-1430.
20. LI, H., ADAMS, B., GUIBAS, L. J., AND PAULY, M. 2009. Robust single-view geometry and motion reconstruction. ACM TOG 28, 5, 175.
21. LI, H., VOUGA, E., GUDYM, A., LUO, L., BARRON, J. T., AND GUSEV, G. 2013. 3D self-portraits. ACM TOG 32, 6, 187.
22. LI, H., YU, J., YE, Y., AND BREGLER, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Transactions on Graphics 32, 4 (July).
23. LIAO, M., ZHANG, Q., WANG, H., YANG, R., AND GONG, M. 2009. Modeling deformable objects from a single depth camera. In Proc. ICCV, 167-174.
24. MITRA, N. J., FLÖRY, S., OVSJANIKOV, M., GELFAND, N., GUIBAS, L. J., AND POTTMANN, H. 2007. Dynamic geometry registration. In Proc. SGP, 173-182.
25. NEWCOMBE, R. A., IZADI, S., HILLIGES, O., MOLYNEAUX, D., KIM, D., DAVISON, A. J., KOHLI, P., SHOTTON, J., HODGES, S., AND FITZGIBBON, A. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proc. ISMAR, 127-136.
26. NIESSNER, M., ZOLLHÖFER, M., IZADI, S., AND STAMMINGER, M. 2013. Real-time 3D reconstruction at scale using voxel hashing. ACM TOG 32, 6, 169.
27. OIKONOMIDIS, I., KYRIAZIS, N., AND ARGYROS, A. A. 2011. Efficient model-based 3D tracking of hand articulations using Kinect. In Proc. BMVC, 1-11.
28. PRADEEP, V., RHEMANN, C., IZADI, S., ZACH, C., BLEYER, M., AND BATHICHE, S. 2013. MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera. In Proc. ISMAR, 83-88.
29. SORKINE, O., AND ALEXA, M. 2007. As-rigid-as-possible surface modeling. In Proc. SGP, 109-116.
30. STARCK, J., AND HILTON, A. 2007. Surface capture for performance-based animation. Computer Graphics and Applications 27, 3, 21-31.
31. SUMNER, R. W., AND POPOVIĆ, J. 2004. Deformation transfer for triangle meshes. In ACM SIGGRAPH 2004 Papers, ACM, New York, NY, USA, SIGGRAPH '04, 399-405.
32. SUMNER, R. W., SCHMID, J., AND PAULY, M. 2007. Embedded deformation for shape manipulation. ACM TOG 26, 3, 80.
33. TAYLOR, J., SHOTTON, J., SHARP, T., AND FITZGIBBON, A. 2012. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In Proc. CVPR, 103-110.
34. TEVS, A., BERNER, A., WAND, M., IHRKE, I., BOKELOH, M., KERBER, J., AND SEIDEL, H.-P. 2012. Animation cartography-intrinsic reconstruction of shape and motion. ACM TOG 31, 2, 12.
35. THEOBALT, C., DE AGUIAR, E., STOLL, C., SEIDEL, H.-P., AND THRUN, S. 2010. Performance capture from multi-view video. In Image and Geometry Processing for 3D-Cinematography, R. Ron-fard and G. Taubin, Eds. Springer, 127ff.
36. TONG, J., ZHOU, J., LIU, L., PAN, Z., AND YAN, H. 2012. Scanning 3D full human bodies using Kinects. TVCG 18, 4, 643-650.
37. VALGAERTS, L., WU, C., BRUHN, A., SEIDEL, H.-P., AND THEOBALT, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM TOG (Proc. SIGGRAPH Asia) 31, 6 (November), 187.
38. VLASIC, D., BARAN, I., MATUSIK, W., AND POPOVIĆ, J. 2008. Articulated mesh animation from multi-view silhouettes. ACM TOG (Proc. SIGGRAPH).
39. VLASIC, D., PEERS, P., BARAN, I., DEBEVEC, P., POPOVIC, J., RUSINKIEWICZ, S., AND MATUSIK, W. 2009. Dynamic shape capture using multi-view photometric stereo. ACM TOG (Proc. SIGGRAPH Asia) 28, 5, 174.
40. WAND, M., ADAMS, B., OVSJANIKOV, M., BERNER, A., BOKELOH, M., JENKE, P., GUIBAS, L., SEIDEL, H.-P., AND SCHILLING, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM TOG 28, 15.
41. WASCHBÜSCH, M., WÜRMLIN, S., COTTING, D., SADLO, F., AND GROSS, M. 2005. Scalable 3D video of dynamic scenes. In Proc. Pacific Graphics, 629-638.
42. WEBER, D., BENDER, J., SCHNOES, M., STORK, A., AND FELLNER, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. Computer Graphics Forum 32, 1, 16-26.
43. WEI, X., ZHANG, P., AND CHAI, J. 2012. Accurate realtime full-body motion capture using a single depth camera. ACM TOG 31, 6 (Nov.), 188.
44. WEISE, T., WISMER, T., LEIBE, B., AND GOOL, L. V. 2009. In-hand scanning with online loop closure. In IEEE International Workshop on 3-D Digital Imaging and Modeling.
45. WEISE, T., LI, H., GOOL, L. V., AND PAULY, M. 2009. Face/off: Live facial puppetry. In Proceedings of the 2009 ACM SIG-GRAPH/Eurographics Symposium on Computer animation (Proc. SCA'09), Eurographics Association, ETH Zurich.
46. WEISE, T., BOUAZIZ, S., LI, H., AND PAULY, M. 2011. Realtime performance-based facial animation. ACM TOG 30, 4, 77.
47. WEISS, A., HIRSHBERG, D., AND BLACK, M. J. 2011. Home 3D body scans from noisy image and range data. In Proc. ICCV, 1951-1958.
48. WHITE, B. S., MCKEE, S. A., DE SUPINSKI, B. R., MILLER, B., QUINLAN, D., AND SCHULZ, M. 2005. Improving the computational intensity of unstructured mesh applications. In Proc. ACM Intl. Conf. on Supercomputing, 341-350.
49. WILAMOWSKI, B. M., AND YU, H. 2010. Improved computation for levenberg-marquardt training. IEEE Trans. Neural Networks 21, 6, 930-937.
50. WU, C., STOLL, C., VALGAERTS, L., AND THEOBALT, C. 2013. On-set performance capture of multiple actors with a stereo camera. ACM TOG 32, 6, 161.
51. YE, G., LIU, Y., HASLER, N., JI, X., DAI, Q., AND THEOBALT, C. 2012. Performance capture of interacting characters with handheld kinects. In Proc. ECCV. Springer, 828-841.
52. ZENG, M., ZHENG, J., CHENG, X., AND LIU, X. 2013. Template-less quasi-rigid shape modeling with implicit loop-closure. In Proc. CVPR, 145-152.