Fusion4D: Real-time performance capture of challenging scenes

ACM Transactions on Graphics

Volumn 35, Issue 4, 2016, Pages

  Dou, Mingsong ^a   Khamis, Sameh ^a   Degtyarev, Yury ^a   Davidson, Philip ^a   Fanello, Sean Ryan ^a   Kowdle, Adarsh ^a   Escolano, Sergio Orts ^a   Rhemann, Christoph ^a   Kim, David ^a   Taylor, Jonathan ^a   Kohli, Pushmeet ^a   Tankovich, Vladimir ^a   Izadi, Shahram ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

4D reconstruction; Multi view; Nonrigid; Real time

Indexed keywords

INTERACTIVE COMPUTER GRAPHICS;

4D RECONSTRUCTION; HIGH QUALITY RECONSTRUCTION; MULTI-VIEWS; NONRIGID; PERFORMANCE CAPTURE; REAL TIME; REAL TIME PERFORMANCE; REAL-TIME TECHNIQUES;

SURFACE RECONSTRUCTION;

EID: 84980010139     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2897824.2925969     Document Type: Conference Paper

References (50)

1. BEELER, T., HAHN, F., BRADLEY, D., BICKEL, B., BEARDSLEY, P., GOTSMAN, C., SUMNER, R. W., AND GROSS, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Transactions on Graphics (TOG) 30, 4, 75.
2. BLEYER, M., RHEMANN, C., AND ROTHER, C. 2011. Patchmatch stereo: Stereo matching with slanted support windows. In Proc. BMVC, vol. 11, 1-11.
3. BOGO, F., BLACK, M. J., LOPER, M., AND ROMERO, J. 2015. Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In ICCV, 2300-2308.
4. BOJSEN-HANSEN, M., LI, H., AND WOJTAN, C. 2012. Tracking surfaces with evolving topology. ACM Trans. Graph. 31, 4, 53.
5. BRADLEY, D., POPA, T., SHEFFER, A., HEIDRICH, W., AND BOUBEKEUR, T. 2008. Markerless garment capture. ACM TOG (Proc. SIGGRAPH) 27, 3, 99.
6. CAGNIART, C., BOYER, E., AND ILIC, S. 2010. Free-form mesh tracking: a patch-based approach. In Proc. CVPR.
7. CHEN, Y., AND MEDIONI, G. 1992. Object modelling by registration of multiple range images. CVIU 10, 3, 144-155.
8. CHEN, J., BAUTEMBACH, D., AND IZADI, S. 2013. Scalable real-time volumetric surface reconstruction. ACM TOG.
9. COLLET, A., CHUANG, M., SWEENEY, P., GILLETT, D., EVSEEV, D., CALABRESE, D., HOPPE, H., KIRK, A., AND SULLIVAN, S. 2015. High-quality streamable free-viewpoint video. ACM TOG 34, 4, 69.
10. CURLESS, B., AND LEVOY, M. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, 303-312.
11. DE AGUIAR, E., STOLL, C., THEOBALT, C., AHMED, N., SEIDEL, H.-P., AND THRUN, S. 2008. Performance capture from sparse multi-view video. ACM TOG (Proc. SIGGRAPH) 27, 1-10.
12. DOU, M., FUCHS, H., AND FRAHM, J.-M. 2013. Scanning and tracking dynamic objects with commodity depth cameras. In Proc. ISMAR, IEEE, 99-106.
13. DOU, M., TAYLOR, J., FUCHS, H., FITZGIBBON, A., AND IZADI, S. 2015. 3d scanning deformable objects with a single rgbd sensor. In CVPR.
14. ENGELS, C., STEWÉNIUS, H., AND NISTÉ R, D. 2006. Bundle adjustment rules. Photogrammetric computer vision 2, 124-131.
15. GALL, J., STOLL, C., DE AGUIAR, E., THEOBALT, C., ROSENHAHN, B., AND SEIDEL, H.-P. 2009. Motion capture using joint skeleton tracking and surface estimation. In Proc. CVPR, IEEE, 1746-1753.
16. GUO, K., XU, F., WANG, Y., LIU, Y., AND DAI, Q. 2015. Robust non-rigid motion tracking and surface reconstruction using l0 regularization. In ICCV, 3083-3091.
17. KRÄHENBÜH, P., AND KOLTUN, V. 2011. Efficient inference in fully connected crfs with gaussian edge potentials. NIPS.
18. KUTULAKOS, K. N., AND SEITZ, S. M. 2000. A theory of shape by space carving. IJCV.
19. LI, H., ADAMS, B., GUIBAS, L. J., AND PAULY, M. 2009. Robust single-view geometry and motion reconstruction. ACM TOG.
20. LOWE, D. G. 2004. Distinctive image features from scale-invariant keypoints. IJCV.
21. MITRA, N. J., FLÖRY, S., OVSJANIKOV, M., GELFAND, N., GUIBAS, L. J., AND POTTMANN, H. 2007. Dynamic geometry registration. In Proc. SGP, 173-182.
22. MORI, M., MACDORMAN, K. F., AND KAGEKI, N. 2012. The uncanny valley [from the field]. Robotics & Automation Magazine, IEEE 19, 2, 98-100.
23. NEWCOMBE, R. A., IZADI, S., HILLIGES, O., MOLYNEAUX, D., KIM, D., DAVISON, A. J., KOHLI, P., SHOTTON, J., HODGES, S., AND FITZGIBBON, A. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proc. ISMAR, 127-136.
24. NEWCOMBE, R. A., FOX, D., AND SEITZ, S. M. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 343-352.
25. PONS-MOLL, G., TAYLOR, J., SHOTTON, J., HERTZMANN, A., AND FITZGIBBON, A. 2015. Metric regression forests for correspondence estimation. IJCV 113, 3, 163-175.
26. PRADEEP, V., RHEMANN, C., IZADI, S., ZACH, C., BLEYER, M., AND BATHICHE, S. 2013. MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera. In Proc. ISMAR, IEEE, 83-88.
27. REVAUD, J., WEINZAEPFEL, P., HARCHAOUI, Z., AND SCHMID, C. 2015. Epicflow: Edge-preserving interpolation of correspondences for optical flow. CVPR.
28. ROSTEN, E., AND DRUMMOND, T. 2005. Fusing points and lines for high performance tracking. In ICCV.
29. RUSINKIEWICZ, S., AND LEVOY, M. 2001. Efficient variants of the icp algorithm. In 3DIM, 145-152.
30. SHOTTON, J., GLOCKER, B., ZACH, C., IZADI, S., CRIMINISI, A., AND FITZGIBBON, A. 2013. Scene coordinate regression forests for camera relocalization in rgb-d images. In CVPR.
31. SMOLIC, A. 2011. 3d video and free viewpoint videofrom capture to display. Pattern recognition 44, 9, 1958-1968.
32. STARCK, J., AND HILTON, A. 2007. Surface capture for performance-based animation. Computer Graphics and Applications 27, 3, 21-31.
33. STOLL, C., HASLER, N., GALL, J., SEIDEL, H., AND THEOBALT, C. 2011. Fast articulated motion tracking using a sums of gaussians body model. In Proc. ICCV, IEEE, 951-958.
34. SUMNER, R. W., SCHMID, J., AND PAULY, M. 2007. Embedded deformation for shape manipulation. ACM TOG 26, 3, 80.
35. TEVS, A., BERNER, A., WAND, M., IHRKE, I., BOKELOH, M., KERBER, J., AND SEIDEL, H.-P. 2012. Animation cartographyintrinsic reconstruction of shape and motion. ACM TOG.
36. THEOBALT, C., DE AGUIAR, E., STOLL, C., SEIDEL, H.-P., AND THRUN, S. 2010. Performance capture from multi-view video. In Image and Geometry Processing for 3D-Cinematography, R. Ronfard and G. Taubin, Eds. Springer, 127ff.
37. VINEET, V., WARRELL, J., AND TORR, P. H. S. 2012. Filter-based mean-field inference for random fields with higher-order terms and product label-spaces. In ECCV.
38. VLASIC, D., BARAN, I., MATUSIK, W., AND POPOVIĆ, J. 2008. Articulated mesh animation from multi-view silhouettes. ACM TOG (Proc. SIGGRAPH).
39. VLASIC, D., PEERS, P., BARAN, I., DEBEVEC, P., POPOVIC, J., RUSINKIEWICZ, S., AND MATUSIK, W. 2009. Dynamic shape capture using multi-view photometric stereo. ACM TOG (Proc. SIGGRAPH Asia) 28, 5, 174.
40. WAND, M., ADAMS, B., OVSJANIKOV, M., BERNER, A., BOKELOH, M., JENKE, P., GUIBAS, L., SEIDEL, H.-P., AND SCHILLING, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM TOG.
41. WANG, S., FANELLO, S. R., RHEMANN, C., IZADI, S., AND KOHLI, P. 2016. The global patch collider. CVPR.
42. WASCHBÜSCH, M., WÜRMLIN, S., COTTING, D., SADLO, F., AND GROSS, M. 2005. Scalable 3D video of dynamic scenes. In Proc. Pacific Graphics, 629-638.
43. WEI, L., HUANG, Q., CEYLAN, D., VOUGA, E., AND LI, H. 2015. Dense human body correspondences using convolutional networks. arXiv preprint arXiv:1511.05904.
44. WEINZAEPFEL, P., REVAUD, J., HARCHAOUI, Z., AND SCHMID, C. 2013. Deepflow: Large displacement optical flow with deep matching. In ICCV.
45. YE, M., AND YANG, R. 2014. Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera. In CVPR, IEEE.
46. YE, M., ZHANG, Q., WANG, L., ZHU, J., YANG, R., AND GALL, J. 2013. A survey on human motion analysis from depth data. In Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Springer, 149-187.
47. ZACH, C. 2014. Robust bundle adjustment revisited. In Computer Vision-ECCV 2014. Springer, 772-787.
48. ZENG, M., ZHENG, J., CHENG, X., AND LIU, X. 2013. Templateless quasi-rigid shape modeling with implicit loop-closure. In Proc. CVPR, IEEE, 145-152.
49. ZHANG, Q., FU, B., YE, M., AND YANG, R. 2014. Quality dynamic human body modeling using a single low-cost depth camera. In CVPR, IEEE, 676-683.
50. ZOLLHÖFER, M., NIESSNER, M., IZADI, S., RHEMANN, C., ZACH, C., FISHER, M., WU, C., FITZGIBBON, A., LOOP, C., THEOBALT, C., ET AL. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG.