Real-time expression transfer for facial reenactment

ACM Transactions on Graphics

Volumn 34, Issue 6, 2015, Pages

  Thies, Justus ^a   Zollhöfer, Michael ^b   Nießner, Matthias ^c   Valgaerts, Levi ^b   Stamminger, Marc ^a   Theobalt, Christian ^b  

^a UNIVERSITY OF ERLANGEN NUREMBERG   (Germany)

^b MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

^c STANFORD UNIVERSITY   (United States)

Author keywords

Depth camera; Expression transfer; Faces; Real time

Indexed keywords

LIGHTING; VIDEO STREAMING;

DEPTH CAMERA; EXPRESSION TRANSFERS; FACES; FACIAL DEFORMATIONS; PARAMETRIC MODELING; REAL TIME; REAL WORLD ENVIRONMENTS; SOURCE EXPRESSIONS;

RENDERING (COMPUTER GRAPHICS);

EID: 84995921764     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2816795.2818056     Document Type: Conference Paper

References (54)

1. ADELSON, E. H., ANDERSON, C. H., BERGEN, J. R., BURT, P. J., AND OGDEN, J. M. 1984. Pyramid methods in image processing. RCA engineer 29, 6, 33-41.
2. ALEXANDER, O., ROGERS, M., LAMBETH, W., CHIANG, M., AND DEBEVEC, P. 2009. The Digital Emily Project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses, ACM, 12:1-12:15.
3. BEELER, T., HAHN, F., BRADLEY, D., BICKEL, B., BEARDSLEY, P., GOTSMAN, C., SUMNER, R. W., AND GROSS, M. 2011. High-quality passive facial performance capture using anchor frames. ACM TOG 30, 4, 75.
4. BICKEL, B., BOTSCH, M., ANGST, R., MATUSIK, W., OTADUY, M., PFISTER, H., AND GROSS, M. 2007. Multi-scale capture of facial geometry and motion. ACM TOG 26, 3, 33.
5. BLANZ, V., AND VETTER, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. SIGGRAPH, ACM Press/Addison-Wesley Publishing Co., 187-194.
6. BLANZ, V., BASSO, C., POGGIO, T., AND VETTER, T. 2003. Reanimating faces in images and video. In Computer graphics forum, Wiley Online Library, 641-650.
7. BLANZ, V., SCHERBAUM, K., VETTER, T., AND SEIDEL, H.-P. 2004. Exchanging faces in images. In Computer Graphics Forum, Wiley Online Library, 669-676.
8. BORSHUKOV, G., PIPONI, D., LARSEN, O., LEWIS, J. P., AND TEMPELAAR-LIETZ, C. 2003. Universal capture: image-based facial animation for "The Matrix Reloaded". In SIGGRAPH Sketches, ACM, 16:1-16:1.
9. BOUAZIZ, S., WANG, Y., AND PAULY, M. 2013. Online modeling for realtime facial animation. ACM TOG 32, 4, 40.
10. BRADLEY, D., HEIDRICH, W., POPA, T., AND SHEFFER, A. 2010. High resolution passive facial performance capture. ACM TOG 29, 4, 41.
11. BURT, P. J., AND ADELSON, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. Communications 31, 532-540.
12. CAO, C., WENG, Y., LIN, S., AND ZHOU, K. 2013. 3D shape regression for real-time facial animation. ACM TOG 32, 4, 41.
13. CAO, C., HOU, Q., AND ZHOU, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM TOG 33, 4, 43.
14. CAO, C., WENG, Y., ZHOU, S., TONG, Y., AND ZHOU, K. 2014. Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG 20, 3, 413-425.
15. CHAI, J.-X., XIAO, J., AND HODGINS, J. 2003. Vision-based control of 3D facial animation. In Proc. SCA, Eurographics Association, 193-206.
16. CHEN, Y., AND MEDIONI, G. G. 1992. Object modelling by registration of multiple range images. Image and Vision Computing 10, 3, 145-155.
17. CHEN, Y.-L., WU, H.-T., SHI, F., TONG, X., AND CHAI, J. 2013. Accurate and robust 3d facial capture using a single rgbd camera. Proc. ICCV, 3615-3622.
18. CHUANG, E., AND BREGLER, C. 2002. Performance-driven facial animation using blend shape interpolation. Tech. Rep. CS-TR-2002-02, Stanford University.
19. COOTES, T. F., EDWARDS, G. J., AND TAYLOR, C. J. 2001. Active appearance models. IEEE TPAMI 23, 6, 681-685.
20. DALE, K., SUNKAVALLI, K., JOHNSON, M. K., VLASIC, D., MATUSIK, W., AND PFISTER, H. 2011. Video face replacement. ACM TOG 30, 6, 130.
21. EISERT, P., AND GIROD, B. 1998. Analyzing facial expressions for virtual conferencing. CGAA 18, 5, 70-78.
22. FYFFE, G., JONES, A., ALEXANDER, O., ICHIKARI, R., AND DEBEVEC, P. 2014. Driving high-resolution facial scans with video performance capture. ACM TOG 34, 1, 8.
23. GARRIDO, P., VALGAERTS, L., WU, C., AND THEOBALT, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM TOG 32, 6, 158.
24. GARRIDO, P., VALGAERTS, L., REHMSEN, O., THORMAEHLEN, T., PEREZ, P., AND THEOBALT, C. 2014. Automatic face reenactment. In Proc. CVPR.
25. GARRIDO, P., VALGAERTS, L., SARMADI, H., STEINER, I., VARANASI, K., PEREZ, P., AND THEOBALT, C. 2015. Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. In Computer Graphics Forum, Wiley-Blackwell.
26. GUENTER, B., GRIMM, C., WOOD, D., MALVAR, H., AND PIGHIN, F. 1998. Making faces. In Proc. SIGGRAPH, ACM, 55-66.
27. HSIEH, P.-L., MA, C., YU, J., AND LI, H. 2015. Unconstrained realtime facial performance capture. In Computer Vision and Pattern Recognition (CVPR).
28. HUANG, H., CHAI, J., TONG, X., AND WU, H.-T. 2011. Lever-aging motion capture and 3D scanning for high-fidelity facial performance acquisition. ACM TOG 30, 4, 74.
29. KEMELMACHER-SHLIZERMAN, I., SANKAR, A., SHECHTMAN, E., AND SEITZ, S. M. 2010. Being John Malkovich. In Proc. ECCV, 341-353.
30. KEMELMACHER-SHLIZERMAN, I., SHECHTMAN, E., GARG, R., AND SEITZ, S. M. 2011. Exploring photobios. ACM TOG 30, 4, 61.
31. LEWIS, J., AND ANJYO, K.-I. 2010. Direct manipulation blendshapes. IEEE CGAA 30, 4, 42-50.
32. LI, K., XU, F., WANG, J., DAI, Q., AND LIU, Y. 2012. A data-driven approach for facial expression synthesis in video. In Proc. CVPR, 57-64.
33. LI, H., YU, J., YE, Y., AND BREGLER, C. 2013. Realtime facial animation with on-the-fly correctives. ACM TOG 32, 4, 42.
34. LIU, Z., SHAN, Y., AND ZHANG, Z. 2001. Expressive expression mapping with ratio images. In Proc. SIGGRAPH, ACM, 271-276.
35. MEYER, M., BARR, A., LEE, H., AND DESBRUN, M. 2002. Generalized barycentric coordinates on irregular polygons. Journal of Graphics Tools 7, 1, 13-22.
36. MÜLLER, C. 1966. Spherical harmonics. Springer.
37. PIGHIN, F., AND LEWIS, J. 2006. Performance-driven facial animation. In ACM SIGGRAPH Courses.
38. PIGHIN, F., HECKER, J., LISCHINSKI, D., SZELISKI, R., AND SALESIN, D. 1998. Synthesizing realistic facial expressions from photographs. In Proc. SIGGRAPH, ACM Press/Addison-Wesley Publishing Co., 75-84.
39. RAMAMOORTHI, R., AND HANRAHAN, P. 2001. A signalprocessing framework for inverse rendering. In Proc. SIGGRAPH, ACM, 117-128.
40. SARAGIH, J. M., LUCEY, S., AND COHN, J. F. 2011. Deformable model fitting by regularized landmark mean-shift. IJCV 91, 2, 200-215.
41. SARAGIH, J. M., LUCEY, S., AND COHN, J. F. 2011. Real-time avatar animation from a single image. In Automatic Face and Gesture Recognition Workshops, 213-220.
42. SHI, F., WU, H.-T., TONG, X., AND CHAI, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM TOG 33, 6, 222.
43. SUMNER, R. W., AND POPOVIĆ, J. 2004. Deformation transfer for triangle meshes. ACM TOG 23, 3, 399-405.
44. SUWAJANAKORN, S., KEMELMACHER-SHLIZERMAN, I., AND SEITZ, S. M. 2014. Total moving face reconstruction. In Proc. ECCV, 796-812.
45. VALGAERTS, L., WU, C., BRUHN, A., SEIDEL, H.-P., AND THEOBALT, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187.
46. VLASIC, D., BRAND, M., PFISTER, H., AND POPOVIĆ, J. 2005. Face transfer with multilinear models. ACM TOG 24, 3, 426-433.
47. WANG, Y., HUANG, X., SU LEE, C., ZHANG, S., LI, Z., SAMARAS, D., METAXAS, D., ELGAMMAL, A., AND HUANG, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. CGF 23, 677-686.
48. WEISE, T., LI, H., GOOL, L. J. V., AND PAULY, M. 2009. Face/Off: live facial puppetry. In Proc. SCA, 7-16.
49. WEISE, T., BOUAZIZ, S., LI, H., AND PAULY, M. 2011. Realtime performance-based facial animation. 77.
50. WILLIAMS, L. 1990. Performance-driven facial animation. In Proc. SIGGRAPH, 235-242.
51. WILSON, C. A., GHOSH, A., PEERS, P., CHIANG, J.-Y., BUSCH, J., AND DEBEVEC, P. 2010. Temporal upsampling of performance geometry using photometric alignment. ACM TOG 29, 2, 17.
52. XIAO, J., BAKER, S., MATTHEWS, I., AND KANADE, T. 2004. Real-time combined 2D+3D active appearance models. In Proc. CVPR, 535-542.
53. ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM TOG 23, 3, 548-558.
54. ZOLLHÖFER, M., NIESSNER, M., IZADI, S., REHMANN, C., ZACH, C., FISHER, M., WU, C., FITZGIBBON, A., LOOP, C., THEOBALT, C., AND STAMMINGER, M. 2014. Real-time Non-rigid Reconstruction using an RGB-D Camera. ACM TOG 33, 4, 156.