Real-time facial animation with image-based dynamic avatars

ACM Transactions on Graphics

Volumn 35, Issue 4, 2016, Pages

  Cao, Chen ^a   Wu, Hongzhi ^a   Weng, Yanlin ^a   Shao, Tianjia ^a   Zhou, Kun ^a  

^a ZHEJIANG UNIVERSITY   (China)

Author keywords

Face tracking; Facial animation; Hair modeling; Imagebased rendering; Virtual avatar

Indexed keywords

ANIMATION; FACE RECOGNITION; INTERACTIVE COMPUTER GRAPHICS; OPTIMIZATION; TOPOLOGY;

FACE TRACKING; FACIAL ANIMATION; HAIR MODEL; IMAGE BASED RENDERING; VIRTUAL AVATAR;

THREE DIMENSIONAL COMPUTER GRAPHICS;

EID: 84980047577     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2897824.2925873     Document Type: Conference Paper

References (62)

1. ALEXANDER, O., ROGERS, M., LAMBETH, W., CHIANG, M., AND DEBEVEC, P. 2009. The digital Emily project: photo-real facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1-12:15.
2. ALEXANDER, O., FYFFE, G., BUSCH, J., YU, X., ICHIKARI, R., JONES, A., DEBEVEC, P., JIMENEZ, J., DANVOYE, E., ANTIONAZZI, B., EHELER, M., KYSELA, Z., AND VON DER PAHLEN, J. 2013. Digital Ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters.
3. AMBERG, B., BLAKE, A., FITZGIBBON, A., ROMDHANI, S., AND VETTER, T. 2007. Reconstructing high quality facesurfaces using model based stereo. In Proceedings of ICCV, 1-8.
4. BALTRUŠAITIS, T., ROBINSON, P., AND MORENCY, L.-P. 2012. 3D constrained local model for rigid and non-rigid facial tracking. In Proceedings of IEEE CVPR, 2610-2617.
5. BARNES, C., SHECHTMAN, E., FINKELSTEIN, A., AND GOLDMAN, D. B. 2009. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (July), 24:1-24:11.
6. BEELER, T., HAHN, F., BRADLEY, D., BICKEL, B., BEARDSLEY, P., GOTSMAN, C., SUMNER, R. W., AND GROSS, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4, 75:1-75:10.
7. BLANZ, V., AND VETTER, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of SIGGRAPH, 187-194.
8. BOUAZIZ, S., WANG, Y., AND PAULY, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graph. 32, 4 (July), 40:1-40:10.
9. BRADLEY, D., HEIDRICH, W., POPA, T., AND SHEFFER, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1-41:10.
10. CAO, C., WENG, Y., LIN, S., AND ZHOU, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graph. 32, 4 (July), 41:1-41:10.
11. CAO, C., HOU, Q., AND ZHOU, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graph. 33, 4 (July), 43:1-43:10.
12. CAO, C., WENG, Y., ZHOU, S., TONG, Y., AND ZHOU, K. 2014. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (Mar.), 413-425.
13. CAO, C., BRADLEY, D., ZHOU, K., AND BEELER, T. 2015. Real-time high-fidelity facial performance capture. ACM Trans. Graph. 34, 4 (July), 46:1-46:9.
14. CASAS, D., ALEXANDER, O., FENG, A. W., FYFFE, G., ICHIKARI, R., DEBEVEC, P., WANG, R., SUMA, E., AND SHAPIRO, A. 2015. Rapid photorealistic blendshapes from commodity rgb-d sensors. In Proceedings of the 19th Symposium on Interactive 3D Graphics and Games, i3D '15, 134-134.
15. CHAI, J.-X., XIAO, J., AND HODGINS, J. 2003. Vision-based control of 3d facial animation. In Symp. Comp. Anim., 193-206.
16. CHAI, M., WANG, L., WENG, Y., YU, Y., GUO, B., AND ZHOU, K. 2012. Single-view hair modeling for portrait manipulation. ACM Trans. Graph. 31, 4 (July), 116:1-116:8.
17. CHAI, M., WANG, L., WENG, Y., JIN, X., AND ZHOU, K. 2013. Dynamic hair manipulation in images and videos. ACM Trans. Graph. 32, 4 (July), 75:1-75:8.
18. CHAI, M., ZHENG, C., AND ZHOU, K. 2014. A reduced model for interactive hairs. ACM Trans. Graph. 33, 4 (July), 1-11.
19. CHAI, M., LUO, L., SUNKAVALLI, K., CARR, N., HADAP, S., AND ZHOU, K. 2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph. 34, 6, 204.
20. DECARLO, D., AND METAXAS, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99-127.
21. DESBRUN, M., MEYER, M., SCHRODER, P., AND BARR, A. H. 1999. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proceedings of ACM SIGGRAPH, 317-324.
22. ESSA, I., BASU, S., DARRELL, T., AND PENTLAND, A. 1996. Modeling, tracking and interactive animation of faces and heads: using input from video. In Computer Animation, 68-79.
23. GARRIDO, P., VALGAERTS, L., WU, C., AND THEOBALT, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6, 158.
24. GARRIDO, P., ZOLLHOFER, M., CASAS, D., VALGAERTS, L., VARANASI, K., PEREZ, P., AND THEOBALT, C. 2016. Reconstruction of personalized 3d face rigs from monocular video. ACM Trans. Graph. to appear.
25. HU, L., MA, C., LUO, L., AND LI, H. 2014. Robust hair capture using simulated examples. ACM Trans. Graph. 33, 4 (July), 126:1-126:10.
26. HU, L., MA, C., LUO, L., WEI, L.-Y., AND LI, H. 2014. Capturing braided hairstyles. ACM Trans. Graph. 33, 6 (Nov.), 225:1-225:9.
27. HU, L., MA, C., LUO, L., AND LI, H. 2015. Single-view hair modeling using a hairstyle database. ACM Trans. Graph. 34, 4 (July), 125:1-125:9.
28. HUANG, J., SHI, X., LIU, X., ZHOU, K., WEI, L.-Y., TENG, S.- H., BAO, H., GUO, B., AND SHUM, H.-Y. 2006. Subspace gradient domain mesh deformation. ACM Trans. Graph. 25, 3 (July), 1126-1134.
29. HUANG, H., CHAI, J., TONG, X., AND WU, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1-74:10.
30. ICHIM, A. E., BOUAZIZ, S., AND PAULY, M. 2015. Dynamic 3d avatar creation from hand-held video input. ACM Trans. Graph. 34, 4 (July), 45:1-45:14.
31. JANCOSEK, M., AND PAJDLA, T. 2011. Multi-view reconstruction preserving weakly-supported surfaces. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 3121-3128.
32. JIMENEZ, J., SCULLY, T., BARBOSA, N., DONNER, C., ALVAREZ, X., VIEIRA, T., MATTS, P., ORVALHO, V., GUTIERREZ, D., AND WEYRICH, T. 2010. A practical appearance model for dynamic facial color. ACM Trans. Graph. 29, 6 (Dec.), 141:1-141:10.
33. JIMENEZ, J., ECHEVARRIA, J. I., OAT, C., AND GUTIERREZ, D. 2011. GPU Pro 2. AK Peters Ltd., ch. Practical and Realistic Facial Wrinkles Animation.
34. KAZHDAN, M., BOLITHO, M., AND HOPPE, H. 2006. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SGP '06, 61-70.
35. LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2008. A closedform solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (Feb), 228-242.
36. LI, Y., SUN, J., TANG, C.-K., AND SHUM, H.-Y. 2004. Lazy snapping. ACM Trans. Graph. 23, 3 (Aug.), 303-308.
37. LI, H., WEISE, T., AND PAULY, M. 2010. Example-based facial rigging. ACM Trans. Graph. 29, 4 (July), 32:1-32:6.
38. LI, H., YU, J., YE, Y., AND BREGLER, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4 (July), 42:1-42:10.
39. LI, H., TRUTOIU, L., OLSZEWSKI, K., WEI, L., TRUTNA, T., HSIEH, P.-L., NICHOLLS, A., AND MA, C. 2015. Facial performance sensing head-mounted display. ACM Trans. Graph. 34, 4, 47.
40. LIU, Y., XU, F., CHAI, J., TONG, X., WANG, L., AND HUO, Q. 2015. Video-audio driven real-time facial animationy. ACM Trans. Graph. 34, 6, 182.
41. LUO, L., LI, H., AND RUSINKIEWICZ, S. 2013. Structure-aware hair capture. ACM Trans. Graph. 32, 4, 76.
42. MCMILLAN, L. 1997. An Image-Based Approach to Three-Dimensional Computer Graphics. PhD thesis, University of North Carolina at Chapel Hill.
43. NAGANO, K., FYFFE, G., ALEXANDER, O., BARBIÇ, J., LI, H., GHOSH, A., AND DEBEVEC, P. 2015. Skin microstructure deformation with displacement map convolution. ACM Trans. Graph. 34, 4 (July), 109:1-109:10.
44. PARIS, S., CHANG, W., KOZHUSHNYAN, O. I., JAROSZ, W., MATUSIK, W., ZWICKER, M., AND DURAND, F. 2008. Hair photobooth: Geometric and photometric acquisition of real hairstyles. ACM Trans. Graph. 27, 3 (Aug.), 30:1-30:9.
45. PIGHIN, F., SZELISKI, R., AND SALESIN, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In Int. Conf. Computer Vision, 143-150.
46. SARAGIH, J., LUCEY, S., AND COHN, J. 2011. Real-time avatar animation from a single image. In AFGR, 213-220.
47. SEITZ, S. M., AND DYER, C. R. 1996. View morphing. In Proceedings of ACM SIGGRAPH, ACM, New York, NY, USA, SIGGRAPH '96, 21-30.
48. SHI, F., WU, H.-T., TONG, X., AND CHAI, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graph. 33, 6, 222.
49. SHUM, H.-Y., CHAN, S.-C., AND KANG, S. B. 2007. Image-Based Rendering. Springer.
50. STICH, T., LINZ, C., ALBUQUERQUE, G., AND MAGNOR, M. 2008. View and time interpolation in image space. In Computer Graphics Forum, vol. 27, Wiley Online Library, 1781-1787.
51. THIES, J., ZOLLHÖFER, M., NIESSNER, M., VALGAERTS, L., STAMMINGER, M., AND THEOBALT, C. 2015. Real-time expression transfer for facial reenactment. ACM Trans. Graph. 34, 6, 183.
52. VALGAERTS, L., WU, C., BRUHN, A., SEIDEL, H.-P., AND THEOBALT, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187.
53. VENKATARAMANA, K., LODHAA, S., AND RAGHAVAN, R. 2005. A kinematic-variational model for animating skin with wrinkles. Computer & Graphics 29, 5 (Oct), 756-770.
54. VLASIC, D., BRAND, M., PFISTER, H., AND POPOVIĆ, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (July), 426-433.
55. WEI, Y., OFEK, E., QUAN, L., AND SHUM, H.-Y. 2005. Modeling hair from multiple views. ACM Trans. Graph. 24, 3 (July), 816-820.
56. WEISE, T., LI, H., GOOL, L. V., AND PAULY, M. 2009. Face/off: Live facial puppetry. In Symp. Computer Animation, 7-16.
57. WEISE, T., BOUAZIZ, S., LI, H., AND PAULY, M. 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4 (July), 77:1-77:10.
58. XU, F., LIU, Y., STOLL, C., TOMPKIN, J., BHARAJ, G., DAI, Q., SEIDEL, H.-P., KAUTZ, J., AND THEOBALT, C. 2011. Videobased characters: Creating new human performances from a multi-view video database. In ACM SIGGRAPH 2011 Papers, SIGGRAPH '11, 32:1-32:10.
59. XU, Z., WU, H.-T., WANG, L., ZHENG, C., TONG, X., AND QI, Y. 2014. Dynamic hair capture using spacetime optimization. ACM Trans. Graph. 33, 6 (Nov.), 224:1-224:11.
60. YANG, F., SHECHTMAN, E., WANG, J., BOURDEV, L., AND METAXAS, D. 2012. Face morphing using 3d-aware appearance optimization. In Proceedings of Graphics Interface 2012, GI '12, 93-99.
61. ZANELLA, V., VARGAS, H., AND ROSAS, L. V. 2007. Active shape models and evolution strategies to automatic face morphing. In Proceedings of the 8th International Conference on Adaptive and Natural Computing Algorithms, Part II, Springer-Verlag, Berlin, Heidelberg, ICANNGA '07, 564-571.
62. ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548-558.