Realtime facial animation with on-the-fly correctives

ACM Transactions on Graphics

Volumn 32, Issue 4, 2013, Pages

  Li, Hao ^a,b   Yu, Jihun ^a   Ye, Yuting ^a   Bregler, Chris ^a,c  

^a Industrial Light and Magic ^*  (United States)

^b Div of Biokinesiology and Physical Therapy   (United States)

^c NEW YORK UNIVERSITY   (United States)

Author keywords

Correctives; Depth sensor; Facial animation; Facial retargeting; Incremental PCA; Performance capture; Real time tracking

Indexed keywords

CORRECTIVES; DEPTH SENSORS; FACIAL ANIMATION; FACIAL RETARGETING; INCREMENTAL PCA; PERFORMANCE CAPTURE; REAL TIME TRACKING;

SENSORS;

ANIMATION;

EID: 84880826267     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2461912.2462019     Document Type: Article

References (50)

1. ALEXANDER, O., ROGERS, M., LAMBETH, W., CHIANG, M., AND DEBEVEC, P. 2009. The digital Emily project: photoreal facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1-12:15.
2. BEELER, T., HAHN, F., BRADLEY, D., BICKEL, B., BEARDSLEY, P., GOTSMAN, C., SUMNER, R. W., AND GROSS, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30 (August), 75:1-75:10.
3. BICKEL, B., LANG, M., BOTSCH, M., OTADUY, M. A., AND GROSS, M. 2008. Pose-space animation and transfer of facial details. In Proceedings of the 2008 ACM SIGGRAPH/ Eurographics Symposium on Computer Animation.
4. BLACK, M. J., AND YACOOB, Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proceedings of the Fifth International Conference on Computer Vision, IEEE Computer Society, Washington, DC, USA, ICCV '95, 374.
5. BLANZ, V., AND VETTER, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM Siggraph 99, ACM Press/Addison-Wesley Publishing Co., 187-194.
6. BORSHUKOV, G., PIPONI, D., LARSEN, O., LEWIS, J. P., AND TEMPELAAR-LIETZ, C. 2005. Universal capture - image-based facial animation for "the matrix reloaded". In ACM SIGGRAPH 2005 Courses.
7. BOTSCH, M., AND SORKINE, O. 2008. On linear variational surface deformation methods. IEEE Transactions on Visualization and Computer Graphics 14, 1 (Jan.), 213-230.
8. BRADLEY, D., HEIDRICH, W., POPA, T., AND SHEFFER, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29 (July), 41:1-41:10.
9. BREGLER, C., AND OMOHUNDRO, S. 1994. Surface learning with applications to lipreading. Advances in neural information processing systems, 43-43.
10. BREGLER, C., COVELL, M., AND SLANEY, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of Computer graphics and interactive techniques.
11. CHAI, J.-X., XIAO, J., AND HODGINS, J. 2003. Vision-based control of 3d facial animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '03, 193-206.
12. CHANDRASEKARAN, S., MANJUNATH, B. S., WANG, Y.-F., WINKELER, J., AND ZHANG, H. 1997. An eigenspace update algorithm for image analysis. CVGIP: Graphical Model and Image Processing, 5, 321-332.
13. CHUANG, E., AND BREGLER, C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep., Stanford University.
14. COLLINS, R., LIU, Y., AND LEORDEANU, M. 2005. Online selection of discriminative tracking features. Pattern Analysis and Machine Intelligence, IEEE Transactions on 27, 10, 1631-1643.
15. COOTES, T. F., EDWARDS, G. J., AND TAYLOR, C. J. 1998. Active appearance models. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Springer, 484-498.
16. COVELL, M., AND BREGLER, C. 1996. Eigen-points. In Image Processing, 1996. Proceedings., International Conference on, vol. 3, IEEE, 471-474.
17. DECARLO, D., AND METAXAS, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38, 2 (July), 99-127.
18. DEMPSTER, A., LAIRD, N., AND RUBIN, D. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 1-38.
19. EKMAN, P., AND FRIESEN, W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto.
20. ESSA, I., BASU, S., DARRELL, T., AND PENTLAND, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of the Computer Animation, IEEE Computer Society, CA '96, 68.
21. FURUKAWA, Y., AND PONCE, J. 2009. Dense 3d motion capture for human faces. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, IEEE, 1674-1681.
22. FYFFE, G., HAWKINS, T., WATTS, C., MA, W.-C., AND DEBEVEC, P. 2011. Comprehensive facial performance capture. In Eurographics 2011.
23. GRABNER, H., LEISTNER, C., AND BISCHOF, H. 2008. Semisupervised on-line boosting for robust tracking. In Proceedings of the 10th European Conference on Computer Vision: Part I, Springer-Verlag, Berlin, Heidelberg, ECCV '08, 234-247.
24. GU, M., AND EISENSTAT, S. C. 1993. A Stable and Fast Algorithm for Updating the Singular Value Decomposition. Tech. Rep. YALEU/DCS/RR-966, Yale University, New Haven, CT.
25. GUENTER, B., GRIMM, C., WOOD, D., MALVAR, H., AND PIGHIN, F. 1998. Making faces. In Proceedings of SIGGRAPH '98, ACM, 55-66.
26. IMAGEMETRICS. 2012. Image metrics live driver SDK http://www.image- metrics.com/livedriver/.
27. KALAL, Z., MATAS, J., AND MIKOLAJCZYK, K. 2009. Online learning of robust object detectors during unstable tracking. In In International Conference on Computer Vision.
28. KIRBY, M., AND SIROVICH, L. 1990. Application of the karhunen-loeve procedure for the characterization of human faces. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 1, 103-108.
29. LI, H., ROIVAINEN, P., AND FORCHEIMER, R. 1993. 3-d motion estimation in model-based facial image coding. IEEE Transactions on PAMI 15, 6, 545-555.
30. LI, H., ADAMS, B., GUIBAS, L. J., AND PAULY, M. 2009. Robust single-view geometry and motion reconstruction. ACM Transactions on Graphics (Proceedings SIGGRAPH Asia 2009) 28, 5.
31. LI, H., WEISE, T., AND PAULY, M. 2010. Example-based facial rigging. ACM Transactions on Graphics (Proceedings SIGGRAPH 2010) 29, 3 (July).
32. PAYSAN, P., KNOTHE, R., AMBERG, B., ROMDHANI, S., AND VETTER, T. 2009. A 3d face model for pose and illumination invariant face recognition.
33. PEARSON, K. 1901. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2, 11, 559-572.
34. PIGHIN, F., AND LEWIS, J. P. 2006. Performance-driven facial animation. In ACM SIGGRAPH 2006 Courses, ACM, New York, NY, USA, SIGGRAPH '06.
35. PIGHIN, F. H., SZELISKI, R., AND SALESIN, D. 1999. Resynthesizing Facial Animation through 3D Model-based Tracking. In Proc. 7th International Conference on Computer Vision, Kerkyra, Greece, 143-150.
36. ROWEIS, S. 1998. EM algorithms for pca and spca. In in Advances in Neural Information Processing Systems, MIT Press, 626-632.
37. RUSINKIEWICZ, S., AND LEVOY, M. 2001. Efficient variants of the icp algorithm. In International Conference on 3-D Digital Imaging and Modeling.
38. RUSINKIEWICZ, S., HALL-HOLT, O., AND LEVOY, M. 2002. Real-time 3D model acquisition. ACM Transactions on Graphics (Proc. SIGGRAPH) 21, 3 (July), 438-446.
39. SARAGIH, J. M., LUCEY, S., AND COHN, J. F. 2011. Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vision 91, 2 (Jan.), 200-215.
40. SKOCAJ, D., AND LEONARDIS, A. 2003. Weighted and robust incremental method for subspace learning. In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, IEEE, 1494-1501.
41. SUGIMOTO, T., FUKUSHIMA, M., AND IBARAKI, T. 1995. A parallel relaxation method for quadratic programming problems with interval constraints. Journal of Computational and Applied Mathematics 60, 12, 219-236.
42. SUMNER, R. W., AND POPOVIĆ , J. 2004. Deformation transfer for triangle meshes. ACM Trans. Graph. 23, 3 (Aug.), 399-405.
43. VALGAERTS, L., WU, C., BRUHN, A., SEIDEL, H.-P., AND THEOBALT, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. In ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2012), vol. 31.
44. VLASIC, D., BRAND, M., PFISTER, H., AND POPOVIĆ , J. 2005. Face transfer with multilinear models. In ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, SIGGRAPH '05, 426-433.
45. WEISE, T., LI, H., GOOL, L. V., AND PAULY, M. 2009. Face/off: Live facial puppetry. In Proceedings of the 2009 ACM SIGGRAPH/ Eurographics Symposium on Computer animation.
46. WEISE, T., BOUAZIZ, S., LI, H., AND PAULY, M. 2011. Realtime performance-based facial animation. ACM Transactions on Graphics (Proceedings SIGGRAPH 2011) 30, 4 (July).
47. WELCH, G., AND BISHOP, G. 1995. An introduction to the kalman filter. Tech. rep., Chapel Hill, NC, USA.
48. WILLIAMS, L. 1990. Performance-driven facial animation. SIGGRAPH Comput. Graph. 24, 4 (Sept.), 235-242.
49. ZHANG, S., AND HUANG, P. 2004. High-resolution, realtime 3d shape acquisition. In Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 3 - Volume 03, IEEE Computer Society, Washington, DC, USA, CVPRW '04, 28-.
50. ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In ACM Annual Conference on Computer Graphics, 548-558.