Transferring of speech movements from video to 3D face space

IEEE Transactions on Visualization and Computer Graphics

Volumn 13, Issue 1, 2007, Pages 58-69

  Zha, Hongbin ^a  

^a PEKING UNIVERSITY   (China)

Author keywords

Facial animation; machine learning; performance driven animation; speech synchronization; visual speech synthesis

Indexed keywords

ALGORITHM; ARTICLE; COMPUTER ASSISTED DIAGNOSIS; COMPUTER GRAPHICS; FACE; HISTOLOGY; HUMAN; IMAGE ENHANCEMENT; INFORMATION RETRIEVAL; METHODOLOGY; MOVEMENT (PHYSIOLOGY); PHYSIOLOGY; SPEECH; THREE DIMENSIONAL IMAGING; VIDEORECORDING;

ALGORITHMS; COMPUTER GRAPHICS; FACE; HUMANS; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; IMAGING, THREE-DIMENSIONAL; INFORMATION STORAGE AND RETRIEVAL; MOVEMENT; SPEECH; VIDEO RECORDING;

EID: 35349018832     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2007.22     Document Type: Article

References (48)

1. I. Albrecht, J. Haber, K. Kahler, M. Schroder, and H.-P. Seidel, “May I Talk to You? Facial Animation from Text,” Proc. 10th Pacific Conf. Computer Graphics and Applications, pp 77–86, 2002
2. S. King and R. Parent, “Creating Speech-Synchronized Animation,” IEEE Trans. Visualization and Computer Graphics, vol. 11, no. 3, pp. 341–352, May/June 2005
3. K. Waters and T.M. Levergood, “DECface: An Automatic Lip-Synchronization Algorithm for Synthetic Faces,” Technical Report CRL 93/4, Digital Equipment Corp., Cambridge Research Lab., 1993
4. M. Frydrych and M. Dobsik, “Toolkit for Animation of Finnish Talking Head,” Proc. ISCA Tutorial and Research Workshop Audio Visual Speech (AVSP '03), pp. 199–204, 2003
5. S. Shan and W. Gao, “Individual 3D Face Synthesis Based on Orthogonal Photos and Speech-Driven Facial Animation,” Proc. Int'l Conf. Image Processing, vol. 3, pp. 238–241, 2000
6. C. Bregler, M. Covell, and M. Slaney, “Video Rewrite: Driving Visual Speech with Audio,” Proc. ACM SIGGRAPH '97, pp. 353–360, 1997
7. Y. Cao, P. Faloutsos, E. Kohler, and F. Pighin, “Real-Time Speech Motion Synthesis from Recorded Motions,” Proc. ACM SIG-GRAPH/Eurographics Symp. Computer Animation, pp. 347–355, 2004
8. Y. Cao, W.C. Tien, P. Faloutsos, and F. Pighin, “Expressive Speech-Driven Facial Animation,” ACM Trans. Graphics, vol. 24, no. 4, pp. 1283–1302, 2005
9. I. Ezzat, G. Geiger, and T. Poggio, “Trainable Videorealistic Speech Animation,” ACM Trans. Graphics, vol. 21, no. 3, pp. 388–398, 2002
10. Y.-J. Chang and T. Ezzat, “Transferable Videorealistic Speech Animation,” Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation (SCA '05), pp. 143–151, 2005
11. J. Chai, J. Xiao, and J. Hodgins, “Vision-Based Control of 3D Facial Animation,” Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation (SCA '03), pp. 193–206, 2003
12. E.S. Chuang, H. Deshpande, and C. Bregler, “Facial Expression Space Learning,” Proc. 10th Pacific Conf. Computer Graphics and Applications, pp. 68–76, 2002
13. D. Vlasic, M. Brand, H. Pfister, and J. Popovic, “Face Transfer with Multilinear Models,” ACM Trans. Graphics, vol. 24, no. 3, pp. 426–433, 2005
14. M. Brand, “Voice Puppetry,” Proc. ACM SIGGRAPH '99, pp. 21–28, 1999
15. I. Jolliffe, Principal Component Analysis. Springer-Verlag, 1986
16. B. Allen, B. Curless, and Z. Popovic, “The Space of All Body Shapes: Reconstruction and Parameterization from Range Scans,” Proc. ACM SIGGRAPH '03, pp. 587–594, 2003
17. H. Pyun, Y. Kim, W. Chae, H.Y. Kang, and S.Y. Shin, “An Example-Based Approach for Facial Expression Cloning,” Proc. Symp. Computer Animation, pp. 167–176, 2003
18. E.S. Chuang and C. Bregler, Performance Driven Facial Animation Using Blendshape Interpolation, Technical Report CS-TR-2002-02, Computer Science, Stanford Univ., 2002
19. V. Blanz and T. Vetter, “A Morphable Model for the Synthesis of 3D Faces,” Proc. ACM SIGGRAPH '99, pp. 187–194, 1999
20. M. Turk and A. Pentland, “Face Recognition Using Eigenfaces,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 586–591, 1991
21. J.B. Kruskal and M. Wish, Multidimensional Scaling. Sage Publications, 1978
22. A. Hyvarinen, J. Karhunen, and E. Oja, Independent Component Analysis. John Wiley & Sons, 2001
23. Y. Cao, P. Faloutsos, and F. Pighin, “Unsupervised Learning for Speech Motion Editing,” Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation, pp. 225–231, 2003
24. J.B. Tenenbaum, V. De Silva, and J.C. Langford, “A Global Geometric Framework for Nonlinear Dimensionality Reduction,” Science, vol. 290, no. 5500, pp. 2319–2323, 2000
25. S. Roweis and L. Saul, “Nonlinear Dimensionality Reduction by Locally Linear Embedding,” Science, vol. 290, no. 5500, pp. 2323–2326, 2000
26. Y. Pei and H. Zha, “Transferring Speech Video onto 3D Realistic Human Faces,” Proc. Pacific Graphics Conf. '05, pp. 13–15, 2005
27. C. de Juan and B. Bodenheimer, “Cartoon Textures,” Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation, pp. 267–276, 2004
28. C. Hu, Y. Chang, R. Feris, and M. Turk, “Manifold Based Analysis of Facial Expression,” Proc. Computer Vision and Pattern Recognition Workshop, pp. 81–81, 2004
29. Y. Wang, X. Huang, C.S. Lee, S. Zhang, Z. Li, D. Samaras, D. Metaxas, A. Elgammal, and P. Huang, “High Resolution Acquisition, Learning and Transfer of Dynamic 3-D Facial Expressions,” Proc. Eurographics '04, pp. 677–686, 2004
30. T. DeRose, M. Kass, and T. Truong, “Subdivision Surfaces in Character Animation,” Proc. ACM SIGGRAPH '98, pp. 85-94,1998
31. J.-Y. Noh and U. Neumann, “Expression Cloning” Proc. ACM SIGGRAPH '01, pp. 277-288, 2001
32. J.Y. Noh, D. Fidaleo, and U. Neumann, “Animated Deformations with Radial Basis Functions,” Proc. ACM Virtual Reality and Software Technology Conf. (VRST '00), pp. 166–174, 2000
33. K. Waters, “A Muscle Model for Animating Three-Dimensional Facial Expression,” Proc. ACM SIGGRAPH '87, pp. 17–24, 1987
34. Y. Lee, D. Terzopoulos, and K. Waters, “Realistic Modeling for Facial Animations,” Proc. ACM SIGGRAPH '95, pp. 55–62, 1995
35. F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D.H. Salesin, “Synthesizing Realistic Facial Expressions from Photographs,” Proc. ACM SIGGRAPH '98, pp. 75–84, 1998
36. R.M. Koch, M.H. Gross, F.R. Carls, D.F. von Buren, G. Fankhauser, and Y.I.H. Parish, “Simulating Facial Surgery Using Finite Element Methods,” Proc. ACM SIGGRAPH '96, pp. 421–428, 1996
37. P. Joshi, W.C. Tien, M. Desbrun, and F. Pighin, “Learning Controls for Blend Shape Based Realistic Facial Animation,” Proc. Symp. Computer Animation, pp. 187–192, 2003
38. K. Na and M.R. Jung, “Hierarchical Retargeting of Fine Facial Motions,” Computer Graphics Forum, vol. 23, no. 3, pp. 687–695, 2004
39. P.J. Sloan, C.F. Rose, and M.F. Cohen, “Shape by Example,” Proc. Symp. Interactive 3D Graphics, pp. 135–143, 2001
40. V. Blanz, C. Basso, T. Poggio, and T. Vetter, “Reanimating Faces in Images and Video,” Computer Graphics Forum, vol. 22, no. 3, pp. 641–650, 2003
41. T.F. Cootes, G.J. Edwards, and C.J. Taylor, “Active Appearance Models,” Proc. Fifth European Conf. Computer Vision, vol. 2, pp. 484–498, 1998
42. M. Escher, I. Pandzic, and N.M. Thalmann, “Facial Deformations for MPEG-4,” Proc. Computer Animation Conf. '98, pp. 56–62, 1998
43. AAM-API, http://www.imm.dtu.dk/~aam/, 2003
44. R.W. Sumner and J. Popovic, “Deformation Transfer for Triangle Meshes,” Proc. ACM SIGGRAPH '04, pp. 399–405, 2004
45. J. Canny, “A Computational Approach to Edge Detection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679–698, Nov. 1986
46. K. Kahler, J. Haber, H. Yamauchi, and H.-P. Seidel, “Head Shop: Generating Animated Head Models with Anatomical Structure,” Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation (SCA '02), pp. 55–64, 2002
47. H. Hatze, “High-Precision Three-Dimensional Photo-Grammetric Calibration and Object Space Reconstruction Using a Modified DLT-Approach,” J. Biomechanics, vol. 21, pp. 533–538, 1988
48. M.M. Cohen and D.W. Massaro, “Modeling Coarticulation in Synthetic Visual Speech,” Models and Techniques in Computer Animation, N.M. Thalmann and D. Thalmann, eds., pp. 139–156. Springer-Verlag, 1993