Trainable videorealistic speech animation

ACM Transactions on Graphics

Volumn 21, Issue 3, 2002, Pages 388-398

  Ezzat, Tony ^a   Geiger, Gadi ^a   Poggio, Tomaso ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Facial animation; Facial modeling; Lip synchronization; Morphing; Optical flow; Speech synthesis

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION; GRAPHICAL USER INTERFACES; IMAGE ANALYSIS; IMAGE UNDERSTANDING; SPEECH SYNTHESIS; THREE DIMENSIONAL COMPUTER GRAPHICS; VIDEO CAMERAS;

MULTIDIMENSIONAL MORPHABLE MODEL; SPEECH ANIMATION; TRAJECTORY SYNTHESIS TECHNIQUE;

ANIMATION;

EID: 0036989560     PISSN: 07300301     EISSN: None     Source Type: Journal    
DOI: 10.1145/566654.566594     Document Type: Conference Paper

References (44)

1. Barron, J. L., Fleet, D. J., and Beauchemin, S. S. 1994, Performance of optical flow techniques. International Journal of Computer Vision 12, 1, 43-77.
2. Beier, T., and Neely, S. 1992. Feature-based image metamorphosis. In Computer Graphics (Proceedings of ACM SIGGRAPH 92), vol. 26(2), ACM, 35-42.
3. Bergen, J., Anandan, P., Hanna, K., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Proceedings of the European Conference on Computer Vision, 237-252.
4. Beymer, D., and Poggio, T. 1996. Image representations for visual learning. Science 272, 1905-1909.
5. Beymer, D., Shashua, A., and Poggio, T. 1993, Example based image analysis and synthesis. Tech. Rep. 1431, MIT AI Lab.
6. Bishop, C. M. 1995. Neural Networks for Pattern Recognition. Clarendon Press, Oxford.
7. Black, A., and Taylor, P. 1997. The Festival Speech Synthesis System. University of Edinburgh.
8. Black, M., Fleet, D., and Yacoob, Y. 2000. Robustly estimating changes in image appearance. Computer Vision and Image Understanding. Special Issue on Robust Statistical Techniques in Image Understanding, 8-31.
9. Blanz, V., and Vetter, T. 1999, A morphable model for the synthesis of 3D faces. In Proceedings of SIGGRAPH 2001. ACM Press / ACM SIGGRAPH. Los Angeles. A. Rockwood, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM, 187-194.
10. Brand, M., and Hertzmann, A. 2000. Style machines. In Proceedings of SIGGRAPH 2000, ACM Press / ACM SIGGRAPH, K. Akeley, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM, 183-192.
11. Brand, M. 1999. Voice puppetry. In Proceedings of SIGGRAPH 1999, ACM Press / ACM SIGGRAPH, Los Angeles, A. Rockwood, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM, 21-28.
12. Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of SIGGRAPH 1997, ACM Press / ACM SIGGRAPH, Los Angeles, CA, Computer Graphics Proceedings, Annual Conference Series, ACM, 353-360.
13. Brooke, N., and Scott, S. 1994. Computer graphics animations of talking faces based on stochastic models. In Intl. Symposium on Speech, Image Processing, and Neural Networks.
14. Burt, P. J., and Adelson, E. H. 1983. The laplacian pyramid as a compact image code. IEEE Trans. on Communications COM-31, 4 (Apr.), 532-540.
15. Chen, S. E., and Williams, L. 1993. View interpolation for image synthesis. In Proceedings of SIGGRAPH 1993, ACM Press / ACM SIGGRAPH, Anaheim, CA, Computer Graphics Proceedings, Annual Conference Series, ACM, 279-288.
16. Cohen, M. M., and Massaro, D. W. 1993. Modeling coarticulation in synthetic visual speech. In Models and Techniques in Computer Animation, N. M. Thalmann and D. Thalmann, Eds. Springer-Verlag, Tokyo, 139-156.
17. Cootes, T. F., Edwards, G. J., and Taylor, C. J. 1998. Active appearance models. In Proceedings of the European Conference on Computer Vision.
18. Cormen, T. H., Leiserson, C. E., and Rivest, R. L. 1989. Introduction to Algorithms, The MIT Press and McGraw-Hill Book Company.
19. Cosatto, E., and Graf, H. 1998. Sample-based synthesis of photorealistic talking heads. In Proceedings of Computer Animation '98, 103-110.
20. Ezzat, T., and Poggio, T. 2000. Visual speech synthesis by morphing visemes. International Journal of Computer Vision 38, 45-57.
21. Girosi, F., Jones, M., and Poggio, T. 1993. Priors, stabilizers, and basis functions: From regularization to radial, tensor, and additive splines. Tech. Rep. 1430, MIT AI Lab, June.
22. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings of SIGGRAPH 1998, ACM Press / ACM SIGGRAPH. Orlando, FL. Computer Graphics Proceedings, Annual Conference Series, ACM, 55-66.
23. Horn, B. K. P., and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence 17, 185-203.
24. Huang, X., Alleva, F., Hon, H.-W., Hwang, M.-Y., Lee, K.-F., and Rosenfeld, R. 1993. The SPHINX-II speech recognition system: an overview (http://sourceforge.net/projects/cmusphinx/). Computer Speech and Language 7, 2, 137-148.
25. Jones, M., and Poggio, T. 1998. Multidimensional morphable models: A framework for representing and maching object classes. In Proceedings of the International Conference on Computer Vision.
26. Lee, S. Y., Chwa, K. Y., Shin, S. Y., and Wolberg, G. 1995. Image metemorphosis using snakes and free-form deformations. In Proceedings of SIGGRAPH 1995, ACM Press / ACM SIGGRAPH, vol. 29 of Computer Graphics Proceedings, Annual Conference Series, ACM, 439-448.
27. Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. In Proceedings of SIGGRAPH 1995, ACM Press / ACM SIGGRAPH, Los Angeles, California, Computer Graphics Proceedings, Annual Conferece Series, ACM, 55-62.
28. Lee, S. Y., Wolberg, G., and Shin, S. Y. 1998. Polymorph: An algorithm for morphing among multiple images. IEEE Computer Graphics Applications 18, 58-71.
29. LeGoff, B., and Benoit, C. 1996. A text-to-audiovisual-speech synthesizer for french. In Proceedings of the International Conference on Spoken Language Processing (ICSLP).
30. Masuko, T., Kobayashi, T., Tamura, M., Masubuchi, J., and Tokuda, K. 1998. Text-to-visual speech synthesis based on parameter generation from hmm. In ICASSP.
31. Moulines, E., and Charpentier, F. 1990. Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones. Speech Communication 9, 453-467.
32. Parke, F. I. 1974. A Parametric Model of Human Faces. PhD thesis, University of Utah.
33. Pearce, A., Wyvill, B., Wyvill, G., and Hill, D. 1986. Speech and expression: A computer solution to face animation. In Graphics Interface.
34. Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 1998, ACM Press / ACM SIGGRAPH, Orlando, FL, Computer Graphics Proceedings, Annual Conference Series, ACM, 75-84.
35. Poggio, T., and Vetter, T. 1992, Recognition and structure from one 2D model view: observations on prototypes, object classes and symmetries. Tech. Rep. 1347, Artificial Intelligence Laboratory, Massachusetts Institute of Technology.
36. Roweis, S. 1998, EM algorithms for PCA and SPCA. In Advances in Neural Information Processing Systems, The MIT Press, M. I. Jordan, M. J. Kearns, and S. A. Solla, Eds., vol. 10.
37. Scott, K., Kagels, D., Watson, S., Rom, H., Wright, J., Lee, M., and Hussey, K. 1994. Synthesis of speaker facial movement to match selected speech sequences. In Proceedings of the Fifth Australian Conference on Speech Science and Technology, vol. 2, 620-625.
38. Silander, K., and Beskow, J. 2000, Wavesurfer - an open source speech tool. In Proc of ICSLP, vol. 4, 464-467.
39. Tenenbaum, J. B., de Silva, V., and Langford, J. C. 2000, A global geometric framework for nonlinear dimensionality reduction, Science 290 (Dec) 2319-2323.
40. Tipping, M. E., and Bishop, C. M. 1999. Mixtures of probabilistic principal component analyzers, Neural Computation 11, 2, 443-482.
41. Wahba, G. 1990. Splines Models for Observational Data. Series in Applied Mathematics, Vol. 59, SIAM, Philadelphia.
42. Waters, K. 1987. A muscle model for animating three-dimensional facial expressions. In Computer Graphics (Proceedings of ACM SIGGRAPH 87), vol. 21(4), ACM 17-24.
43. Watson, S., Wright, J., Scott, K., Kagels, D., Freda, D., and Hussey, K. 1997. An advanced morphing algorithm for interpolating phoneme images to simulate speech. Jet Propulsion Laboratory, California Institute of Technology.
44. Wolberg, G. 1990. Digital Image Warping. IEEE Computer Society Press, Los Alamitos, CA.