Expressive speech animation synthesis with phoneme-level controls

Computer Graphics Forum

Volumn 27, Issue 8, 2008, Pages 2096-2113

  Deng, Z ^a   Neumann, U ^b  

^a University of Houston   (United States)

^b University of Southern California   (United States)

Author keywords

Data driven; Facial animation; Facial expression; Motion capture; Phoneme isomap; Speech animation

Indexed keywords

ANIMATION; COST FUNCTIONS; DYNAMIC PROGRAMMING; IMAGE ANALYSIS; LEVEL CONTROL;

DATA DRIVEN; FACIAL ANIMATION; FACIAL EXPRESSIONS; MOTION CAPTURE; PHONEME-ISOMAP; SPEECH ANIMATION;

SPEECH;

EID: 56749182606     PISSN: 01677055     EISSN: 14678659     Source Type: Journal    
DOI: 10.1111/j.1467-8659.2008.01192.x     Document Type: Article

References (55)

1. Arikan O., Forsyth D. A. : Interactive motion generation from examples. ACM Transactions on Graphics 21 (2002 483 490.
2. Blanz V., Basso C., Poggio T., Vetter T. : Reanimating faces in images and video. Computer Graphics Forum 22, 3 (2003 641 650.
3. Bregler C., Covell M., Slaney M. : Video rewrite: Driving visual speech with audio. Proceedings of ACM SIGGRAPH'97 (1997 pp. 353 360.
4. Busso C., Deng Z., Neumann U., Narayanan S. : Natural head motion synthesis driven by acoustic prosody features. Computer Animation and Virtual Worlds 16, 3-4 (2005 283 290.
5. Brand M., Hertzmann A. : Style machines. In Proc. of ACM SIGGRAPH'00 (2000 ACM Press/Addison-Wesley Publishing Co., pp. 183 192.
6. Bevacqua E., Pelachaud C. : Expressive audio-visual speech. Journal of Visualization and Computer Animation 15, 3-4 (2004 297 304.
7. Brand M. : Voice pupperty. Proceedings of ACM SIGGRAPH'99 (1999 pp. 21 28.
8. Blanz V., Vetter T. : A morphable model for the synthesis of 3D faces. Proceedings of ACM SIGGRAPH'99 (1999 pp. 187 194.
9. Chuang E., Bregler C. : Moodswings: Expressive speech animation. ACM Transactions on Graphics 24, 2 (2005 331 347.
10. Chuang E. S., Deshpande H., Bregler C. : Facial expression space learning. In Proceedings of Pacific Graphics'2002 (2002 pp. 68 76.
11. Cao Y., Faloutsos P., Kohler E., Pighin F. : Real-time speech motion synthesis from recorded motions. In Proceedings of Symposium on Computer Animation (2004 pp. 345 353.
12. Cao Y., Faloutsos P., Pighin F. : Unsupervised learning for speech motion editing. In Proceedings of Symposium on Computer Animation (2003 pp. 225 231.
13. Cosatto E., Graf H. P. : Audio-visual unit selection for the synthesis of photo-realistic talking-heads. In Proceedings of ICME (2000 pp. 619 622.
14. Cohen M. M., Massaro D. W. : Modeling coarticulation in synthetic visual speech. In Models and Techniques in Computer Animation, Springer Verlag, N. M. Thalmann, D. Thalmann (eds.) 1993 139 156.
15. http://www.speech.cs.cmu.edu/cgi-bin/cmudict, 2005.
16. Deng Z., Chiang P. Y., Fox P., Neumann U. : Animating blendshape faces by cross-mapping motion capture data. In Proceedings of ACM SIGGGRAPH Symposium on Interactive 3D Graphics and Games (2006 pp. 43 48.
17. Deng Z., Lewis J. P., Neumann U. : Synthesizing speech animation by learning compact speech co-articulation models. In Proceedings of Computer Graphics International (2005 pp. 19 25.
18. Deng Z., Neumann U. : eFASE: Expressive facial animation synthesis and editing with phoneme-level controls. In Proceedings of Symposium on Computer Animation (Vienna, Austria, 2006 Eurographics Association, pp. 251 259.
19. Deng Z., Neumann U. : Data-Driven 3D Facial Animation. Springer-Verlag Press, 2007.
20. Deng Z., Noh J. Y. : Computer facial animation: A survey. In Data-Driven 3D Facial Animation (Eds : Z. Deng and U. Neumann 2007 Springer Press, pp. 1 28.
21. Deng Z., Neumann U., Lewis J. P., Kim T. Y., Bulut M., Narayanan S. : Expressive facial animation synthesis by learning speech co-articulations and expression spaces. IEEE Transactions on Visualization and Computer Graphics 12, 6 (2006 1523 1534.
22. Ezzat T., Geiger G., Poggio T. : Trainable videorealistic speech animation. ACM Transactions on Graphics 21, 3 (2002 388 398.
23. http://www.cstr.ed.ac.uk/projects/festival/, 2004.
24. Guenter B., Grimm C., Wood D., Malvar H., Pighin F. : Making faces. Proceedings of ACM SIGGRAPH'98 (1998 pp. 55 66.
25. Grochow K., Martin S. L., Hertzmann A., Popović Z. : Style-based inverse kinematics. ACM Transactions on Graphics 23, 3 (2004 522 531.
26. Hastie T., Ribshirani R., Friedman J. : The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer-Verlag, 2001.
27. Kovar L., Gleicher M. : Automated extraction and parameterization of motions in large data sets. ACM Transactions on Graphics 23, 3 (2004 559 568.
28. Kovar L., Gleicher M., Pighin F. : Motion graphs. ACM Transactions on Graphics 21, 3 (2002 473 482.
29. Kshirsagar S., Garchery S., Thalmann N. M. : Feature point based mesh deformation applied to mpeg-4 facial animation. In DEFORM'00/AVATARS'00: Proceedings of AVATARS'2000 Workshop on Deformable Avatars (Deventer, The Netherlands, The Netherlands, 2001 Kluwer, B.V., pp. 24 34.
30. Kähler K., Haber J., Seidel H. P. : Geometry-based muscle modeling for facial animation. In Proceedings of Graphics Interface'2001 (Toronto, Ont., Canada, Canada, 2001 Canadian Information Processing Society, pp. 37 46.
31. King S. A., Parent R. E. : Creating speech-synchronized animation. IEEE Transactions on Visual Graphics 11, 3 (2005 341 352.
32. Kshirsagar S., Thalmann N. M. : Visyllable based speech animation. Computer Graphics Forum 22, 3 (2003 631 639.
33. Lewis J. P., Cordner M., Fong N. : Pose space deformation: A unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of ACM SIGGRAPH'2000 (2000 pp. 165 172.
34. Lewis J. P. : Automated lip-sync: Background and techniques. Journal of Visualization and Computer Animation 2, 4 (1991 118 122.
35. Lee Y. C., Terzopoulos D., Waters K. : Realistic modeling for facial animation. Proceedings of ACM SIGGRAPH'95 (1995 pp. 55 62.
36. Li Y., Wang T., Shum H. Y. : Motion texture: a two-level statistical model for character motion synthesis. ACM Transactions on Graphics 21, 3 (2002 465 472.
37. Ma J., Cole R., Pellom B., Ward W., Wise B. : Accurate automatic visible speech synthesis of arbitrary 3D model based on concatenation of diviseme motion capture data. Computer Animation and Virtual Worlds 15 (2004 1 17.
38. Ma J., Cole R., Pellom B., Ward W., Wise B. : Accurate visible speech synthesis based on concatenating variable length motion capture data. IEEE Transactions on Visualization and Computer Graphics 15, 5 (2005 485 500.
39. Noh J. Y., Neumann U. : Expression cloning. In Proceedings of ACM SIGGRAPH'01 (2001 pp. 277 288.
40. Pullen K., Bregler C. : Motion capture assisted animation: Texturing and synthesis. ACM Transactions on Graphics 21, 3 (2002 501 508.
41. Pelachaud C. : Communication and Coarticulation in Facial Animation. PhD Thesis, University of Pennsylvania (1991).
42. Pighin F., Hecker J., Lischinski D., Szeliski R., Salesin D. H. : Synthesizing realistic facial expressions from photographs. In Proceedings of ACM SIGGRAPH'98 (1998 pp. 75 84.
43. Pierre D. A. : Optimization Theory with Applications. Dover Publications, Inc., 1986.
44. Pyun H., Kim Y., Chae W., Kang H. W., Shin S. Y. : An example-based approach for facial expression cloning. In Proceedings of Symposium on Computer Animation (2003 pp. 167 176.
45. Parke F. I., Waters K. : Computer Facial Animation. A K Peters, Wellesley, Massachusets, 1996.
46. Rose C., Cohen M. F., Bodenheimer B. : Verbs and adverbs: Multidimensional motion interpolation. IEEE CG&A 18, 5 (1998 32 40.
47. Safonova A., Hodgins J. K., Pollard N. S. : Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Transactions Graphics 23, 3 (2004 514 521.
48. Sifakis E., Neverov I., Fedkiw R. : Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3 (2005 417 425.
49. Tenenbaum J. B., Silva V. D., Langford J. C. : A global geometric framework for nonlinear dimensionality reduction. Science 290, 5500 (2000 2319 2333.
50. Vlasic D., Brand M., Pfister H., Popović J. : Face transfer with multilinear models. ACM Transactions on Graphics 24, 3 (2005 426 433.
51. Waters K., Frisble J. : A coordinated muscle model for speech animation. Proceedings of Graphics Interface'95 (1995 pp. 163 170.
52. Williams L. : Performance-driven facial animation. In Proceedings of ACM SIGGRAPH'90 (1990 ACM Press, pp. 235 242.
53. Witkin A., Popović Z. : Motion warping. In Proceedings of ACM SIGGRAPH'95 (1995 pp. 105 108.
54. Zhang Q., Liu Z., Guo B., Shum H. : Geometry-driven photorealistic facial expression synthesis. In Proceedings of Symposium on Computer Animation (2003 pp. 177 186.
55. Zhang L., Snavely N., Curless B., Seitz S. M. : Spacetime faces: High resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (2004 548 558.