Accurate automatic visible speech synthesis of arbitrary 3D models based on concatenation of diviseme motion capture data

Computer Animation and Virtual Worlds

Volumn 15, Issue 5, 2004, Pages 485-500

  Ma, Jiyong ^a   Cole, Ronald ^b   Pellom, Bryan ^b   Ward, Wayne ^b   Wise, Barbara ^b  

^a UNIVERSITY OF COLORADO   (United States)

^b NONE

Author keywords

Animated speech; Coarticulation modelling; Face animation; Speech animation; Visible speech; Visual speech synthesis

Indexed keywords

ANIMATION; AUDIO SYSTEMS; CALIBRATION; CAMERAS; IMAGE PROCESSING; MOTION ESTIMATION; MOTION PICTURES; THREE DIMENSIONAL COMPUTER GRAPHICS;

ANIMATED SPEECH; COARTICULATION MODELING; FACE ANIMATION; SPEECH ANIMATION; VISIBLE SPEECH; VISUAL SPEECH SYNTHESIS;

SPEECH SYNTHESIS;

EID: 23444432515     PISSN: 15464261     EISSN: None     Source Type: Journal    
DOI: 10.1002/cav.11     Document Type: Article

References (39)

1. Bregler C, Covell M, Slaney M. Video rewrite: driving visual speech with audio. In Proceedings of ACM SIGGRAPH Computer Graphics. ACM Press: New York, 1997; 353-360.
2. Ezzat T, Geiger G, Poggio T. Trainable video realistic speech animation. In Proceedings of ACM SIGGRAPH Computer Graphics, San Antonio, TX, 2002; pp 388-398.
3. Ezzat T, Poggio T. MikeTalk: a talking facial display based on morphing visemes. In Proceedings of Computer Animation, Philadelphia, PA, 1998; pp 96-102.
4. Cosatto E, Graf HP. Sample-based synthesis of photorealistic talking heads. In Proceedings of Computer Animation, Philadelphia, PA, 1998; pp 103-110.
5. Graf HP, Cosatto E, Ezzat T. Face analysis for the synthesis of photo-realistic talking heads. In Proceedings of Face and Gesture Recognition, 2000; pp 189-194.
6. Pellom B, Hacioglu K. Recent improvements in the SONIC ASR system for noisy speech: the SPINE task. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vol. 1, Hong Kong, 6-10 April 2003; pp 4-7.
7. Cole R, Massaro DW, Villiers J de, Rundle B, Shobaki K, Wouters J, Cohen M, Beskow J, Stone P, Connors P, Tarachow A, Solcher D. New tools for interactive speech and language training: using animated conversational agents in the classrooms of profoundly deaf children. In Proceedings of ESCA/SOCRRATES Workshop on Method and Tool Innovations for Speech Science Education, 1999.
8. Parke F. Computer generated animation of faces. In Proceedings of ACM National Conference, Boston, MA, 1972; pp 451-457.
9. Terzopoulos D, Waters K. Physically-based facial modeling, analysis, and animation. Journal of Visualization and Computer Animation 1990; 1(4): 73-80.
10. Williams L. Performance-driven facial animation. In Proceedings of ACM SIGGRAPH Computer Graphics 1990; 24(4): 235-242.
11. Kleiser J. A fast, efficient, accurate way to represent the human face. State of the Art in Facial Animation, ACM, SIGGRAPH Computer Graphics Tutorials 1989; 22: 37-40.
12. Magnenat Thalmann N, Primeau E, Thalmann D. Abstract muscle action procedures for human face animation. Visual Computer 1988; 3(5): 290-297.
13. Noh JY, Neumann U. Expression cloning. In Proceedings of of ACM SIGGRAPH Computer Graphics, Los Angeles CA, 2001; pp 277-288.
14. Chuang E, Deshpande H, Bregler C. Facial expression space learning. In Proceedings of Pacific Graphics, Beijing, 2002; pp 68-76.
15. Guenter B, Grimm C, Wood D, Malvar H, Pighin F. Making faces. In Proceedings of SIGGRAPH Computer Graphics 1998; pp 55-66.
16. Kshirsagar S, Molet T, Magnenat-Thalmann N. Principal components of expressive speech animation. In Proceedings of Computer Graphics International 2001; pp 38-44.
17. Bregler C, Loeb L, Chuang E, Deshpande H. Turning to the masters: motion capturing cartoons. In Proceedings of SIGGRAPH Computer Graphics 2002; pp 399-407.
18. Jackson PL. The theoretical minimal unit for visual speech perception: visemes and coarticulation. Volta Review 1988; 90(5): 99-115.
19. Woodward MF, Barber CG. Phoneme perception in lip-reading. Journal of Speech and Hearing Research 1960; 3: 212-220.
20. Hardcastle WJ, Hewlett N. Coarticulation: Theory, Data and Techniques. Cambridge University Press: Cambridge, UK, 1999.
21. Woods JC. Lip-Reading: A Guide for Beginners (2nd edn). Royal National Institute for Deaf people: London, 1994.
22. Pelachaud C, Badler N, Steedman M. Linguistic issues in facial animation. In Proceedings of Computer Animation, Magnenat-Thalmann N, Thalmann D (eds). Springer: Berlin, 1991; 15-30.
23. Kent RD, Minifie FD. Coarticulation in recent speech production models. Journal of Phonetics 1977; 5: 115-135.
24. Cohen MM, Massaro DW. Modeling coarticulation in synthetic visual speech. In Proceedings of Computer Animation, Thalman NM, Thalman D (eds). Springer: Tokyo, 1993; 139-156.
25. Löfqvist A. Speech as audible gestures. In Speech Production and Speech Modeling, Hardcastle WJ, Marchai A (eds). Kluwer: Dordrecht, 1990; 289-322.
26. Nielson G. Scattered data modeling. IEEE Computer Graphics and Applications 1993; 13(1): 60-70.
27. Vetter T, Poggio T. Linear object classes and image synthesis from a single example image. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997; 19(7): 733-742.
28. Blanz V, Vetter T. A morphable model for synthesis of 3D faces. In Proceedings of SIGGRAPH Computer Graphics, Los Angeles, 1999; pp 187-194.
29. Kalberer GA, Gool LV. Face animation based on observed 3D speech dynamics. In Proceedings of Computer Animation, 2001; pp 20-27.
30. Pighin F, Szeliski R, Salesin DH. Modeling and animating realistic faces from images. International Journal of Computer Vision 2002; 50(2): 143-169.
31. Blanz V, Basso C, Poggio T, Vetter T. Reanimating faces in images and video. Computer Graphics Forum 2003; 22(3): 641-650.
32. Gertz EM, Wright SJ. Object-oriented software for quadratic programming. ACM Transactions on Mathematical Software 2003; 29: 58-81.
33. Bai ZJ, Demmel J, Dongarra J, Ruhe A, Vorst HVD. Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. Society for Industrial and Applied Mathematics: Philadelphia, PA, 2000.
34. n blending of curves and surfaces Blending functions. Journal of Visual Computer 2001; 17(1): 1-13.
35. Bellman R. Dynamic Programming. Princeton University Press: Princeton, NJ, 1957.
36. Feng G. Data smoothing by cubic spline filters. IEEE Transactions on Signal Processing 1998; 46(10): 2790-2796.
37. University of Edinburgh. The festival speech synthesis system: http://www.cstr.ed.ac.uk/projects/festival/ [2003].
38. Ma JY, Cole R. Animating visible speech and facial expressions. Visual Computer Journal 2004 (to appear).
39. Cole R, Van Vuuren S, Pellom B, Hacioglu K, Ma JY, Movellan J, Schwartz S, Wade-Stein D, Ward W, Yan J. Perceptive animated interfaces: first steps toward a new paradigm for human-computer interaction. Proceedings of the IEEE 2003; 91(9): 1391-1405.