Lifelike talking faces for interactive services

Proceedings of the IEEE

Volumn 91, Issue 9, 2003, Pages 1406-1428

  Cosatto, Eric ^a,b   Ostermann, Jörn ^a,b   Graf, Hans Peter ^a,b   Schroeter, Juergen ^a,b  

^a AT AND T LABS RESEARCH   (United States)

^b IEEE   (United States)

Author keywords

Avatar; Computer graphics; Face animation; MPEG 4; Sample based graphics; Speech synthesizer; Text to speech (TTS); Video based rendering; Visual text to speech (VTTS)

Indexed keywords

ANIMATION; COMPUTER GRAPHICS; DATABASE SYSTEMS; ELECTRONIC COMMERCE; IMAGE ANALYSIS; INTERNET; NETWORK PROTOCOLS; PERSONAL COMPUTERS; PERSONAL DIGITAL ASSISTANTS; SPEECH SYNTHESIS;

AVATAR; FACE ANIMATION; MPEG-4; SAMPLE-BASED GRAPHICS; SPEECH SYNTHESIZERS; TEXT-TO-SPEECH (TTS); VIDEO-BASED RENDERING; VISUAL TEXT-TO-SPEECH (VTTS);

HUMAN COMPUTER INTERACTION;

EID: 10044281988     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2003.817141     Document Type: Article

References (112)

1. J. Gratch, J. Rickel, E. Andre, N. Badler, J. Cassell, and B. Petajan, "Creating interactive virtual humans: Some assembly required," IEEE Intell. Syst., vol. 17, pp. 54-63, July/Aug. 2002.
2. A. Hunt and A. Black, "Unit selection in a concatenative speech synthesis system using a large speech database," in Proc. ICASSP, vol. 1, 1996, pp. 373-376.
3. J. Schroeter, The fundamentals of text-to-speech synthesis, presented at VoiceXML Forum. Online Available: htlp://www.voicexml.org/Review/Mar2001/ features/tts.html
4. W. L. Johnson, J. W. Rickel, and J. C. Lester, "Animated pedagogical agents: Face-to-face interaction in interactive learning environments," Int. J. Artif. Intell. Educ., vol. 11, pp. 47-78, 2000.
5. C. Elliott, J. Rickel, and J. Lester, "Lifelike pedagogical agents and affective computing: An exploratory synthesis," in Lecture Notes in Computer Science, Artificial Intelligence Today, M. Wooldridge and M. Veloso, Eds. Heidelberg, Germany: Springer-Verlag, 1999, vol. 1600, pp. 195-212.
6. J. Lester, S. G. Towns, C. B. Callaway, J. L. Voerman, and J. FitzGerald, "Deictic and emotive communication in animated pedagogical agents," in Embodied Conversational Agents. J. Cassell, J. Sullivan, S. Prevost, and E. Churchill, Eds. Boston, MA: MIT Press, 2000, pp. 123-154.
7. H. Andre, T. Rist, and J. Muller, "Guiding the user through dynamically generated hypermedia presentations with a life-like character," in Proc. Intelligent User Interfaces, 1998, pp. 21-28.
8. T. Rist, E. Andre, and J. Muller, "Adding animated presentation agents to the interface," in Proc. Intelligent User Interfaces, 1997, pp. 79-86.
9. A. Don, T. Oren, and B. Laurel, "Guides 3.0." in Proc. Video ACM CHI, 1993, pp. 447-448.
10. S. Gibbs and C. Breiteneder, "Video widgets and video actors," in Proc. VIST, 1993, pp. 179-185.
11. A. E. Milewski and G. E. Blonder, "System and Method for Providing Structured Tours of Hypertext Files," U.S. Patent 5 760 771, June 2, 1998.
12. T. Bickmore, L. Cook, E. Churchill, and J. W. Sullivan. "Animated autonomous personal representatives," in Proc. Int. Conf. Autonomous Agents, 1998. pp. 8-15.
13. J. R. Suler, "From ASCII to holodecks: Psychology of an online multimedia community," presented at the Convention Amer. Psychological Association, Chicago, IL, 1997.
14. I. S. Pandzic, T. K. Capin, E. Lee, N. Magnenat-Thalmann, and D. Thalmann, "A flexible architecture for virtual humans in networked collaborative virtual environments," in Proc. Eurographics, vol. 16, 1997, pp. 177-188.
15. B. Darner, C. Kekenes, and T. Huffman, "Inhabited digital spaces," in Proc. ACM CHI, 1996, pp. 9-10.
16. J. H. Walker, L. Sproull, and R. Subramani, "Using a human face in an interface," in Proc. ACM CHI, 1994, pp. 85-91.
17. L. Sproull, M. Subramani, S. Kiesler, J. H. Walker, and K. Waters, "When the interface is a face," in Proc. Human-Computer Interaction, vol. 11, 1996, pp. 97-124.
18. J. Ostermann, "E-COGENT: An electronic convincing agent," in MPEG-4 Facial Animation: The Standard, implementation and Applications, I. S. Pandzic and R. Forchheimer, Eds. Chichester, U.K.: Wiley, 2002, pp. 253-264.
19. J. Ostermann and D. Millen, "Talking heads and synthetic speech: An architecture for supporting electronic commerce," in Proc. ICME, 2000, p. MA2.3.
20. S. Parise, S. Kiesler, L. Sproull, and K. Waters, "My partner is a real dog: Cooperation with social agents," in Proc CSCW, 1996, pp. 399-408.
21. Anthropics Online. Available: http://www.anthropics.com/
22. Face2face animation Online. Available: http://www.f2fanimation.com
23. Famous3D Online. Available: http://www.iamous3d.com/
24. LifeFX Online. Available: http://www.lifefx.com/
25. Lipsinc Online. Available, http://www.lipsinc.com/
26. Pulse3D Online. Available: http://www.pulse3d.com
27. Y. Lee, D. Terzopoulos, and K. Waters, "Realistic modeling for facial animations," in Proc. ACM SIGGRAPH, 1995, pp. 55-62.
28. P. Kalra, A. Mangili, N. Magnenat-Thalmann, and D. Thalmann, "Simulation of facial muscle actions based on rational free form deformation," in Proc. Eurographics, 1992, pp. 65-69.
29. F. I. Parke and K. Waters, Computer Facial Animation. Wellesley, MA: A. K. Peters, 1996.
30. I. S. Pandzic and R. Forchheimer, MPEG4 Facial Animation - The Standard, Implementations and Applications. Chichester, U.K.: Wiley, 2002.
31. M. M. Cohen und D. W. Massaro, "Modeling coarticulation in synthetic visual speech," in Models and Techniques in Computer Animation, N. Magnenat-Thalmann and D. Thalmann, Eds. Tokyo, Japan: Springer-Verlag, 1993, pp. 139-156.
32. T. Ezzat and T. Poggio, "MikeTalk: A talking facial display based on morphing visemes," Proc. IEEE Computer Animation, pp. 96-102, 1998.
33. T. Ezzat, G. Geiger, and T. Poggio, "Trainable videorealistic speech animation," in Proc. ACM SIGGRAPH, 2002, pp. 388-397.
34. C. Bregler, M. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio," in Proc. ACM SIGGRAPH, 1997, pp. 353-360.
35. E. Cosatto and H. P. Graf. "Sample-based synthesis of photo-realistic talking heads," Proc. IEEE Computer Animation, pp. 103-110, 1998.
36. _, "Photo-realistic talking heads from image samples," IEEE Trans. Multimedia, vol. 2, pp. 152-163, Sept. 2000.
37. 4020/RGB 3D Scanner With Color Digitizer, Cyberware Laboratory, Inc., Monterey, CA, 1990.
38. Eyetronics 3D scanning solutions Online. Available: http://www.cyelronics.com
39. B. Guenter, C. Grimm, D. Wood, H. Malvar, and F. Pighin, "Making faces," in Proc. ACM SIGGRAPH, 1998, pp. 55-66.
40. F. Pighin, J. Hecker, D. Lichinski, R. Szeliski, and D. H. Salesin, "Synthesizing realistic facial expressions from photographs," in Proc. ACM SIGGRAPH, 1998, pp. 75-84.
41. V. Blanz and T. Vetter, "A morphable model for the synthesis of 3D faces," in Proc. ACM SIGGRAPH, 1999, pp. 353-360.
42. G. Kalberer and L. Van Gool, "Lip animation based on observed 3D speech dynamics," in Proc. SPIE, vol. 4309, Videometrics and Optical Methods for 3D Shape Management, S. F. El-Hakim and A. Gruen, Eds., 2001, pp. 16-25.
43. S. Kshirsagar and N. Magnenat-Thalmann, "Virtual humans personified," in Proc. Autonomous Agents Conf. (AAMAS), 2002, pp. 356-359.
44. C. G. Fisher. "Confusions among visually perceived consonants," J. Speech Hearing Res., vol- 11, pp. 796-804, 1968.
45. C. Pdaehaud, "Visual text-to-Speech," in MPEG-4 Facial Animation: The Standard, Implementation and Applications, I. S. Pandzic and K. Forchheimcr, Eds. Chichester, U.K.: Wiley, 2002, pp. 125-140.
46. ISO/IEC IS 14496-2 Visual, 1999.
47. P. Ladefoged, A Course in Phonetics. 4th ed. Fort Worth, TX: Harcourt Brace Jovanovich, 2001.
48. J. Beskow, "Rule-based visual speech synthesis." in Proc. Eurospeech, 1995, pp. 299-302.
49. I. Lehiste, Suprasegmentals. Cambridge, MA: MIT Press, 1970.
50. N. Campbell and A. Black, "Prosody and the selection of source units for concatenative synthesis," in Progress in Speech Svnthesis, J. van Santen, R. Sproat, J. Olive, and J. Hirschberg, Eds. Berlin, Germany: Springer-Verlag, 1996, pp. 279-282.
51. P. Ekman and W. V. Friesen, Unmasking the Face. A Guide to Recognizing Emotions From Facial Clues. Englewood Cliffs, NJ: Prentice-Hall, 1975.
52. J. B. Bavelas and N. Chovil, "Faces in dialogue," in The Psychology of Facial Expression, J. A. Russell and J. M. Fernez-Dos, Eds. Cambridge, U.K.: Cambridge Univ. Press, 1997, pp. 334-346.
53. M. Argyle and M. Cook, Gaze and Mutual Gaze. Cambridge, U.K.: Cambridge Univ. Press, 1976.
54. G. Collier, Emotional Expression. Hillsdale, NJ: Lawrence Erlbaum Assoc., 1985.
55. M. Costa, T. Chen, and F. Lavagetto, "Visual prosody analysis for realistic motion synthesis ot 3D head models," in Proc. Int. Conf. Augmented Virtual Environments and 3D Imaging, 2001, pp. 343-346.
56. H. P. Graf, E. Cosatto, V. Strom, and F. J. Huang, "Visual prosody: Facial movements accompanying speech," in Proc. Int. Conf. Automatic Face and Gesture Recognition, 2002, pp. 396-401.
57. K. Waters and T. Levergood, "An automatic lip-synchronization algorithm for synthetic faces," in Proc. ACM Multimedia. 1994, pp. 149-156.
58. J. Cassell, H. Vilhjálmsson, and T. Bickmore, "BEAT: The behavior expression animation toolkit," in Proc. ACM SIGGRAPH, 2001, pp. 477-486.
59. E. Vatikiotis-Bateson, K. G. Munhall, M. Hirayama, Y. Kasahara, and H. Yehia, "Physiology-based synthesis of audiovisual speech," in Proc. 4th Speech Production Seminar: Models and Data, 1996, pp. 241-244.
60. J. Ostermann, M. Beutnagel, A. Fischer, and Y. Wang, "Integration of talking heads and text-to-speech synthesizers for visual TTS," presented at the ICSLP, Sydney, Australia, 1999.
61. J. Ostermann, "PlayMail - Put words into other people's mouth," in MPEG-4 Facial Animation: The Standard, Implementation and Applications, I. S. Pandzic and R. Forchheimer, Eds. Chichester, U.K.: Wiley, 2002, pp. 241-251.
62. R. Sproat and J. Olive, "An approach to text-to-speech synthesis," in Speech Coding and Synthesis, W. B. Kleijn and K. K. Paliwal, Eds. Amsterdam, The Netherlands: Elsevier, 1995.
63. J. Schroeter, A. Conkie, A. Syrdal, M. Beutnagel, M. Jilka, V. Strom, Y. J. Kim, H. G. Kang, and D. Kapilow, "A perspective on the next challenges for TTS research," presented at the IEEE Signal Processing Workshop Speech Synthesis, Santa Monica, CA, 2002.
64. M. Bulut, S. S. Narayanan, and A. K. Syrdal, "Expressive speech synthesis using a concatenative synthesizer," presented at the ICSLP, Denver, CO, 2002.
65. P. Hong, Z. Wen, and T. S. Huang, "Real-time speech-driven face animation with expressions using neural networks," IEEE Trans. Neural Networks, vol. 13, pp. 100-111, Jan. 2002.
66. M. Brand, "Voice puppetry," in ACM SIGGRAPH, 1999, pp.21-28.
67. S. Nakamura, "HMM-based transmodal mapping from audio speech to talking faces," in Proc. IEEE Int. Workshop Neural Networks Signal Processing, vol. 10, 2000, pp. 33-42.
68. J. P. Lewis and F. I. Parke, "Automated lipsynch and speech synthesis for character animation," in Proc. ACM CHI+CG, 1987, pp. 143-147.
69. M. Nahas, H. Huitric, and M. Saintourens, "Animation of a B-spline figure," Vis. Comput., vol. 5, no. 3, pp. 212-216, 1988.
70. X. Huang, F. Alleva, H. Hon, M. Hwang, K. Lee, and R. Rosenfeld, "The SPHINX-II speech recognition system: An overview," Comput. Speech Lang., vol. 2, pp. 137-148, 1993.
71. L. Williams, "Performance driven facial animation," in Proc. ACM SIGGRAPH, 1990, pp. 235-242.
72. M. Tekalp and J. Ostermann, "Face and 2-D mesh animation in MPEG-4," Signal Process. Image Commun., vol. 15, pp. 387-421, 2000.
73. F. Lavagetto and R. Pockaj, "The facial animation engine: Toward a high-level interface for the design of MPEG-4 compliant animated faces," Proc. IEEE CSVT, vol. 9, no. 2, pp. 277-289, 1999.
74. T. S. Perry, "And the Oscar goes to ...," IEEE Spectr., vol. 38, pp. 42-49, 2001.
75. P. S. Heckbert, "Survey of texture mapping." IEEE Comput. Graph. Appl., pp. 56-67, Nov. 1986.
76. ISO/IEC IS 14496-1 Systems, 1999.
77. ISO/IEC 14772-1: Information Technology-Computer Graphics and Image Processing-The Virtual Reality Modeling Language-Part I: Functional Specification and UTF-8 Encoding, 1997.
78. J. Hartman and J. Wernecke, The VRML Handbook. Reading, MA: Addison-Wesley, 1996.
79. ISO/IEC IS 14496-3 Audio, 1999.
80. T. K. Capin, E. Petajan, and J. Ostermann, "Efficient modeling of virtual humans in MPEG-4," in Proc. ICME, 2000, p. TPS9.1.
81. _, "Very low bitrate coding of virtual human animation in MPEG-4," in Proc. ICME, 2000, p. TPS9.2.
82. F. I. Parke, "Parameterized models for facial animation," IEEE Comput. Graph. Appl., vol. 2, pp. 61-68, Nov. 1982.
83. K. Waters, "A muscle model of animating three dimensional facial expression," Comput. Graph., vol. 22, no. 4, pp. 17-24, 1987.
84. J. Ostermann and E. Haratsch, "An animation definition interface: Rapid design of MPEG-4 compliant animated faces and bodies," in Proc. Int. Workshop Synthetic-Natural Hybrid Coding and Three Dimensional Imaging, 1997, pp. 216-219.
85. Proc. 2nd Int. Workshop Recognition, Analysis and Tracking of Faces and Gestures in Realtime Systems. Los Alamitos, CA: IEEE Comput. Soc. Press, 2001.
86. Proc. 5th Int. Conf. Automatic Face and Gesture Recognition. Los Alamitos, CA: IEEE Comput. Soc. Press, 2002.
87. H. P. Graf, E. Cosatto, and G. Potamianos, "Robust recognition of faces and facial features with a mulli-modal system," in Proc. IEEE Conf. Systems, Man, and Cybernetics, 1997, pp. 2034-2039.
88. R. Y. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses," IEEE J. Robot. Automat., vol. RA-3, pp. 323-344, Aug. 1987.
89. D. G. Lowe, "Fining parameterized three dimensional models to images," IEEE Trans. Pattern Anal. Machine Intell., vol. 13, pp. 441-450, May 1991.
90. D. Oberkampf, D. Dementhon, and L. Davis, "Iterative pose estimation using coplanar feature points," Univ. Maryland, College Park, CVL, CAR TR-677, 1993.
91. D. F. Dementhon and L. S. Davis, "Model-based object pose in 25 lines of code," Int. J. Comput. Vis., vol. 15, pp. 123-141, 1995.
92. H. Li, P. Roivainen, and R. Forcheimer, "3-D motion estimation in model-based facial image coding," IEEE Trans. Pattern Anal Machine Intell., vol. 15, pp. 545-55, June 1993.
93. D. DeCarlo and D. Metaxas, "Optical flow constraints on deformable models with applications to face tracking." Int. J. Comput. Vis., vol. 38, no. 2, pp. 99-127, 2000.
94. C. W. Wightman and D. Talkin, "The Aligner: Text to speech alignment using Markov models and a pronunciation dictionary," presented at the 2nd ESCA/IEEE Workshop Speech Synthesis, New Paltz, NY, 1994.
95. G. Potamianos and H. P. Graf, "Linear discriminant analysis for speechreading," Proc. IEEE Workshop Multimedia Signal Processing, pp. 221-226, 1998.
96. M. Beutnagel, A. Conkie, J. Schroeter, Y. Stylianou, and A. Syrdal, "The AT&T next-gen TTS system," J. Acoust. Soc. Amer., pt. 2, vol. 105, p. 1030, 1999.
97. E. Cosatto, M.S. thesis. Ecole Polytechnique Federale Lausanne, Lausanne, Switzerland, 2002.
98. K. Silverman, M. Beckman, J. Pitrelli, M. Ostendorf, C. Wightman, P. Price, J. Pierrehumbert, and J. Hirschberg, "TOBI: A standard for labeling English prosody," in Proc. ICSLP, vol. 2, 1992, pp. 981-984.
99. ITU Telecom, "Methodology for the subjective assessment of the quality of television pictures,", Recommendation ITU-R BT.500-10 1, 2000.
100. I. S. Pandzic, J. Ostermann, and D. Millen, "User evaluation: Synthetic talking faces for interactive services," Vis. Comput. J., vol. 15, no. 7-8, pp. 330-340, 1999.
101. J. J. Williams and A. K. Katsaggelos, "An HMM-based speech-to-video synthesizer," IEEE Trans. Neural Networks (Special Issue on Intelligent Multimedia), vol. 13, pp. 900-915, July 2002.
102. D. W. Massaro, Perceiving Talking Faces: From Speech Perception to a Behavioral Principle. Cambridge, MA: MIT Press, 1998.
103. E. Levin, R. Pieraccini, W. Eckert, G. DiFabbrizio, and S. Narayanan, "Spoken language dialogue: From theory to practice," presented at the IEEE Workshop Automatic Speech Recognition and Understanding (ASRU), Keystone, CO, 1999.
104. A. L. Gorin, J. H. Wright, G. Riccardi, A. Abella, and T. Alonso, "Semantic information processing of spoken language," presented at the ATR Workshop MultiLingual Speech Communication, Kyoto, Japan, 2000.
105. A. L. Gorin, J. H. Wright, and G. Riccardi, "How may I help you?," Speech Commun., vol. 23, pp. 113-127, 1997.
106. (2001) SMIL, Synchronized Multimedia Integration Language (SMIL 2.0) Specification. W3C Proposed Recommendation. Online. Available: http://www.w3.org/TR/smil20/
107. PlayMail Online, Available: http://playmail.research.att.com
108. Real Networks Online, Available: http://www.real.com
109. Microsoft Online. Available: http://www.microsoft.com
110. A. C. Andres del Valle and J. Ostermann. "3D talking head customization by adapting a generic model to one uncalibrated picture," in Proc. ISCAS 2001, vol. 2, 2001, pp. 325-328.
111. M. V. Mani and J. Ostermann, "Cloning of MPEG-4 face models," presented at the Int. Workshop Very Low Bitrate Video Coding (VLBV 01), Athens, Greece, 2001.
112. J. Ostermann, J. Rurainsky, and R. Civanlar, "Real-time streaming for the animation of talking faces in multiuser environments," presented at the ISCAS 2002, Phoeniz, AZ, 2002.