Humanoid audio-visual avatar with emotive text-to-speech synthesis

IEEE Transactions on Multimedia

Volumn 10, Issue 6, 2008, Pages 969-981

  Tang, Hao ^a   Fu, Yun ^a   Tu, Jilin ^a   Hasegawa Johnson, Mark ^a   Huang, Thomas S ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

3 D face modeling and animation; Audio visual avatar; Emotive speech synthesis; Human computer interaction; Multimodal system; TTS

Indexed keywords

ANIMATION; FEATURE EXTRACTION; FLOW INTERACTIONS; HUMAN COMPUTER INTERACTION; KNOWLEDGE MANAGEMENT; SPEECH; SPEECH SYNTHESIS; THREE DIMENSIONAL; THREE DIMENSIONAL COMPUTER GRAPHICS; VIRTUAL REALITY;

3-D FACE MODELING AND ANIMATION; AUDIO-VISUAL AVATAR; EMOTIVE SPEECH SYNTHESIS; MULTIMODAL SYSTEM; TTS;

SPEECH COMMUNICATION;

EID: 54949115779     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2008.2001355     Document Type: Article

References (36)

1. A. Jaimes, N. Sebe, and D. Gatica-Perez, "Human-centered computing : A multimedia perspective," in ACM Conf. on Multimedia, 2006, pp. 855-864.
2. H. Tang, Y. Fu, J. Tu, T. S. Huang, and M. Hasegawa-Johnson, "EAVA: A 3D emotive audio-visual avatar," in 2008 IEEE Workshop on Applications of Computer Vision (WACV'08), Jan. 2008, Copper Mountain, CO.
3. Y. Fu and N. Zheng, "M-face: An appearance-based photorealistic model for multiple facial attributes rendering," IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 7, pp. 830-842, Jul. 2006.
4. Y. Fu, R. Li, T. S. Huang, and M. Danielsen, "Real-time multimodal human-avatar interaction," IEEE Trans. Circuits Syst. Video Technol. 2008, to appear.
5. Y. Fu, R. Li, T. S. Huang, and M. Danielsen, "Real-time humanoid avatar for multimodal human-machine interaction," in IEEE Conf. ICME'07, 2007, pp. 991-994.
6. P. Hong, Z. Wen, and T. S. Huang, "Real-time speech-driven face animation with expressions using neural networks," IEEE Trans. Neural Netw., vol. 13, no. 4, pp. 916-927, 2002.
7. P. Hong, Z. Wen, and T. S. Huang, "iFace: A 3D synthetic talkingn face," Int. J. Image Graph., vol. 1, no. 1, pp. 19-26, 2001.
8. Z. Wen and T. S. Huang, 3D Face Processing: Modeling, Analysis and Synthesis, 1st ed. Berlin, Germany: Springer, 2004.
9. M. Mancini, R. Bresin, and C. Pelachaud, "A virtual head driven by music expressivity," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 6, pp. 1833-1841, 2007.
10. V. Blanz and T. Vetter, "A morphable model for the synthesis of 3D faces," Proc. SIGGRAPH'99, pp. 187-194, 1999.
11. F. I. Parke, "Computer generated animation of faces," in Pioc. ACM Nat. Conf., 1972, pp. 451-457.
12. H. Ishii, C. Wisneski, J. Orbanes, B. Chun, and J. Paradiso, "PingPongPlus: Design of an athletic-tangible interface for computer-supported cooperative play," Proc. ACM SIGCHI'99, pp. 394-401, 1999.
13. I. Pandzic and R. Forchheimer, Eds., MPEG-4 Facial Animation, Chichester, U.K.: Wiley, 2002.
14. "DAZ3D," [Online]. Available: http://www.daz3d.com/
15. K.-H. Choi and J.-N. Hwang, "Constrained optimization for audio-to-visual conversion," IEEE Trans. Signal Process., vol. 52, no. 6, pp. 1783-1790, Jun. 2004.
16. J. Ma, R. Cole, B. Pellom, W. Ward, and B. Wise, "Accurate visible speech synthesis based on concatenating variable length motion capture data," IEEE Trans. Vis. Comput. Graph., vol. 12, no. 2, pp. 266-276, 2006.
17. A. Mehrabian, "Communication without words," Psychol. Today, vol. 2, pp. 53-56, 1968.
18. T. Dutoit, An Introduction to Text-to-Speech Synthesis. Dordrecht, The Netherlands: Kluwer, 1997.
19. M. Schröder, "Speech and Emotion Research: An Overview of Research Frameworks and a Dimensional Approach to Emotional Speech Synthesis," Ph.D. Thesis, Res. Rep., Institute of Phonetics, Saarland Univ., Saarsland, Germany, 2004, vol. 7 of Phonus.
20. M. Schröder, Can Emotions be Synthesized Without Controlling Voice Quality? Phonus 4, Res. Rep., Inst. Phonetics, Univ. Saarsland, Germany, pp. 37-55, 2004.
21. M. Schröder and M. Grice, "Expressing vocal effort in concatenative synthesis," in Proc. 15th Int. Conf. of Phonetic, Barcelona, Spain, 2003, pp. 2589-2592.
22. J. E. Cahn, "Generating Expression in Synthesized Speech," Master's thesis, MIT Media Lab, , 1989.
23. I. R. Murray, "Simulating Emotion in Synthetic Speech," Ph.D. thesis, Univ. Dundee, Dundee, U.K., 1989.
24. F. Burkhardt, "Simulation Emotionaler Sprechweise mit Sprachsyntheseverfahren," Ph.D. thesis, Tech. Univ. Berlin, Germany, 2000.
25. A. Iida, "Corpus-Based Speech Synthesis With Emotion," Ph.D. Thesis, Univ. Keio, Tokyo, Japan, 2002.
26. G. Hofer, "Emotional Speech Synthesis," Master thesis, Univ. Edinburgh, Edinburgh, U.K., 2004.
27. J. F. Pitrelli, R. Bakis, E. M. Eide, R. Fernandez, W. Hamza, and M. A. Picheny, "The IBM expressive text-to-speech synthesis system for american english," IEEE Trans. Audio, Speech Lang. Process, vol. 14, no. 4, pp. 1099-1108, 2006.
28. H. Tao and T. S. Huang, "Explanation-based facial motion tracking using a piecewise bezier volume deformation model," in IEEE Conf. CVPR'99, 1999, pp. 611-617.
29. P. Ekman and W. V. Friesen, The Facial Action Coding System. Palo Alto, CA: Psychological Press, 1977.
30. The Festival Project [Online]. Available: http://www.cstr.ed.ac.uk/ projects/festival/
31. TheMBROLA Project [Online]. Available: http://mambo.ucsc.edu/psl/mbrola/
32. J. M. Montero, J. Gutiérrez-Arriola, J. Colás, J. Macías-Guarasa, E. Enriquez, and J. M. Pardo, "Development of an emotional speech synthesiser in Spanish," in Proc. Eumspeech '99, Budapest, Hungary, 1999.
33. F. Burkhardt, "Emofilt: The simulation of emotional speech by prosody-transformation," in Proc. INTERSPEECH-2005, Lisbon, Portugal, 2005, pp. 509-512.
34. Y. Cao, P. Faloutsos, and F. Pighin, "Expressive speech-driven facial animation," ACM Trans. Graph., vol. 24, no. 4, 2005.
35. S. Kshirsagar, T. Molet, and N. Magnenat-Thalmann, "Principal components of expressive speech animation," in Proc. Int. Conf. on Computer Graphics, 2001, pp. 38-44.
36. P. Ekman, T. S. Huang, T. Sejnowski, and J. Hager, in Final Report to NSF of the Planning Workshop on Facial Expression Understanding, San Francisco, CA, 1993, Available from HIL-0984, UCSF.