Emotional audio-visual speech synthesis based on PAD

IEEE Transactions on Audio, Speech and Language Processing

Volumn 19, Issue 3, 2011, Pages 570-582

  Jia, Jia ^a   Zhang, Shen ^a   Meng, Fanbo ^a   Wang, Yongxin ^a   Cai, Lianhong ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

and DominanceSubmissiveness (PAD); ArousalNonarousal; Audio visual speech; boosting Gaussian mixture model (GMM); emotion; facial expression; Pleasure Displeasure

Indexed keywords

AND DOMINANCESUBMISSIVENESS (PAD); AROUSALNONAROUSAL; AUDIO-VISUAL SPEECH; EMOTION; FACIAL EXPRESSIONS; GAUSSIAN MIXTURE MODEL; PLEASURE-DISPLEASURE;

AUDIO ACOUSTICS; GAUSSIAN DISTRIBUTION; SPEECH COMMUNICATION; SPEECH SYNTHESIS;

SPEECH ANALYSIS;

EID: 78650033338     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2010.2052246     Document Type: Article

References (41)

1. E. Cosatto, J. Ostermann, H. P. Graf, and J. Schroeter, "Lifelike talking faces for interactive services authors," Proc. IEEE, Special Iss. Human-Computer Multimodal Interface, vol. 91, no. 9, pp. 1406-1429, Sep. 2003.
2. Z. Y. Wu, S. Zhang, L. H. Cai, and H. M. Meng, "Real-time synthesis of Chinese visual speech and facial expressions using MPEG-4 FAP features in a three-dimensional avatar," in Proc. Int. Conf. Spoken Lang. Process., 2006, pp. 1802-1805.
3. R. W. Picard, "Affective Computing," MIT Media Lab., Perceptual Comput. Section, Mass. Inst. Technol., Cambridge, MA, Tech. Rep., 1995.
4. C. Darwin, The Expression of the Emotions in Man and Animals. London, U.K.: John Murray, 1872.
5. P. Ekman, "Facial expression and emotion," Amer. Psychol., vol. 48, no. 4, pp. 384-392, Apr. 1993.
6. J. A. Russell, J. Bachorowski, and J. Fernández-Dols, "Facial and vocal expressions of emotion," Annu. Rev. Psychol., vol. 2002, pp. 329-349, 2003.
7. M. Schröder, "Emotional speech synthesis: A review," in Proc. Eurospeech, Aalborg, Denmark, 2001, vol. 1, pp. 561-564.
8. E. Bevacqua and C. Pelachaud, "Expressive audio-visual speech," Comput. Animat. Virtual Worlds, vol. 15, no. 3-4, pp. 297-304, 2004.
9. M. Fabri and D. J. Moore, "The use of emotionally expressive avatars in collaborative virtual environments," in Proc. Symp. Empathic Interact. With Synth. Charact., 2005.
10. H. Tang, Y. Fu, J. Tu, M. Hasegawa-Johnson, and T. S. Huang, "Humanoid audiovisual avatar with emotive text-to-speech synthesis," IEEE Trans. Multimedia, vol. 10, pp. 969-981, Oct. 2008.
11. N. Audibert, D. Vincent, V. Auberg, and O. Rosec, "Expressive speech synthesis: Evaluation of a voice quality centered coder on the different acoustic dimensions," in Proc. Speech Prosody, 2006.
12. M. Bulut, S. Lee, and S. Narayanan, "A statistical approach for modeling prosody features using pos tags for emotional speech synthesis," in Proc. Int. Conf. Acoust. Speech, Signal, Process., Honolulu, HI, Apr. 2007, pp. 1237-1240.
13. J. Cabral and L. Oliveira, "Emovoice: A system to generate emotion in speech," in Proc. Interspeech, Pittsburgh, PA, 2006.
14. N. Campbell, "Synthesis units for conversational speech using phrasal segments," in Proc. Autumn Meet. Acoust. Soc. Jpn., 2004.
15. J. Tao, Y. Kang, and A. Li, "Prosody conversion from neutral speech to emotional speech," IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 4, pp. 1145-1154, Jul. 2006.
16. H. Kawanami, Y. Iwami, T. Toda, H. Saruwatari, and K. Shikamo, "GMM-based voice conversion applied to emotional speech synthesis," IEEE Trans. Speech Audio Process., vol. 7, no. 6, pp. 697-708, Nov. 1999.
17. Y. Du and X. Lin, "Emotional facial expression model building," Pattern Recognition Lett., vol. 24, no. 16, pp. 2923-2934, 2003.
18. A. Raouzaiou, N. Tsapatsoulis, K. Karpouzis, and S. Kollias, "Parameterized facial expression synthesis based on MPEG-4," EURASIP J. Appl. Signal Process., vol. 2002, no. 10, pp. 1021-1038, 2002.
19. Z. Ruttkay, H. Noot, and P. Hagen, "Emotion disc and emotion squares: Tools to explore the facial expression space," Comput. Graphics Forum, vol. 22, no. 1, pp. 49-53, 2003.
20. I. Albrecht, M. Schröder, J. Haber, and H. P. Seidel, "Mixed feelings: expression of non-basic emotions in a muscle-based talking head," Virtual Reality, vol. 8, no. 4, pp. 201-212, 2005.
21. C. Pelachaud and M. Bilvi, "Computational model of believable conversational agents," in Communication in Multiagent Systems, M. P. Huget, Ed. New York: Springer-Verlag, 2003, vol. 2650, Lecture Notes in Computer Science, pp. 300-317.
22. C. Busso and S. Narayanan, "Interrelation between speech and facial gestures in emotional utterances: A single subject study," IEEE Trans. Speech, Audio,. Speech, Lang. Process., vol. 15, no. 8, pp. 2331-2347, Nov. 2007.
23. Z. Zeng, P. Maja, G. I. Roisman, and S. Thomas, "A survey of affect recognition methods: Audio, visual and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
24. A. Mehrabian, "Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament," Current Psychol.: Development., Learn., Personal., Soc., vol. 14, pp. 261-292, 1996.
25. W. Xiong, D. Cui, F. Meng, and L. Cai, "Analysis and conversion of emotional speech based on the prosodic features," in Proc. 8th Phon. Conf. China (PCC), 2008.
26. E. Moulines and F. Charpentier, "Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones," Speech Commun., vol. 9, no. 5-6, pp. 453-467, Dec. 1990.
27. X. Zhang, J. Xu, and L. Cai, "Prosodic boundary prediction based on maximum entropy model with error-driven modification," in Proc. ISCSLP, Singapore, 2006.
28. A. Kain and M. W. Macon, "Spectral voice conversions of text-tospeech synthesis," in Proc. Int. Conf. Acoust, Speech, Signal Process., 1998, pp. 285-288.
29. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting," J. Comput. Syst. Sci., vol. 55, pp. 119-139, 1997.
30. Y. Freund and R. E. Schapire, "A short introduction to boosting," J. Jpn. Soc. Artif. Intell., vol. 14, pp. 771-780, 1999.
31. International Standard, Information Technology Coding of Audio-Visual Objects. Part 2: Visual; Amendment 1: Visual Extensions., International Organization for Standardization Std, ISO/IEC 14496-2: 1999/Amd. 1: 2000(E).
32. I. S. Pand'zić and R. Forchheimer, MPEG-4 Facial Animation - The Standard, Implementations and Applications. New York: Wiley, 2002.
33. S. Zhang, Z.Wu, and L. Cai, "Region-based facial expression synthesis on a three-dimensional avatar," in Proc. China Conf. Human Comput. Interact., 2006.
34. M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with gabor wavelets," in Proc. 3rd IEEE Conf. Face Gesture Recogn., 1998, pp. 200-205.
35. X. Li, H. Zhou, S. Song, T. Ran, and X. Fu, "The reliability and validity of the chinese version of abbreviated pad emotion scales," in Proc. Int. Conf. Affective Comput. Intell. Interact., 2005.
36. F. Lavagetto and R. Pockaj, "An efficient use of MPEG-4 FAP interpolation for facial animation at 70 bits/frame," IEEE Trans. Circuits Syst. Video Technol., vol. 11, pp. 1085-1097, Nov. 2001.
37. Z. Wang, L. Cai, and H. Ai, "A dynamic viseme model for personalizing a talking head," in Proc. 6th Int. Conf. Signal Process., Beijing, China, Aug. 2002.
38. Z. Wu, L. Cai, and H. M. Meng, "Multi-level fusion of audio and visual features for speaker identification," in Proc. Int. Conf. Biometrics (ICB2006), 2006, pp. 493-499. (Pubitemid 43856410)
39. Z. Wu, L. Cai, and H. M. Meng, "DBN based audio-visual correlative model for audio-visual speech synthesis," in Proc. NCMMSC2005 (in Chinese), Beijing, China, 2005, pp. 334-337.
40. "MATLAB Function Reference," 1-D Data Interpolation MathWorks.
41. "Xface: MPEG-4 based open source toolkit for 3d facial animation," in Proc. AVI04, Working Conf. Adv. Visual Interfaces, Gallipoli, Italy, May 25-28, 2004.