Photo-Real Lips Synthesis with Trajectory-Guided Sample Selection

7th ISCA Workshop on Speech Synthesis, SSW 2010

Volumn , Issue , 2010, Pages 217-222

  Wang, Lijuan ^a   Qian, Xiaojun ^b   Han, Wei ^c   Soong, Frank K ^a  

^a MICROSOFT RESEARCH ASIA   (China)

^b CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

^c SHANGHAI JIAO TONG UNIVERSITY   (China)

Author keywords

photo real; talking head; trajectory guided; visual speech synthesis

Indexed keywords

IMAGE PROCESSING; MAXIMUM LIKELIHOOD; SPEECH COMMUNICATION; SPEECH SYNTHESIS; TRAJECTORIES;

AUDIO-VISUAL DATABASE; HIDDEN-MARKOV MODELS; LIP MOVEMENTS; PHOTO-REAL; REAL IMAGES; SAMPLES SELECTION; SPEECH SIGNALS; TALKING HEADS; TRAJECTORY-GUIDED; VISUAL SPEECH SYNTHESIS;

HIDDEN MARKOV MODELS;

EID: 84996687897     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. E. Cosatto and H.P. Graf, “Photo-realistic talking heads from image samples”, IEEE Trans. Multimedia, 2000, vol. 2, no. 3, pp. 152-163.
2. C. Bregler, M. Covell, M. Slaney, “Video Rewrite: Driving Visual Speech with Audio,” In Proc. ACM SIGGRAPH 97, Los Angeles, CA, 1997, pp. 353-360.
3. F. Huang, E. Cosatto, H.P. Graf, “Triphone based unit selection for concatenative visual speech synthesis,” Proc. ICASSP 2002. Vol. 2, 2002 pp.2037-2040.
4. T. Ezzat, G. Geiger, and T. Poggio, “Trainable video realistic speech animation,” Proc. ACM SIGGRAPH2002, San Antonio, Texas, 2002, pp. 388-398.
5. W. Mattheyses, L. Latacz, W. Verhelst, H. Sahii, “Multimodal Unit Selection for 2D Audiovisual Text-to-Speech Synthesis,” Proc. MLMI 2008, The Netherlands, 2008, pp. 125-136.
6. K. Liu, J. Ostermann, “Realistic Facial Animation System for Interactive Services,” Proc. Interspeech2008, Brisbane, Australia, Sept. 2008, pp.2330-2333.
7. K. Tokuda, H. Zen, etc., “The HMM-based speech synthesis system (HTS),” http://hts.ics.nitech.ac.jp/.
8. S. Sako, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “HMM-based Text-To-Audio-Visual Speech Synthesis,” ICSLP 2000.
9. L. Xie, Z.Q. Liu, “Speech Animation Using Coupled Hidden Markov Models,” Pro. ICPR'06, August 2006, pp. 1128-1131.
10. Z.H. Ling and R.H. Wang, “HMM-based unit selection using frame sized speech segments,” Proc. Interspeech 2006, Sep. 2006, pp. 2034-2037.
11. Z.J. Yan, Y. Qian, F. Soong, “Rich-Context Unit Selection (RUS) Approach to High Quality TTS,” Proc. ICASSP 2010, March 2010, pp.4798-4801.
12. B. Theobald, S. Fagel, G. Bailly, and F. Elisei, “LIPS2008: Visual Speech Synthesis Challenge,” Proc. Interspeech2008, Brisbane, Australia, Sept. 2008, pp.2310-2313.
13. T. Chen, “Audiovisual speech processing,” Signal Processing Magazine, IEEE Vol.18, Issue 1, Jan. 2001, pp.9-21.
14. S.A. King, R.E. Parent,”Creating speech-synchronized animation,” Visualization and Computer Graphics, IEEE Transactions on Vol. 11, Issue 3, May-June 2005, pp.341-352.
15. E. Cosatto and H.P. Graf, “Sample-based synthesis of photo-realistic talking heads,” Proc. IEEE Computer Animation, pp. 103-110, 1998.
16. T. Ezzat, T. Poggio, “Miketalk: A talking facial display based on morphing visemes,” Proc. Computer Animation, June 1998, pp. 96-102.
17. B.J. Theobald, J.A. Bangham, I.A. Matthews, G.C. Cawley, “Near videorealistic synthetic talking faces: implementation and evaluation,” Speech Communication 2004, Vol. 44, pp.127-140.
18. K. Liu, A.Weissenfeld, J. Ostermann, “Parameterization of Mouth Images by LLE and PCA for Image-Based Facial Animation,” Proc. ICASSP 2006, Vol. V, May 2006, pp.461-464.
19. Q. Wang, W. Zhang, X. Tang, H.Y. Shum, “Real-time Bayesian 3-d pose tracking,” IEEE Transactions on Circuits and Systems for Video Technology 16(12) (2006), pp.1533-1541.
20. S. Nakamura, “Statistical Multimodal Integration for AudioVisual Speech Processing,” IEEE Transactions on Neural Networks, Vol.13, No.4, July 2002, pp.854-866.
21. S. Lucey, T. Chen, S. Sridharan, and V. Chandran, “Integration Strategies for Audio-Visual Speech Processing: Applied to Text-Dependent Speaker Recognition,” IEEE Transactions on Multimedia, Vol.7, No.3, June 2005, pp.495-506.
22. K. Tokuda, T. Masuko, T. kobayashi and S. Imai, “Speech synthesis using HMMs with dynamic features,” Proc. ICASSP 1996, Vol. I, pp. 389-392.
23. K. Tokuda, T. Masuko, N. Miyazaki, T. Kobayashi, “Hidden Markov models based on Multi-space probability distribution for pitch pattern modeling,” Proc. ICASSP 1999, Vol. I, pp.229-232.
24. T. Toda, A. Black, K. Tokuda, “Spectral Conversion Based on Maximum Likelihood Estimation Considering Global Variance of Converted Parameter,” Proc. ICASSP 2005, Vol. I, pp. 9-12.
25. P. Perez, M. Gangnet, A. Blake, “Poisson Image Editing,” Proc. ACM SIGGRAPH2003, pp.313-318.
26. X. Huang, A. Acero, H. Hon, Y. Ju, J. Liu, S. Merdith, and M. Plumpe, “Recent improvements on Microsoft's trainable text-to-speech system - Whistler,” Proc. ICASSP 1997, pp. 959-962.
27. R.E. Donovan, and E.M. Eide, “The IBM trainable speech synthesis system,” Proc. ICSLP 1998, pp.1703-1706.
28. T. Hirai, and S. Tenpaku, “Using 5 ms segments in concatenative speech synthesis,” Proc. of 5th ISCA Speech Synthesis Workshop, Pittsburgh, PA, USA, 2004, pp. 37-42.
29. A. Hunt and A. Black, "Unit selection in a concatenative speech synthesis system using a large speech database," Proc. ICASSP 1996, pp. 373-376.
30. J.P. Lewis, "Fast Normalized Cross-Correlation," Industrial Light & Magic.
31. P. Perez, M. Gangnet, A. Blake, “Poisson Image Editing,” in ACM Transactions on Graphics (SIGGRAPH'03), 22(3), pp.313-318.