Realistic mouth-synching for speech-driven talking face using articulatory modelling

IEEE Transactions on Multimedia

Volumn 9, Issue 3, 2007, Pages 500-510

  Xie, Lei ^b   Liu, Zhi Qiang ^a,b  

^a IEEE   (United States)

^b CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Articulatory model; Baum Welch DBN inversion (DBNI); Dynamic Bayesian networks (DBNs); Facial animation; Mouth synching; Talking face

Indexed keywords

ARTICULATORY MODELS; BAUM-WELCH DBN INVERSION (DBNI); DYNAMIC BAYESIAN NETWORKS (DBNS); FACIAL ANIMATION; TALKING FACES;

ALGORITHMS; BAYESIAN NETWORKS; COMPUTER SIMULATION; FACE RECOGNITION; HIDDEN MARKOV MODELS; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; SPEECH PROCESSING;

ANIMATION;

EID: 33947583073     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2006.888009     Document Type: Article

References (34)

1. J. Ostermann and A. Weissenfeld, "Talking faces-technologies and applications," in Proc. of ICPR'04, Aug. 2004, vol. 3, pp. 826-833.
2. E. Cosatto, J. Ostermann, H. P. Graf, and J. Schroeter, "Lifelike talking faces for interactive services," Proc. IEEE, vol. 91, no. 9, pp. 1406-1428, Sep. 2003.
3. F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin, "Synthesizing realistic facial expressions from photographs," in Proc. ACM SIGGRAPH'98, 1998, vol. 3, pp. 75-84.
4. K. Kaehler, J. Haber, H. Yamauchi, and HP Seidel, "Head shop: Generating animated head models with anatomical structure," in Proc. ACM SIGGRAPH'02, 2002, pp. 55-63.
5. C. Bregler, M. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio," in Proc. ACM SIGGRAPH'97, 1997.
6. T. Ezzat, G. Geiger, and T. Poggio, "Trainable videorealistic speech animation," in Proc. ACM SIGGRAPH, 2002, pp. 388-397.
7. E. Cosatto and H. Graf, "Sample-based synthesis of photo-realistic talking heads," in Proc. IEEE Computer Animation, 1998, pp. 103-110.
8. _. "Photo-realistic talking heads from image samples," IEEE Trans. Multimedia, vol. 2, pp. 152-163, 2000.
9. E. Eide, A. Aaron, R. Bakis, P. Cohen, R. Donovan, W. Hamza, T. Mathes, M. Picheny, M. Polkosky, M. Smith, and M. Viswanathan, "Recent improvements to the IBM trainable speech synthesis system," in Proc. ICASSP'03, 2003, vol. 1, pp. 708-711.
10. M. M. Cohen and D. W. Massaro, "Modeling coarticulation in synthetic visual speech," in Models and Techniques in Computer Animation, M. Magnenat-Thalmann and D. Thalmann, Eds. Tokyo, Japan: Springer-Verlag, 1993, pp. 139-156.
11. P. Hong, Z. Wen, and T. S. Huang, "Real-time speech-driven face animation with expressions using neural networks," IEEE Trans. Neural Networks, vol. 13, no. 4, pp. 916-927, 2002.
12. D. W. Massaro, J. Beskow, M. M. Cohen, C. L. Fry, and T. Rodriguez, "Picture my voice: Audio to visual speech synthesis using artificial neural networks," in Proc. AVSP'99, 1999, pp. 133-138.
13. L. R. Rabiner and B. H. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993.
14. E. Yamamoto, S. Nakamura, and K. Shikano, "Lip movement synthesis from speech based on hidden Markov models," Speech Commun., vol. 26, no. 1-2, pp. 105-115, 1998.
15. M. Brand, "Voice puppetry," in Proc. ACM SIGGRAPH'99, 1999, pp. 21-28.
16. T. Chen, "Audiovisual speech processing: Lip reading and lip synchronization," IEEE Signal Process. Mag., vol. 18, no. 1. pp. 9-21, 2001.
17. Y. Cao, P. Faloutsos, E. Kohler, and F. Pighin, "Real-time speech motion synthesis from recorded motions," in Eurographics/ACM SIGGRAPH Symp. Computer Animation, 2004, pp. 347-355.
18. K. Choi and J. N. Hwang, "Baum-Weich hidden Markov model inversion for reliable audio-to-visual conversion," in Proc. IEEE 3rd Workshop Multimedia Signal Processing, 1999, pp. 175-180.
19. S. Y. Moon and J. N. Hwang, "Noisy speech recognition using robust inversion of hidden Markov models," in Proc. ICASSP'95, 1995, pp. 145-148.
20. K. Choi, Y. Luo, and J. Hwang, "Hidden Markov model inversion for audio-to-visual conversion in an MPEG-4 facial animation system," J. VLSI Signal Process., no. 29, pp. 51-61, 2001.
21. S. Fu, R. Gutierrez-Osuna, A. Esposito, K. P. Kakumanu, and O. N. Garcia, "Audio/visual mapping with cross-modal hidden Markov models," IEEE Trans. Multimedia, vol. 7, pp. 243-251, 2005.
22. J. Goldsmith, Autosegmental and Metrical Phonology, W. Hardcastle and N. Hewlett, Eds. New York: Basil Blackwell, 1990.
23. L. Xie, "Research on Key Issues of Audio Visual Speech Recognition," Ph.D. dissertation, Northwestern Polytechnical Univ., Xian, China, 2004.
24. K. Kirchhoff, "Combining articulatory and acoustic information for speech recognition in noisy and reverberant environments," in Proc. ICSLP'98, 1998, pp. 891-894.
25. F. V. Jensen, Bayesian Networks and Decision Graphs. New York: Springer-Verlag, 2001.
26. G. G. Zweig, "Bayesian network structures and inference techniques for automatic speech recognition," Comput. Speech Lang., vol. 17, pp. 173-193, 2003.
27. M. Richardson, J. Bilmes, and C. Diorio, "Hidden-articulator Markov models for speech recognition," Speech Commun., vol. 41, pp. 511-529, 2003.
28. J. A. Bilmes, G. Zweig, T. Richardson, K. Filali, K. Livescu, P. Xu, K. Jackson, Y. Brandman, E. Sandness, E. Holtz, J. Torres, and B. Byrne, "Discriminatively structured graphical models for speech recognition," in Tech. Rep. JHU 2001 Summer Workshop, 2001.
29. L. E. Baum and G. R. Sell, "Growth functions for transformations on manifolds," Pacific J. Math., vol. 27, no. 2, pp. 211-227, 1968.
30. A. Dempster, A. N. Laird, and D. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc. B, vol. 39, pp. 89-111, 1977.
31. K. Murphy, "Dynamic Bayesian Networks: Representation, Inference and Learning," Ph.D. dissertation, Univ. California, Berkeley, 2002.
32. L. Xie and Z. Ye, The JEWEL Audio-Visual Dataset for Facial Animation 2005, Tech. Rep. RCMT 05-11.
33. S. Young, G. Evermann, D. Kershaw, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book Eng. Dept., Cambridge Univ., Cambridge, U.K., 2002 [Online]. Available: http://htk.eng.cam.ac.uk/, 3.2
34. P. Pèrez, M. Gangnet, and A. Blake, "Poisson image editing," ACM Trans. Graphics (SIGGRAPH'03), vol. 22, no. 3, pp. 313-318, 2003.