Realistic visual Speech synthesis based on hybrid concatenation method

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 3, 2009, Pages 469-477

  Tao, Jianhua ^a   Xin, Le ^a   Yin, Panrong ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Fused hidden markov model (HMM); Inversion; Speech driven facial animation; Unit concatenation; Visual speech synthesis

Indexed keywords

A-FRAMES; COMPUTING EFFICIENCY; FACIAL ANIMATION; FACIAL EXPRESSIONS; FUSED HIDDEN MARKOV MODEL (HMM); GAUSSIAN MIXTURE MODELS; HIGH QUALITY; HYBRID CONCATENATION; INVERSION; LOOSE SYNCHRONIZATIONS; MAPPING METHOD; REAL-TIME APPLICATION; RUNNING SPEED; SECOND LAYER; TIGHTLY-COUPLED; TWO LAYERS; UNIT CONCATENATION; UNIT SELECTION; VISUAL SPEECH SYNTHESIS; VITERBI SEARCH;

ANIMATION; MAPPING; OBJECT RECOGNITION; SPEECH SYNTHESIS; SYNTHESIS (CHEMICAL);

HIDDEN MARKOV MODELS;

EID: 70350437421     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2011538     Document Type: Article

References (35)

1. L. Xin, J. H. Tao, and T. N. Tan, "Dynamic audio-visual mapping using fused hidden Markov model inversion method, " in Proc. ICIP, San Antonio, TX, 2007, pp. 293-296.
2. J. H. Tao and P. R. Yin, "Speech driven face animation based on dynamic concatenation model, " J. Inf. Computat. Sci., vol. 4, no. 1, pp. 271-280, 2007.
3. P. R. Yin, L. Y. Zhao, L. X. Huang, and J. H. Tao, "Expressive face animation synthesis based on dynamic mapping method, " in Affective Computing and Intelligent Interaction, ser. Lecture Notes in Computer Science, A. Paiva, R. Prada, and R. W. Picard, Eds. New York: Springer-Verlag, 2007, pp. 1-11.
4. H. Pan, S. Levinson, T. S. Huang, and Z. P. Liang, "A fused hidden Markov model with application to bimodal speech processing, " IEEE Trans. Signal Process., vol. 52, no. 3, pp. 573-581, Mar. 2004.
5. Z. H. Zeng, J. L. Tu, B. Pianfetti, M. Liu, T. Zhang, Z. Q. Zhang, T. S. Huang, and S. Levinsion, "Audio-visual affect recognition through multi-stream fusedHMMfor HCI, " in Proc. CVPR, 2005, pp. 967-972.
6. E. Cosatto, G. Potamianos, and H. P. Graf, "Audio-visual unit selection for the synthesis of photo-realistic talking-heads, " in Proc. IEEE Int. Conf. Multimedia Expo (ICME), 2000, pp. 619-622.
7. C. Bregler, M. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio, " in Proc. ACM SIGGRAPH, 1997, pp. 353-360.
8. L. Xie and Z. Liu, "Speech animation using coupled Hidden Markov Models, " in Proc. 18th Int. Conf. Pattern Recognition (ICPR), Hong Kong, China, 2006, pp. 1128-1131.
9. D. W. Massaro, J. Beskow, M. M. Cohen, C. L. Fry, and T. Rodriguez, "Picture my voice: Audio to visual speech synthesis using artificial neural networks, " in Proc. AVSP, Santa Cruz, CA, 1999, pp. 133-138.
10. T. Ezzat and T. Poggio, "MikeTalk: A talking facial display based on morphing visemes, " in Proc. Comput. Animation Conf., Philadelphia, PA, 1998, pp. 96-102.
11. E. Yamamoto, S. Nakamura, and K. Shikano, "Lip movement synthesis from speech based on Hidden Markov Models, " Speech Commun., vol. 26, pp. 105-115, 1998.
12. P. Y. Hong, Z. Wen, and T. S. Huang, "Real-time speech-driven face animation with expressions using neural networks, " IEEE Trans. Neural Netw., vol. 13, no. 4, pp. 916-927, 2002.
13. M. Brand, "Voice puppetry, " in Proc. SIGGRAPH, 1999, pp. 21-28.
14. P. R. Yin and J. H. Tao, "Dynamic mapping method based speech driven face animation system, " in Affective Computing and Intelligent Interaction, ser. Lecture Notes in Computer Science, J. Tao, T. Tie, and R. W. Picard, Eds. New York: Springer-Verlag, 2005.
15. M. Tekalp and J. Ostermann, "Face and 2-D mesh animation in MPEG-4, " Signal Process.: Image Commun., vol. 15, pp. 387-421, 2000.
16. J. Q.Wang, K. H.Wong, P. A. Pheng, H. M. Meng, and T. T.Wong, "A real-time Cantonese text-to-audiovisual speech synthesizer, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. ICASSP, 2004, pp. 653-659.
17. A. Verma, L. V. Subramaniam, N. Rajput, C. Neti, and T. A. Faruquie, "Animating expressive faces across languages, " IEEE Trans. Multimedia, vol. 6, no. 6, pp. 791-800, Dec. 2004.
18. J. H. Tao and T. H. Tan, "Emotional Chinese Talking Head System, " in Proc. ACM6th Int. Conf. Multimodal Interfaces (ICMI), State College, PA, 2004, pp. 273-280.
19. R. Gutierrez-Osuna, P. K. Kakumanu, A. Esposito, O. N. Garcia, A. Bojorquez, J. L. Castillo, and I. Rudomin, "Speech-driven facial animation with realistic dynamics, " IEEE Trans. Multimedia, vol. 7, no. 1, pp. 33-42, Feb. 2005.
20. Y. Huang, S. Lin, X. Ding, B. Guo, and H. Shum, "Real-time Lip synchronization based on Hidden Markov Models, " in Proc. ACCV, 2002, pp. 176-181.
21. R. Rao, T. Chen, and R. M. Mersereau, "Audio-to-visual conversion for multimedia communication, " IEEE Trans. Ind. Electron., vol. 45, no. 1, pp. 15-22, Feb. 1998.
22. S. L. Fu, R. Gutierrez-Osuna, A. Esposito, P. K. Kakumanu, and O. N. Garcia, "Audio/visual mapping with cross-modal Hidden Markov models, " IEEE Trans. Multimedia, vol. 7, no. 3, pp. 243-252, Apr. 2005.
23. Y. Li and H. Y. Shum, "Learning dynamic audio-visual mapping with input-output Hidden Markov Models, " IEEE Trans. Multimedia, vol. 8, no. 3, pp. 542-549, Apr. 2006.
24. M. Brand and N. Oliver, "Coupled hidden markov models for complex action recognition, " in Proc. Comput. Vis. Pattern Recognition, 1997, pp. 201-206.
25. K. S. Arun, T. S. Huang, and S. D. Blostein, "Least-square fitting of two 3-D point sets, " IEEE Trans. Pattern Anal. Mach. Intell., vol. PAMI-9, no. 5, pp. 698-700, May 1987.
26. Y. Li, F. Yu, Y. Q. Xu, E. Chang, and H. Y. Shum, "Speech-driven cartoon animation with emotions, " in Proc. 9th ACM Int. Conf. Multimedia, 2001, pp. 365-371.
27. R. Rao and T. Chen, "Exploiting audio-visual correlation in coding of talking head sequences, " in Proc. Picture Coding Symp., Melbourne, Australia, Mar. 1996, pp. 653-658.
28. D. Berndt and J. Clifford, "Using dynamic timewarping to find patterns in time series, " in Proc. KDD Workshop, 1994, pp. 359-370.
29. C. Kim, W. Kim, B. Park, C. Hong, and M. Jeong, "Smoothing techniques via the bezier curve, " Commun. Statist.-Theory Methods, vol. 28, no. 7, pp. 1577-1597, 1999.
30. Y. Chen, M. Chu, E. Chang, J. Liu, and R. Liu, "Voice conversion with smoothedGMM and MAP adaptation, " in Proc. Eurospeech, 2003, pp. 2413-2416.
31. J. H. Tao, Y. G. Kang, and A. J. Li, "Prosody conversion from neutral speech to emotional speech, " IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 4, pp. 1145-1154, Jul. 2006.
32. R. Tato, R. Santos, R. Kompe, and J. M. Pardo, "Emotional space improves emotion recognition, " in Proc. ICSLP, Denver, CO, 2002, pp. 2029-2032.
33. N. Campbell, "Perception of affect in speech-Towards an automatic processing of paralinguistic information in spoken conversation, " in Proc. ICSLP, Jeju, Korea, 2004, pp. 881-884.
34. G. McLachlan and T. Krishnan, "The EM algorithm and extensions, " in Wiley Series in Probability and Statistics. New York:Wiley, 1997.
35. L. Xin, Q. Wang, J. H. Tao, X. Tang, T. Tan, and H. Shum, "Automatic 3D face modeling from video, " in Proc. ICCV, 2005, vol. 2, pp. 1193-1199.