Speech-to-video synthesis using MPEG-4 compliant visual features

IEEE Transactions on Circuits and Systems for Video Technology

Volumn 14, Issue 5, 2004, Pages 682-692

  Aleksic, Petar S ^a   Katsaggelos, Aggelos K ^a  

^a Northwestern University   (United States)

Author keywords

Audio visual speech recognition; Correlation hidden Markov models (CHMMs); Facial animation parameters (FAPs); Speech to video synthesis

Indexed keywords

ALGORITHMS; CORRELATION METHODS; DATA REDUCTION; MARKOV PROCESSES; SIGNAL TO NOISE RATIO; SPEECH RECOGNITION; SPEECH SYNTHESIS; SYNCHRONIZATION; TOPOLOGY;

AUDIO-VISUAL SPEECH RECOGNITION; CORRELATION HIDDEN MARKOV MODELS (CHMM); FACIAL ANIMATION PARAMETERS (FAP); SPEECH-TO-VIDEO SYNTHESIS;

IMAGE COMPRESSION;

EID: 2542499812     PISSN: 10518215     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSVT.2004.826760     Document Type: Article

References (32)

1. R. Lippman, "Speech recognition by machines and humans," Speech Commun., vol. 22, no. 1, pp. 1-15, July 1997.
2. Y. Gong, "Speech recognition in noisy environments: A survey," Speech Commun., vol. 16, pp. 261-291, 1995.
3. L. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993.
4. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, pp. 257-286, Feb. 1989.
5. F. Lavagetto, "Converting speech into lip movements: A multimedia telephone for hard of hearing people," IEEE Trans. Rehab. Eng., vol. 3, pp. 1-14, Mar. 1995.
6. A. Simons and S. Cox, "Generation of mouth shapes for a synthetic talking head," Proc. Inst. Acoust., vol. 12, pp. 475-482, 1990.
7. T. Chen and R. R. Rao, "Audio-visual integration in multimedia communication," Proc. IEEE, vol. 86, pp. 837-852, May 1998.
8. C. Bregler, M. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio," in Proc. ACM SIGGRAPH, 1997, pp. 353-360.
9. K. Choi and J.-N. Hwang, "Baum-welch hidden Markov model inversion for reliable audio-to-video conversion," in Proc. IEEE 3rd Workshop Multimedia Signal Processing, 1999, pp. 175-180.
10. S. Moon and J.-N. Hwang, "Robust speech recognition based on joint model and feature space optimization of hidden Markov models," IEEE Trans. Neural Networks, vol. 8, pp. 194-204, Mar. 1997.
11. S. Nakamura, "Fusion of audio-visual information for integrated speech processing," in Proc. Third Int. Conf. Audio- and Video-Based Biometric Person Authentication, Halmstad, Sweden, 2001, pp. 127-143.
12. J. J. Williams and A. K. Katsaggelos, "An HMM-based speech-to-video synthesizer," IEEE Trans. Neural Networks, vol. 3, pp. 900-915, July 2002.
13. J. J. Williams, A. K. Katsaggelos, and D. C. Garstecki, "Subjective analysis of an HMM-based visual speech synthesizer," in Proc. SPIE Conf. Human Vision and Electronic Imaging, vol. 4299, San Jose, CA, Jan. 2001, pp. 544-555.
14. Text for ISO/IEC FDIS 14496-2 Visual, ISO/IEC JTC1/SC29/WG11 N2502, Nov. 1998.
15. Text for ISO/IEC FDIS 14496-1 Systems, ISO/IEC JTC1/SC29AVG11 N2502, Nov. 1998.
16. G. A. Abrantes, FACE-Facial Animation System, Version 3.3.1: Instituto Superior Tecnico, (c), 1997-98.
17. F. Lavagetto and R. Pockaj, "An efficient use of MPEG-4 FAP interpolation for facial animation at 70 bits/frame," IEEE Trans. Circuits Syst. Video Technol., vol. 11, pp. 1085-1097, Oct. 2001.
18. P. S. Aleksic, J. J. Williams, Z. Wu, and A. K. Katsaggelos, "Audio-visual speech recognition using MPEG-4 compliant visual features," EURASIP J. Appl. Signal Processing, pp. 1213-1227, 2002.
19. _, "Audio-visual continuous speech recognition using MPEG-4 compliant visual feature," in Proc. Int. Conf. Image Processing, Rochester, NY, Sept. 2002, pp. 960-963.
20. L. E. Bernstein, Lipreading Corpus V-VI: Disc 3. Washington, DC: Gallaudet University, 1991.
21. (1990) Carnegie Melon University Pronunciation Dictionary. [Online]. Available: http://www.speech.cs.cmu.edu/cgi-bin/cmudict
22. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New York: Wiley, 2001.
23. E. Petajan, "Automatic lipreading to enhance speech recognition," in Proc. IEEE Conf. Computer Vision and Pattern Recognition, San Francisco, CA, 1985, pp. 40-47.
24. D. G. Stork and M. E. Hennecke, Eds., Speechreading by Man and Machine. New York: Springer-Verlag, 1996.
25. C. Neti, G. Potamianos, J. Luettin, I. Matthews, H. Glotin, D. Vergyri, J. Sison, A. Mashari, and J. Zhou, Workshop Audio-Visual Speech Recognition, Final Report. Baltimore, MD, Oct. 2000.
26. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audio-visual speech," Proc. IEEE, vol. 91, pp. 1306-1326, Sept. 2003.
27. G. Potamianos, J. Luettin, and C. Neti, "Hierarchical discriminant features for audio-visual LVCSR," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, vol. 1, 2001, pp. 165-168.
28. C. Bregler and Y. Conig, "Eigenlips' for robust speech recognition," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, Adelaide, Australia, 1994, pp. 669-672.
29. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, pp. 141-151, Mar. 2000.
30. H. Glotin, D. Vergyri, C. Neti, G. Potamianos, and J. Luettin, "Weighting schemes for audio-visual fusion in speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, vol. 1, 2001, pp. 165-168.
31. S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland, The HTK Book. Cambridge, U.K.: Entropic Ltd., 2002.
32. P. S. Aleksic and A. K. Katsaggelos, "Speech-to-video synthesis using facial animation parameters," in Proc. Int. Conf. Image Processing, Barcelona, Spain, Sept. 2003, pp. 1-4.