Audio-visual speech recognition using MPEG-4 compliant visual features

Eurasip Journal on Applied Signal Processing

Volumn 2002, Issue 11, 2002, Pages 1213-1227

  Aleksic, Petar S ^a   Williams, Jay J ^a   Wu, Zhilin ^a   Katsaggelos, Aggelos K ^a  

^a Northwestern University   (United States)

Author keywords

Audio visual speech recognition; Facial animation parameters; Snake

Indexed keywords

ACOUSTIC SIGNAL PROCESSING; ANIMATION; FEATURE EXTRACTION; GESTURE RECOGNITION; PRINCIPAL COMPONENT ANALYSIS; SIGNAL TO NOISE RATIO; VIDEO SIGNAL PROCESSING; WHITE ACOUSTIC NOISE;

AUDIO-VISUAL SIGNAL PROCESSING; AUTOMATIC SPEECH RECOGNITION (ASR);

SPEECH RECOGNITION;

EID: 0036874915     PISSN: 11108657     EISSN: None     Source Type: Journal    
DOI: 10.1155/S1110865702206162     Document Type: Article

References (37)

1. Q. Summerfield, "Lipreading and audio-visual speech perception," Philos. Trans. Roy. Soc. London Ser. B, vol. 335, pp. 71-78, 1992.
2. K. W. Grant and L. D. Braida, "Evaluating the articulation index for auditory visual input," Journal of the Acoustical Society of America, vol. 89, no. 6, pp. 2952-2960, 1991.
3. J. J. Williams, Speech-to-video conversion for individuals with impaired hearing, Ph.D. thesis, Northwestern University, Evanston, Ill, USA, June 2000.
4. J. J. Williams and A. K. Katsaggelos, "An HMM-based speech-to-video synthesizer," IEEE Trans. on Neural Networks, vol. 13, no. 4, pp. 900-915, 2002, Special Issue on Intelligent Multi-media.
5. Q. Summerfield, "Some preliminaries to a comprehensive account of audio-visual speech perception," in Hearing by Eye: The Psychology of Lip-Reading, B. Dodd and R. Campbell, Eds., pp. 97-113, Lawrence Erlbaum Associates, Hillside, Minn, USA, 1987.
6. R. Lippman, "Speech recognition by machines and humans," Speech Communication, vol. 22, no. 1, pp. 1-15, 1997.
7. Y. Gong, "Speech recognition in noisy environments: A survey," Speech Communication, vol. 16, no. 3, pp. 261-291, 1995.
8. E. Petajan, "Automatic lipreading to enhance speech recognition," in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 40-47, San Francisco, Calif, USA, 1985.
9. E. Petajan, B. Bischoff, D. Bodoff, and N. M. Brooke, "An improved automatic lipreading system to enhance speech recognition," in CHI-88, pp. 19-25, ACM, Washington, DC, USA, 1988.
10. A. J. Goldschen, O. N. Garcia, and E. Petajan, "Continuous optical automatic speech recognition by lipreading," in Proc. IEEE 28th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, Calif, USA, October 1994.
11. D. G. Stork and M. E. Hennecke, Eds., Speechreading by Man and Machine, Springer-Verlag, New York, NY, USA, 1996.
12. E. Petajan and H. P. Graf, "Robust face feature analysis for automatic speechreading and character animation," in Proc. 2nd Int. Conf. Automatic Face and Gesture Recognition, pp. 357-362, Killington, Vt, USA, 1996.
13. C. Neti, G. Potamianos, J. Luettin, et al., "Audio-visual speech recognition," Tech. Rep., Johns Hopkins University, Baltimore, Md, USA, October 2000.
14. G. Potamianos, J. Luettin, and C. Neti, "Hierarchical discriminant features for audio-visual LVCSR," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 1, pp. 165-168, Salt Lake City, Utah, USA, 2001.
15. G. Potamianos and H. P. Graph, "Discriminative training of HMM stream exponents for audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 6, pp. 3733-3736, Seattle, Wash, USA, 1998.
16. C. Bregler and Y. Conig, ""Eigenlips" for robust speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, pp. 669-672, Adelaide, Australia, 1994.
17. J. Luettin and N. A. Thacker, "Speechreading using probabilistic models," Computer Vision and Image Understanding, vol. 65, no. 2, pp. 163-178, 1997.
18. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, 2000.
19. C. Neti, G. Iyengar, G. Potamianos, A. Senior, and B. Maison, "Perceptual interfaces for information interaction: Joint processing of audio and visual information for human-computer interaction," in Proc. International Conference on Spoken Language Processing, vol. III, pp. 11-14, Beijing, China, October 2000.
20. R. Kaucic, B. Dalton, and A. Blake, "Real-time lip tracking for audio-visual speech recognition applications," in Proc. European Conference on Computer Vision, vol. 2, pp. 376-387, Cambridge, UK, 1996.
21. G. Chiou and J.-N. Hwang, "Lipreading from color video," IEEE Trans. Image Processing, vol. 6, no. 8, pp. 1192-1195, 1997.
22. M. T. Chan, Y. Zhang, and T. S. Huang, "Real-time lip tracking and bimodal continuous speech recognition," in Proc. IEEE 2nd Workshop on Multimedia Signal Processing, pp. 65-70, Redondo Beach, Los Angeles, Calif, USA, December 1998.
23. A. L. Yuille, P. W. Hallinan, and D. S. Cohen, "Feature extraction from faces using deformable templates," International Journal of Computer Vision, vol. 8, no. 2, pp. 99-111, 1992.
24. Text for ISO/IEC FDIS 14496-2 Visual, ISO/IEC JTC1/SC29/WG11 N2502, November 1998.
25. Text for ISO/IEC FDIS 14496-1 Systems, ISO/IEC JTC1/SC29/ WG11 N2502, November 1998.
26. F. Lavagetto and R. Pockaj, "An efficient use of MPEG-4 FAP interpolation for facial animation at 70 bits/frame," IEEE Trans. Circuits and Systems for Video Technology, vol. 11, no. 10, pp. 1085-1097, 2001.
27. E. Petajan, "Approaches to visual speech processing based on the MPEG-4 face animation standard," in Proc. IEEE International Conference on Multimedia and Expo(I), pp. 575-587, New York, NY, USA, 30 July-2 August 2000.
28. L. Bernstein and S. Eberhardt, "Johns Hopkins lipreading corpus I-II," Tech. Rep., Johns Hopkins University, Baltimore, Md, USA, 1986.
29. G. Abrantes, FACE-Facial Animation System, version 3.3.1, Institute Superior Técnico, (c), Lisbon, Portugal, 1997-1998.
30. M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active con-tour models," International Journal of Computer Vision, vol. 1, no. 4, pp. 321-331, 1988.
31. L. D. Cohen and I. Cohen, "Finite element methods for active contour models and balloons for 2-D and 3-D images," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1131-1147, 1993.
32. C. Xu and J. L. Prince, "Gradient vector flow: A new external force for snakes," in Proc. IEEE International Conf. on Computer Vision and Pattern Recognition, pp. 66-71, Puerto Rico, June 1997.
33. C. Xu and J. L. Prince, "Snakes, shapes, and gradient vector flow," IEEE Trans. Image Processing, vol. 7, no. 3, pp. 359-369, 1998.
34. R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision, McGraw-Hill, New York, NY, USA, 1995.
35. S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland, The HTK Book, Entropie, Cambridge, UK, 1999.
36. L. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, Prentice-Hall, Englewood Cliffs, NJ, USA, 1993.
37. H. Glotin, D. Vergyri, C. Neti, G. Potamianos, and J. Luettin, "Weighting schemes for audio-visual fusion in speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 1, pp. 165-168, Salt Lake City, Utah, USA, 2001.