Audio-visual speech modeling for continuous speech recognition

IEEE Transactions on Multimedia

Volumn 2, Issue 3, 2000, Pages 141-151

  Dupont, Steéphane ^a   Luettin, Juergen ^b  

^a FACULTÉ POLYTECHNIQUE DE MONS   (Belgium)

^b IDIAP RESEARCH INSTITUTE   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

MARKOV PROCESSES; MATHEMATICAL MODELS; SENSOR DATA FUSION; SIGNAL TO NOISE RATIO; SPEECH ANALYSIS; SPEECH RECOGNITION;

AUDIO-VISUAL SPEECH MODELING; HIDDEN MARKOV MODELS; JOINT TEMPORAL MODELING; PERCEPTUAL LINEAR PREDICTION; SENSOR INTEGRATION;

MULTIMEDIA SYSTEMS;

EID: 0034270644     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/6046.865479     Document Type: Article

References (48)

1. A. Q. Summerfield, "Lipreading and audio-visual speech perception," Philos. Trans. R. Soc. London B. vol. 335, pp. 71-78, 1992.
2. W. H. Sumby and I. Pollak, "Visual contributions to speech intelligibility in noise," J. Acoust. Soc. Amer., vol. 26, pp. 212-215, 1954.
3. K. W. Grant and L. D. Braida, "Evaluating the articulation index for auditory-visual input," J. Acoust. Soc. Amer., vol. 89, no. 6, pp. 2952-2960, 1991.
4. J. Luettin, "Visual Speech and Speaker Recognition," Ph.D. dissertation, Univ. Sheffield, Sheffield, U.K., 1997.
5. D. Reisberg, J. McLean, and A. Goldfield, "Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli," in Hearing by Eye: The Psychology of Lip-Reading, B. Dodd and R. Campbell, Eds. London, U.K.: Lawrence Erlbaum, 1987, pp. 97-113.
6. H. McGurk and J. Mac Donald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, 1976.
7. R. P. Lippmann, "Speech recognition by machines and humans," Speech Commun., vol. 22, pp. 1-15, 1997.
8. Y. Gong, "Speech recognition in noisy environments: A survey," Speech Commun., vol. 16, pp. 261-291, 1995.
9. R. Cole, L. Hirschmann, and L. Atlas et al., "The challenge of spoken language processing: Research directions for the nineties," IEEE Trans. Speech Audio Processing, vol. 3, no. 1, pp. 1-20, 1995.
10. B. P. Yuhas, M. H. Goldstein, T. J. Sejnowski, and R. E. Jenkins, "Neural network models of sensory integration for improved vowel recognition," Proc. IEEE, vol. 78, pp. 1658-1668, Oct. 1990.
11. C. Bregler and S. M. Omohundro, "Nonlinear manifold learning for visual speech recognition," in IEEE Int. Conf. Computer Vision, Piscataway, NJ, 1995, pp. 494-499.
12. P. L. Silsbee and A. C. Bovik, "Computer lipreading for improved accuracy in automatic speech recognition," IEEE Trans. Speech Audio Processing, vol. 4, no. 5, pp. 337-351, 1996.
13. G. Potamianos, H. P. Graf, and E. Cosatto, "An image transform approach for HMM based automatic lipreading," in Proc. IEEE Int. Conf. Image Processing, 1998, pp. 173-177.
14. K. Mase and A. Pentland, "Automatic lipreading by optical flow analysis," Syst. Comput. Jpn., vol. 22, no. 6, 1991.
15. E. D. Petajan, "Automatic lipreading to enhance speech recognition," in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1985, pp. 40-47.
16. T. Coianiz, L. Torresani, and B. Capril, "2D deformable models for visual speech analysis," in Speechreading by Humans and Machines: Models, Systems and Applications, D. G. Stork and M. E. Hennecke, Eds. Berlin, Germany: Springer-Verlag, 1996, vol. 150 of NATO ASI Series, Series F: Computer and Systems Sciences, pp. 391-398.
17. J. Luettin and N. A. Thacker, "Speechreading using probabilistic models," Comput. Vis. Image Understand., vol. 65, no. 2, pp. 163-178, Feb. 1997.
18. S. Basu, N. Oliver, and A. Pentland, "3D modeling and tracking of human lip motion," in Proc. IEEE Int. Conf. Computer Vision, 1998.
19. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Active shape models-their training and application," Comput. Vis. Image Understand., vol. 61, pp. 38-59, Jan. 1995.
20. A. Lanitis, C. J. Taylor, and T. F. Cootes, "Automatic interpretation and coding of face images using flexible models," IEEE Trans. Pattern Anal. Machine Intell., vol. 19, pp. 743-756, July 1997.
21. J. A. Neider and R. Mead, "A simplex method for function optimization," Comput. J., vol. 7, no. 4, pp. 308-313, 1965.
22. M. S. Gray, J. R. Movellan, and T. J. Sejnowski, "Dynamic features for visual speechreading: A systematic comparison," in Advances in Neural Information Processing Systems, M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, vol. 9.
23. M. J. Tomlinson, M. J. Rüssel, and N. M. Brooke, "Integrating audio and visual information to provide highly robust speech recognition," in Proc. IEEE Int. Conf. Acoust. Speech and Signal Processing, 1996, pp. 821-824.
24. L. Braida, "Crossmodal integration in the identification of consonants," Q. J. Exp. Psych., vol. 43A, no. 3, pp. 647-677, 1991.
25. N. P. Erber and C. L. De Filippo, "Voice-mouth synthesis of tactual/visual perception of /pa, ba, ma/," J. Acoust. Soc. Amer, vol. 64, pp. 1015-1019, 1978.
26. K. P. Green and J. L. Miller, "On the role of visual rate information in phonetic perception," Percept. Psychophys., vol. 38, no. 3, pp. 269-276, 1985.
27. H. Fletcher, Speech and Hearing in Communication. New York: Krieger, 1953.
28. J. B. Allen, "How do humans process and recognize speech?," IEEE Trans. Speech Audio Processing, vol. 2, no. 4, pp. 567-577, 1994.
29. H. Bourlard and S. Dupont, "Sub-band-based speech recognition," in Proc. IEEE Int.. Conf. Acoustic Speech and Signal Processing, Apr. 1997, pp. 1251-1254.
30. L. Rabiner and B. H. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall Signal Processing Series, 1993.
31. M.I. Jordan, Z. Ghahramani, and L. K. Saul, "Hidden Markov decision trees," in Advances in Neural Information Processing Systems, M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, vol. 9.
32. G. Gravier, M. Sigelle, and G. Chollet, "Toward Markov random field modeling of speech," in Proc. Int. Conf. Spoken Language Processing, Sydney, Australia, Dec. 1998.
33. H. Sakoe, "Two level DP matching-a dynamic time warping based pattern matching algorithm for continuous speech recognition," IEEE Trans. IECE Jpn., vol. 3, 1979.
34. A. P. Varga and R. K. Moore, "Hidden Markov model decomposition of speech and noise," in Proc. IEEE Int. Conf. Acoustic Speech and Signal Processing, 1990, pp. 845-848.
35. T. Aral and S. Greenberg, "Speech intelligibility in the presence of crosschannel spectral asynchrony," in Proc. ICASSP, 1998, pp. 933-936.
36. D. W. Massaro and M. M. Cohen, "Perceiving asynchronous bimodal speech in consonant vowel and vowel syllables," Speech Commun., vol. 13, pp. 127-134, 1993.
37. P. M. Smeele et al., "Intelligibility of audio-visually desynchronized speech: Asymmetrical effect of phoneme position," in Proc. Int. Conf. Spoken Language Processing, Alberta, Canada, 1992, pp. 65-68.
38. H. Bourlard and N. Morgan, Connectionist Speech Recognition-A Hybrid Approach. Norwell, MA: Kluwer, 1994.
39. G. Potamianos and H. P. Graf, "Discriminative training of hmm stream exponents for audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoustic Speech and Signal Processing, Seattle, WA, 1998, pp. 3733-3736.
40. N.B. Yoma, F. R. Mclnnes, and M. A. Jack, "Weighted viterbi algorithm and state duration modeling for speech recognition in noise," in Proc. IEEE Int. Conf. Acoustic Speech and Signal Processing, Seattle, WA, 1998, pp. 709-712.
41. S. Pigeon and L. Vandendorpe, "The M2VTS multimodal face database (release 1.00)," in Proc. of the First International Conference on Audioand Video-based Biometrie Person Authentication, Crans-Montana, Switzerland, 1997.
42. H. Hermansky, "Perceptual linear predictive (PLP) analysis of speech," J. Acoust. Soc. Amer., vol. 87, no. 4, pp. 1738-1752, Apr. 1990.
43. S. Furui, "Speaker indépendant isolated word recognizer using dynamic features of speech spectrum," IEEE Trans. Acoust., Speech, Signal Processing, vol. 34, no. 1, pp. 52-59, 1986.
44. G. Chollet, J. L. Cochard, C. Jaboulet, A. Constantinescu, and P. Langlais, Swiss French polyphone and polyvar: Telephone speech databases to model inter and intra-speaker variability, , IDIAP, Martigny, Switzerland, 1996.
45. N. N. Mirghafori, "A Multi-Band Approach to Automatic Speech Recognition," Ph.D. dissertation, Int. Comput. Sei. Inst., Berkeley, CA, Jan. 1999.
46. H. Hermansky and N. Morgan, "Rasta processing of speech," IEEE Trans. Speech Audio Processing, vol. 2, no. 4, pp. 578-589, 1994.
47. ULg, "ULg-acoustics laboratory-The MADRAS project (1998, Aug.). Online Available http://www.montefiore.ulg.ac.be/services/acous/homelab.html
48. S. Dupont and C. Ris, "Assessing local noise level estimation methods," in Workshop on Robust Methods for Speech Recognition in Adverse Conditions (Nokia, COST249, IEEE) Tampere, Finland, May 1999, pp. 115-118.