Computer lipreading for improved accuracy in automatic speech recognition

IEEE Transactions on Speech and Audio Processing

Volumn 4, Issue 5, 1996, Pages 337-350

  Silsbee, Peter L ^a   Bovik, Alan C ^a  

^a The University of Texas at Austin   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUDIO SYSTEMS; CHARACTER RECOGNITION; COMPUTER VISION; ERRORS; MARKOV PROCESSES; MATHEMATICAL MODELS; PERFORMANCE;

AUDIOVISUAL SYSTEM; AUTOMATIC SPEECH RECOGNITION SYSTEMS; HIDDEN MARKOV MODELS;

SPEECH RECOGNITION;

EID: 0030247984     PISSN: 10636676     EISSN: None     Source Type: Journal    
DOI: 10.1109/89.536928     Document Type: Article

References (46)

1. A. Erell and M. Weintraub, "Energy-conditioned spectral estimation for recognition of noisy speech," IEEE Trans, Speech Audio Processing, vol. 1, no. 1, pp. 84-89, Jan. 1993.
2. Y. Ephraim, "Gain-adapted hidden Markov models for recognition of clean and noisy speech," IEEE Trans. Acoust., Speech, Signal Processing, vol. 40, no. 4, pp. 725-735, Apr. 1992.
3. H. Hermansky, "Perceptual linear predictive (PLP) analysis of speech," J. Acoust. Soc. Amer., vol. 87, no. 4, pp. 1738-1752, Apr. 1990.
4. O. Ghitza, "Auditory nerve representation as a basis for speech processing," in S. Furui and M. M. Sondhi, Eds., Advances in Speech Signal Processing. New York: Marcel Dekker, 1992, pp. 453-485.
5. J.-C. Junqua, H. Wakita, and H. Hermansky, "Evaluation and optimization of perceptually based ASR front end," IEEE Trims. Speech Audio Processing, vol. 1, no. 1, pp. 39-48, Jan. 1993.
6. W. A. Woods, "Language processing for speech understanding," in A. Waibel and K.-F. Lee, Eds., Readings in Speech Recognition. San Mateo, CA: Morgan Kaufman, 1990, pp. 519-533.
7. S. R. Young, A. G. Hauptmann, W. H. Ward, E. T. Smith, and P. Werner, "High level knowledge sources in usable speech recognition systems," in A. Waibel and K.-F. Lee, Eds., Readings in Speech Recognition, San Mateo, CA: Morgan Kaufmann, 1990, pp. 538-549.
8. K.-F. Lee, Large-Vocabulary "Speaker-independent continuous speech recognition: The SPHINX system," Ph.D. thesis, Carnegie-Mellon Univ., Pittsburgh, PA, 1988.
9. A. Waibel, "Prosodie knowledge sources for word hypothesization in a continuous speech recognition system," in A. Waibel and K.-F. Lee, Eds., Readings in Speech Recognition. San Mateo, CA: Morgan Kaufmann, 1990, pp. 534-537.
10. E. D. Petajan, "Automatic lipreading to enhance speech recognition," Ph.D. thesis, Univ. of Illinois, Urbana, IL, 1984.
11. C. Bregler and S. M. Omohundro, "Surface learning with applications to lipreading," in J. D. Cowan, G. Tesauro, and J. Alspector, Eds, Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 1994, pp. 43-50, vol. 6.
12. C. Bregler, S. M. Omohundro, and Y. Konig, "A hybrid approach to bimodal speech recognition," in Proc. 28th Ann. Asilomar Conf. Signals, Syst., Comput., vol. 1, Pacific Grove, CA, Nov. 1994, pp. 556-560.
13. P. Cosi, E. M. Caldognetto, K. Vagges, G. A. Mian, and M. Contolini, "Bimodal recognition experiments with recurrent neural networks," in Proc. Int. Conf. Acoust., Speech, Signal Processing, vol. 2, 1994, pp. 11/553-556.
14. P. Duchnowski, U. Meier, and A. Waibel, "See me, hear me: Integrating automatic speech recognition and lip-reading," in Proc. Int. Conf. Spoken Language Processing, 1994.
15. K. E. Finn and A. A. Montgomery, "Automatic optically-based recognition of speech," Patt. Recogn. Lett., vol. 8, no. 3, pp. 159-164, 1988.
16. A. J. Goldschen, "Continuous automatic speech recognition by lipreading," Ph.D. thesis, Dept. of Elect. Eng. and Comput. Sci., George Washington Univ., 1993.
17. M. E. Hennecke, K. V. Prasad, and D. G. Stork, "Using deformable templates to infer visual speech dynamics," in Proc. 28th Ann. Asilomar Conf. Signals, Syst., Comput., vol. 1, Pacific Grove, CA, Nov. 1994, pp. 578-582.
18. J. R. Movellan, "Visual speech recognition with stochastic networks," in G. Tesauro, D. Touretzky, and T. Leen, Eds., Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, vol. 7, 1995, pp. 851-858.
19. S. Nishida, "Speech recognition enhancement by lip information," in Proc. Comput. Human Interfaces '86, pp. 198-204.
20. E. D. Petajan, "An improved automatic lipreading system to enhance speech recognition," Tech. Rep. 11251-871012-111TM, AT&T Bell Labs., 1987.
21. R. R. Rao and R. M. Mersereau, "Lip modeling for visual speech recognition," in Proc. 28th Ann. Asilomar Conf. Signals, Syst., Comput., vol. 1, Pacific Grove, CA, Nov. 1994, pp. 587-590.
22. J. Robert-Ribes, "Modèles d'integration audio-visuelle de signaux linguistiques: De la perception humaine à la reconnaissance automatique des voyelles," Ph.D. thesis. Institut National Polytechnique de Grenoble, Grnoble, France, 1995.
23. K. Mase and A. Pentland, "Lip reading: Automatic visual recognition of spoken words," Opt. Soc. Amer. Topical Mtg. Machine Vision, June 1989, pp. 1565-1570.
24. B. P. Yuhas, M. H. Goldstein, T. J. Sejnowski, and R. E. Jenkins, "Neural network models of sensory integration for improved vowel recognition," Proc. IEEE, vol. 78, no. 10, pp. 1658-1668, Oct. 1990.
25. D. G. Stork, G. Wolff, and E. Levine, "Neural network lipreading system for improved speech recognition," in Proc. Int. Joint Conf. Neural Networks, 1992, pp. 285-295.
26. P. L. Silsbee, "Sensory integration in audiovisual automatic speech recognition," in 28th Ann. Asilomar Conf. Signals, Syst., Comput., vol. I, Nov. 1994, pp. 561-565.
27. P. L. Silsbee and A. C. Bovik, "Visual lipreading by computer to improve automatic speech recognition accuracy," Tech. Rep., TR93-02-90, Univ. of Texas Comput. Vision Res. Center, Austin, TX, 1993.
28. G. J. Wolff, K. V. Prasad, D. G. Stork, and M. E. Hennecke, "Lipreading by neural networks: Visual preprocessing, learning and sensory integration," in J. D. Cowan, G. Tesauro, and J. Alspector, Eds., Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 1994, pp. 1027-1034, vol. 6.
29. J. Wu et al., "Neural network vowel recognition jointly using voice features and mouth shape image," Patt. Recogn., vol. 24, no. 10, pp. 921-927, 1991.
30. J. P. Haton, "Automatic recognition of noisy speech," in New Advances and Trends in Speech Recognition and Coding, NATO Advanced Study Institute, 1993.
31. W. H. Sumby and I. Pollock, "Visual contribution to speech intelligibility in noise," J. Acoust. Soc. Amer., vol. 26, pp. 212-215, 1954.
32. Q. Summerfield, "Some preliminaries to a comprehensive account of audio-visual speech perception," in B. Dodd and R. Campbell, Eds., Hearing by Eye: The Psychology of Lip-reading. London: Lawrence Erlbaum, 1987, pp. 3-51.
33. D. Reisbcrg, "Easy to hear but hard to understand: a lip-reading advantage with intact auditory stimuli," in B. Dodd and R. Campbell, Eds., Hearing by Eye: The Psychology of Lip-reading. London: Lawrence Erlbaum, 1987, pp. 97-113.
34. D. W. Massaro, "Speech perception by ear and eye," in B. Dodd and R. Campbell, Eds., Hearing by Eye: Tlie Psychology of Lip-reading. London: Lawrence Erlbaum, 1987, pp. 53-83.
35. H. McGurk and J. MacDonald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, 1976.
36. K. Mogford, "Lip-reading in the prelingually deaf," in B. Dodd and R. Campbell, Eds., Hearing by Eye: The Psychology of Lip-reading. London: Lawrence Erlbaum, 1987, pp. 191-211.
37. D. W. Massaro, Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry. Hillsdale, NJ: Lawrence Erlbaum, 1987.
38. K. W. Berger, Speechreading: Principles and Methods. Baltimore, MD: National Education, 1972.
39. P. L. Jackson, A. A. Montgomery, and C. A. Binnie, "Perceptual dimensions underlying vowel lipreading performance," J. Speech Hearing Res., vol. 19, pp. 796-812, 1976.
40. M. McGrath, A. Q. Summerfield, and N. M. Brooke, "Roles of lips and teeth in lipreading vowels," Proc. Inst. Acoust., pp. 401-408, 1984.
41. P. L. Silsbee, "Computer lipreading for improved accuracy in automatic speech recognition," Ph.D. thesis, Univ. of Texas, 1993.
42. V. M. Gupta, M. Lennig, and P. Mermelstein, "Integration of acoustic information in a large vocabulary word recognizer," in Proc. Int. Conf. Acoust., Speech, Signal Processing, 1987, pp. 697-700.
43. K.-F. Lee, Automatic Speech Recognition: The Development of the SPHINX System. Bostion: Kluwer, 1989.
44. W. H. Equitz, "A new vector quantization clustering algorithm," IEEE Trans. Acoust., Speech, Signal-Processing, vol. 37, no. 10, pp. 1568-1575, Oct. 1989.
45. R. M. Gray, "Vector quantization," IEEE Acoust., Speech, Signal Processing Mag., vol. 2, pp. 4-29, Apr. 1984.
46. D. H. Ballard and C. M. Brown, Computer Vision. Englewood Cliffs, NJ, Prentice-Hall, 1982.