Visual model structures and synchrony constraints for audio-visual speech recognition

IEEE Transactions on Audio, Speech and Language Processing

Volumn 14, Issue 3, 2006, Pages 1082-1089

  Hazen, Timothy J ^a,b  

^a IEEE   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Audio visual speech recognition; Lip reading; Multimodal speech processing

Indexed keywords

AUDIO VISUAL SPEECH RECOGNITION (AVSR); LIP READING; MULTIMODAL SPEECH PROCESSING; WORD ERROR RATE;

AUDIO ACOUSTICS; BIT ERROR RATE; HIDDEN MARKOV MODELS; SIGNAL TO NOISE RATIO; SPEECH PROCESSING; VISUAL COMMUNICATION;

SPEECH RECOGNITION;

EID: 34047263009     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSA.2005.857572     Document Type: Article

References (36)

1. C. Benoit, "The intrinsic bimodality of speech communication and the synthesis of talking faces," in The Structure of Multimodal Dialogue II, M. M. Taylor, F. Nel, and D. Bouwhuis, Eds. Amsterdam, The Netherlands: John Benjamins, 2000, pp. 485-502.
2. C. Chibelushi, F. Deravi, and J. Mason, "A review of speech-based bimodal recognition," IEEE Trans. Multimedia, vol. 4, no. 1, pp. 23-37, Mar. 2002.
3. G. Potamianos, C. Neti, G. lyengar, and E. Helmuth, "Large- vocabulary audio-visual speech recognition by machines and humans," in Proc. Eurospeech, Aalborg, Denmark, Sep. 2001, pp. 1293-1296.
4. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. Senior, "Recent advances in the automatic recognition of audiovisual speech," Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, Sep. 2003.
5. W. Sumby and I. Pollack, "Visual contribution to speech intelligibility in noise," J. Acoust. Soc. Amer., vol. 26, no. 2, pp. 212-215, Mar. 1954.
6. G. Potamianos and C. Neti, "Audio-visual speech recognition in challenging environments," in Proc. Eurospeech, Geneva, Switzerland, Sep. 2003, pp. 1293-1296.
7. J. Glass, "A probabilistic framework for segment-based speech recognition," Comput. Speech Lang., vol. 17, no. 2-3, pp. 137-152, Apr./Jul. 2003.
8. T. Hazen, K. Saenko, C. La, and J. Glass, "A segment-based audio-visual speech recognizer: Data collection, development, and initial experiments," in Proc. Int. Conf. Multimodal Interfaces, PA, Oct. 2004.
9. M. Chan, Y. Zhang, and T. Huang, "Real-time lip tracking and bimodal continuous speech recognition," in Proc. Workshop on Multimedia Signal Processing, Redondo Beach, CA, 1998, pp. 65-70.
10. I. Matthews, J. Bangham, and S. Cox, "Audio-visual speech recognition using multiscale nonlinear image decomposition," in Proc. Int. Conf. Spoken Language Processing, Philadelphia, PA, 1996, pp. 38-41.
11. S. Pigeon and L. Vandendorpe, "The M2VTS multimodal face database," in Proc. Audio- Video-based Biometric Person Authentication Workshop, 1997.
12. S. Chu and T. Huang, "Bimodal speech recognition using coupled hidden Markov models," in Proc. Int. Conf. Spoken Language Processing, vol. II, Beijing, China, Oct. 2000.
13. E. Patterson, S. Gurbuz, Z. Tufekci, and J. Gowdy, "CUAVE: A new audio-visual database for multi-modal human-computer interface research," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 2, Orlando, FL, May 2002, pp. 2017-2020.
14. K. Messer, J. Matas, J. Kittler, and K. Jonsson, "XM2VTSDB: The extended M2VTS database," in Audio- and Video-based Biometric Person Authentication, AVBPA'99, Washington, DC, Mar. 1999, pp. 72-77. 16 IDIAP-RR 99-02.
15. S. Chu and T. Huang, "Audio-visual speech modeling using coupled hidden Markov models," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 2, Orlando, FL, May 2002, pp. 2009-2012.
16. C. Sanderson, "The VidTIMIT database," in IDIAP Communication 02-06. Martigny, Switzerland: IDIAP, Aug. 2002.
17. V. Zue, S. Seneff, and J. Glass, "Speech database development: TIMIT and beyond," Speech Commun., vol. 9, no. 4, pp. 351-356, 1990.
18. A. Varga and H. Steeneken, "Assessment for automatic speech recognition II: NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems," Speech Commun., vol. 12, no. 3, pp. 247-251, Jul. 1993.
19. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, Sep. 2000.
20. A. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, and K. Murphy, "A coupled HMM for audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 2, Orlando, FL, May 2002, pp. 2013-2016.
21. G. Gravier, G. Potamianos, and C. Neti, "Asynchrony modeling for audio-visual speech recognition," in Proc. Human Language Technology Conf., San Diego, CA, Mar. 2002, pp. 1-6.
22. U. Meier, R. Stiefelhagen, J. Yang, and A. Waibel, "Toward unrestricted lip reading," Int. J. Pattern Recognit. Artif. Intell., no. 14, pp. 571-585, Aug. 2000.
23. M. Heckmann, F. Berthommier, and K. Kroschel, "Optimal weighting of posteriors for audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 1, Salt Lake City, UT, May 2001, pp. 161-164.
24. H. Glotin, D. Vergyri, C. Neti, G. Polamianos, and J. Luettin, "Weighting schemes for audio-visual fusion in speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 1, Salt Lake City, UT, May 2001, pp. 165-168.
25. S. Tamura, K. Iwano, and S. Furui, "A stream-weight optimization method for audio-visual speech recognition using multi-stream HMMs," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 1, Montreal, QC, Canada, May 2004, pp. 857-860.
26. C. Fisher, "Confusions among visually perceived consonants," J. Speech Hearing Res., vol. 11, pp. 796-804, 1968.
27. K. Berger, Speechreading: Principles and Methods. Baltimore, MD: National Educational, 1972.
28. T. Chen and R. Rao, "Audio-visual integration in multimodal communication," Proc. IEEE, vol. 86, no. 5, pp. 837-852, May 1998.
29. P. Silsbee, "Sensory integration in audiovisual automatic speech recognition," in Proce. 28th Annual Asilomar Conf. Signals, Systems, and Computers, vol. I, Pacific Grove, CA, Oct./Nov. 1994, pp. 561-565.
30. A. Rogozan, "Discriminative learning of visual data for audiovisual speech recognition," Int. J. Artif. Intell. Tools, vol. 8, no. 1, pp. 43-52, Mar. 1999.
31. C. Neti et al., "Audio-visual speech recognition," Center Lang. Speech Process., The Johns Hopkins Univ., Baltimore, MD, Tech. Rep., 2000.
32. K. Finn and A. Montgomery, "Automatic optically-based recognition of speech," Pattern Recognit. Lett., vol. 8, no. 3, pp. 159-164, Oct. 1988.
33. C. Bregler, H. Hild, S. Manke, and A. Waibel, "Improving connected letter recognition by lipreading," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 1, Minneapolis, MN, Apr. 1993, pp. 557-560.
34. G. Sannier, S. Balcisoy, N. Magnenat-Thalmann, and D. Thalmann, "VHD: A system for directing real-time virtual actors," Vis. Comput., vol. 15, no. 7/8, pp. 320-329, Nov. 1999.
35. S. Young, J. Odell, and P. Woodland, "Tree-based state tying for high accuracy acoustic modeling," in Proc. ARPA Human Language Technology Workshop, Princeton, NJ, Mar. 1994, pp. 307-312.
36. S. Bengio, "An asynchronous hidden Markov model for audio-visual speech recognition," in Advances in Neural Information Processing Systems, NIPS 75, S. Decker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, pp. 1237-1244.