Audio-visual speech recognition based on optimized product HMMs and GMM based-MCE-GPD stream weight estimation

IEICE Transactions on Information and Systems

Volumn E86-D, Issue 3, 2003, Pages 454-463

  Kumatani, Kenichi ^a   Nakamura, Satoshi ^b  

^a SHARP CORPORATION   (Japan)

^b ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

Audio visual speech recognition; Bi modal; Generalized probabilistic descent (GPD); Minimum classification error (MCE); Stream weight

Indexed keywords

ALGORITHMS; AUDITION; COMPUTATIONAL METHODS; DATABASE SYSTEMS; ERROR ANALYSIS; IMAGE PROCESSING; MARKOV PROCESSES; OPTIMIZATION; PROBABILITY; VISION;

GENERALIZED PROBABILISTIC DESCENT (GPD);

SPEECH RECOGNITION;

EID: 0038381727     PISSN: 09168532     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (29)

1. E. Vatikiotis-Bateson, I.M. Eigsti, S. Yano, and K. Munhall, "Eye movement of perceivers during audio-visual speech intelligibility in noise," Perception and Psychophysics, vol.60, no.6, pp.926-940, 1998.
2. W.H. Sumby and I. Pollack, "Visual contribution to speech inteligibility in noise," J. Acoust. Soc. Am., vol.26, pp.212-215, March 1954.
3. C. Benoit, T. Guiard Marigny, B. LeGoffand, and A. Adjoudani, "Which components of the face do humans and machines best speechread?," in Speechreading by Humans and Machines: Models, Systems and Applications, NATO ASI Series, pp.315-328, Springer-Verlag, 1996.
4. C. Bregler, H. Hild, S. Manke, and A. Waibel, "Improving connected letter recognition by lipreading," Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'93), vol.1, pp.557-560, April 1993.
5. S. Basu, C. Neti, N. Rajput, A. Senior, L. Subramaniam, and A. Verma, "Audio-visual large vocabulary continuous speech recognition in the broadcast domain," Workshop on Multimedia Signal Processing, pp.475-481, Dec. 1998.
6. S. Nakamura, R. Nagai, and K. Shikano, "Improved bimodal speech recognition using tied-mixture HMMs and 5000 word Audio-Visual Synchronous database," Proc. EUROSPEECH'97, pp.1623-1626, 1997.
7. M.J. Tomlinson, M.J. Russell, and N.M. Brooke, "Integrating audio and visual information to provide highly robust speech recognition," Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'96), vol.2, pp.821-824, May 1996.
8. S. Dupont and J. Luettin, "Using the multi-stream approach for continuous audio-visual speech recognition: experiments on the M2VTS DATABASE," Proc. International Conference on Spoken Language Processing (ICSLP'98), vol.4, pp.1283-1286, Oct. 1998.
9. S. Dupont and J. Luettin, "Audio-visual speech modelling for continuous speech recognition," IEEE Trans. Multimed., vol.2, no.3, pp.141-151, Sept. 2000.
10. A. Rogozan, P. Deleglise, and M. Alissali, "Adaptive determination of audio and visual weights for Automatic speech recognition," Proc. Europ. Tut. Work. Audio-Visual Speech Process (AVSP), pp.61-64, Sept. 1997.
11. G. Potamianos and C. Neti, "Stream confidence estimation for audio-visual speech recognition," Proc. International Conference on Spoken Language Processing (ICSLP'00), vol.3, pp.746-749, Oct. 2000.
12. S. Nakamura, H. Ito and K. Shikano, "Stream weight optimization of speech and lip image sequence for Audio-Visual speech recognition," Proc. International Conference on Spoken Language Processing (ICSLP'00), vol.3, pp.20-23, Oct. 2000.
13. G. Potamianos and H.P. Graf, "Discriminative training of HMM stream exponents for Audio-Visual speech recognition," Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'98), vol.6, pp.3733-3736, May 1998.
14. C. Miyajima, K. Tokuda, and Tadashi Kitamura, "Audio-Visual speech recognition using MCE-based HMMs and model-dependent stream weights," Proc. International Conference on Spoken Language Processing (ICSLP'00), vol.2, pp.1023-1026, Oct. 2000.
15. K. Kumatani, S. Nakamura, and K. Shikano, "An adaptive integration based on product HMM for audio-visual speech recognition," Proc. IEEE International Conference Multimedia and Expo (ICME'01), vol.1, Aug. 2001.
16. H. Bourlard and S. Doupont, "A new ASR approach based on independent processing and recombination of partial frequency bands," Proc. International Conference on Spoken Language Processing (ICSLP'96), pp.426-429, 1996.
17. A. Morris, A. Hagen, H. Glotin, and H. Bourlard, "Multistream adaptive evidence combination for noise robust ASR," Speech Commun, vol.34, pp.25-40, 2001.
18. J. Hernando, "Maximum likelihood weighting of dynamic speech features for CDHMM speech recognition," Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'97), vol.2, pp.1267-1270, April 1997.
19. J. Luettin, N.A. Thacker, and S.W. Beet, "Visual speech recognition using active shape models and hidden Markov models," Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-96), vol.2, pp.817-820, May 1996.
20. S. Nakamura, "Overview on recent activities in multi-modal corpora," COCOSDA Workshop, Oct. 2000.
21. S. Matsuda, M. Nakai, H. Shimodair, and S. Sagayama, "Asynchronous-transition HMM," Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'00), pp.1005-1008, May 2000.
22. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative association of orofacial and vocal-tract shapes," Proc. Europ. Tut. Work. Audio-Visual Speech Process (AVSP), pp.41-44, Sept. 1997.
23. S. Nakamura and E. Yamamoto, "Speech-to-lip movement synthesis by maximizing audio-visual joint probability based on the EM algorithm," J. VLSI Signal Processing, vol.27, no.1/2, pp.119-126, 2001.
24. K. Kakihara, S. Nakamura, and K. Shikano, "Speech-to-face movement synthesis based on HMMs," Proc. IEEE International Conference Multimedia and Expo (ICME'00), no.MP7.07, 2000.
25. S. Ogata, K. Murai, S. Nakamura, and S. Morisima, "Model based lip synchronization with an automatic translation system," Proc. IEEE International Conference Multimedia and Expo (ICME'01), Aug. 2001.
26. X.D. Huang, Y. Ariki, and N.A. Jack, Hidden Markov Models for Speech Recognition, Edinburgh Information Technology Series, Edinburgh University Press, Edinburgh, 1990.
27. W. Chou, B.-H. Juang, and C.-H. Lee, "Segmental GPD training of HMM based speech recognizer," Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'92), vol.1, pp.473-476, May 2000.
28. W. Chou, B.-H. Juang, C.-H. Lee, and F.K. Soong, "A minimum error rate pattern recognition approach to speech recognition," J. Pattern Recog. Art. Intell., Col. VIII, pp.5-31, 1994.
29. S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland, HTK-Hidden Markov Model Toolkit, Version 3.0, Microsoft Corporation, 2000.