Audio-visual deep learning for noise robust speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2013, Pages 7596-7599

  Huang, Jing ^a   Kingsbury, Brian ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Audio visual speech recognition; Deep belief networks; Noise robustness

Indexed keywords

AUDIO VISUAL SPEECH RECOGNITION; AUTOMATIC SPEECH RECOGNITION; DECISION FUSION METHODS; DEEP BELIEF NETWORK (DBN); DEEP BELIEF NETWORKS; GAUSSIAN MIXTURE MODEL; NOISE ROBUST SPEECH RECOGNITION; NOISE ROBUSTNESS;

SIGNAL PROCESSING; SPEECH RECOGNITION;

ACOUSTIC NOISE;

EID: 84890465549     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2013.6639140     Document Type: Conference Paper

References (24)

1. W.H. Sumby and I. Pollack (1954), "Visual contribution to speech intelligibility in noise," in J. Acoustical Society America, 26: 212-215.
2. T. Chen and R.R. Rao (1998), "Audio-visual integration in multi-modal communication," in Proc. IEEE, 86(5): 837-852.
3. A. Janin, D. Ellis and N. Morgan (1999) "Multi-stream speech recognition: Ready for prime time?", in Proc. Europ. Conf. Speech Technol., pp. 591-594, 1999.
4. S. Dupont and J. Luettin (2000), "Audio-visual speech modeling for continuous speech recognition," in IEEE Trans. Multimedia, 2(3): 141-151.
5. C.C. Chibelushi, F. Deravi, and J.S.D. Mason (2002), "A review of speech-based bimodal recognition," in IEEE Trans. Multimedia, 4(1): 23-37.
6. M. Heckmann, F. Berthommier, and K. Kroschel (2002), "Noise adaptive stream weighting in audio-visual speech recognition," in EURASIP J. Appl. Signal Process., 2002(11): 1260-1273.
7. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A.W. Senior (2003), "Recent advances in the automatic recognition of audio-visual speech," in Proc. IEEE, 91(9): 1306-1326.
8. G. Potamianos (2006), "Audio-Visual Speech Recognition," in Encyclopedia of Language and Linguistics, Second Edition, (Speech Technology Section-Computer Understanding of Speech), K. Brown (Ed. In Chief), Elsevier, Oxford, United Kingdom, ISBN: 0-08-044299-4, 2006.
9. J. Huang, G. Potamianos, J. Connell and C. Neti (2004), "Audio-visual speech recognition using an infrared headset," in Speech Communication 44(4), 83-96.
10. E. Marcheret, S. Chu, V. Goel, G. Potamianos (2004), "Efficient Likelihood Computation in Multi-Stream HMM Based Audio-Visual Speech Recognition," in Int. Conf. Speech and Language Processing, 2004.
11. D. Povey and P. C. Woodland, "Minimum Phone Error and I-smoothing for Improved Discriminative Training," in Proceedings of ICASSP, 2002.
12. D. Povey, B. Kingsbury, L. Mangu, G. Saon, H. Soltau, G. Zweig, "fMPE: Discriminatively trained features for speech recognition," in Proceedings of ICASSP, 2005.
13. J. Huang and D. Povey, "Discriminatively Trained Features Using fMPE for Multi-Stream Audio-Visual Speech Recognition," in Proceedings of Interspeech, 2005.
14. J. Huang and K. Visweswariah, "Combined Discriminative Training for Multi-Stream HMM-based Audio-Visual Speech Recognition," in Proceedings of Interspeech, 2009.
15. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition," in IEEE Signal Processing Magazine, 29(6): 82-97, 2012.
16. A. Mohamed, G. Hinton, G. Penn, "Understanding how Deep Belief Networks perform acoustic modelling," in Proceedings of ICASSP, 2012.
17. P. Duchnowski, U. Meier, and A. Waibel, "See me, hear me: Integrating automatic speech recognition and lipreading," in Proceedings of ICSLP, 1994.
18. U. Meier, W. Hurst and P. Duchnowski, "Adaptive Bimodal Sensor Fursion for Automatic Speechreading," in Proceedings of ICASSP, 1996.
19. T. Lewis and D. Powers, "Audio-Visual Speech Recognition using Red Exclusion and Neural Networks," in Journal of Research and Practice in Information Technology, 2003.
20. M. Kim, J. Ryu, and E. Kim, "Speech Recognition by Integrating Audio, Visual and Contextual Features Based on Neural Networks," in Advances in Natural Computation, Lecture Notes in Computer Science, 2005.
21. J. Ngiam, A. Khosla, J. Nam, H. Lee and A.Ng, "Multimodal Deep Learning", in International Conference on Machine Learning, 2011.
22. A. Garg, G. Potamianos, C. Neti, T. Huang, "Frame-Dependent Multi-Stream Reliability Indicators for Audio-Visual Speech Recognition," in Int. Conf. Acoustic Speech and Signal Processing, 2003.
23. V. Pitsikalis, A. Katsamanis, G. Papandreou, and P. Maragos. "Adaptive multimodal fusion by uncertainty compensation," in Proceedings of ICSLP, 2006.
24. G. E. Hinton, S. Osindero, and Y. Teh. "A Fast Learning Algorithm for Deep Belief Nets," in Neural Computation, vol. 18, pp. 1527-1554, 2006.