Robust audio-visual speech recognition under noisy audio-video conditions

IEEE Transactions on Cybernetics

Volumn 44, Issue 2, 2014, Pages 175-184

  Stewart, Darryl ^a   Seymour, Rowan ^b   Pass, Adrian ^c   Ming, Ji ^a  

^a QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

^b UNIVERSITY OF WASHINGTON   (United States)

^c Pace Micro Technology plc   (United Kingdom)

Author keywords

Automatic speech recognition; human computer interaction; speech recognition

Indexed keywords

AUDIO VISUAL SPEECH RECOGNITION; AUTOMATIC SPEECH RECOGNITION; FRAME-BY-FRAME BASIS; INTEGRATION APPROACH; MPEG-4 VIDEO COMPRESSION; ROBUST RECOGNITION; STREAM INTEGRATION; WEIGHTING APPROACHES;

ACOUSTIC NOISE; CRIME; EXPERIMENTS; HUMAN COMPUTER INTERACTION; SPEECH RECOGNITION; VIDEO STREAMING;

AUDIO ACOUSTICS;

ALGORITHM; ARTICLE; AUTOMATED PATTERN RECOGNITION; AUTOMATIC SPEECH RECOGNITION; COMPUTER SIMULATION; HUMAN; METHODOLOGY; PATTERN RECOGNITION; PHYSIOLOGY; SIGNAL NOISE RATIO; SOUND DETECTION; STATISTICAL ANALYSIS; STATISTICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; PATTERN RECOGNITION, AUTOMATED; PATTERN RECOGNITION, PHYSIOLOGICAL; PATTERN RECOGNITION, VISUAL; SIGNAL-TO-NOISE RATIO; SOUND SPECTROGRAPHY; SPEECH RECOGNITION SOFTWARE;

EID: 84893400545     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2013.2250954     Document Type: Article

References (33)

1. H. McGurk and J. MacDonald, "Hearing lips and seeing voices," Nature, vol. 264, no. 5588, pp. 746-748, 1976.
2. A. Adjoudani and C. Benôit, "On the integration of auditory and visual parameters in an HMM-based ASR," in Speechreading by Humans and Machines, D. G. Stork and M. E. Hennecke, Eds. Berlin, Germany: Springer, 1996, pp. 461-471.
3. T. Chen, "Audiovisual speech processing. lip reading and lip synchronization," IEEE Signal Process. Mag., vol. 18, no. 1, pp. 9-21, Jan. 2001. (Pubitemid 32287667)
4. G. Potamianos, J. Luettin, and C. Neti, "Hierarchical discriminant features for audio-visual LVCSR," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2001, pp. 165-168. (Pubitemid 32839213)
5. R. Goecke, G. Potamianos, and C. Neti, "Noisy audio feature enhancement using audio-visual speech data," in Proc. Int. Conf. Acoust. Speech Signal Process., 2002, pp. 2025-2028.
6. J. Luettin, G. Potamianos, and C. Neti, "Asynchronous stream modeling for large vocabulary audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2001, pp. 169-172. (Pubitemid 32839214)
7. M. Wollmer, F. Eyben, B. Schuller, and G. Rigoll, "A multi-stream ASR framework for BLSTM modeling of conversational speech," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2011, pp. 4860-4863.
8. J. Huang and K. Visweswariah, "Improved decision trees for multistream HMM-based audio-visual continuous speech recognition," in Proc. Workshop IEEE Autom. Speech Recognit. Understanding, Nov. 2009, pp. 228-231.
9. R. Rajavel and P. S. Sathidevi, "A novel algorithm for acoustic and visual classifiers decision fusion in audio-visual speech recognition system," Signal Process. Int. J., vol. 4, no. 1 pp. 23-37, 2010.
10. V. Estellers, M. Gurban, and J. Thiran, "On dynamic stream weighting for audio-visual speech recognition," IEEE Trans. Audio Speech Language Process., vol. 20, no. 4, pp. 1145-1157, May 2012.
11. D. B. Dean, P. J. Lucey, S. Sridharan, and T. J. Wark, "Fused HMMadaptation of multi-stream HMMS for audio-visual speech recognition," in Proc. 8th Annu. Conf. Int. Speech Commun. Assoc., 2007, pp. 666-669.
12. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, Sep. 2000.
13. A. V. Nefian, L. Liang, X. Pi, X. Liu, and K. Murphy, "Dynamic Bayesian networks for audio-visual speech recognition," EURASIP J. Appl. Signal Process., pp. 1274-1288, Nov. 2002.
14. M. Heckmann, F. Berthommier, and K. Kroschel, "Noise adaptive stream weighting in audio-visual speech recognition," EURASIP J. Appl. Signal Process., vol. 11, pp. 1260-1273, Nov. 2002.
15. M. Heckmann, F. Berthommier, and K. Kroschel, "Optimal weighting of posteriors for audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., vol. 1. May 2001, pp. 161-164. (Pubitemid 32839212)
16. L. Terry, D. Shiell, and A. Katsaggelos, "Feature space video stream consistency estimation for dynamic stream weighting in audio-visual speech recognition," in Proc. 15th IEEE Int. Conf. Image Process., Oct. 2008, pp. 1316-1319.
17. X. Zhang, C. C. Broun, R. M. Mersereau, and M. A. Clements, "Automatic speechreading with applications to human-computer interfaces," EURASIP J. Appl. Signal Process., vol. 11, pp. 1228-1247, Nov. 2002.
18. S. Nakamura, H. Ito, and K. Shikano, "Stream weight optimization of speech and lip image sequence for audiovisual speech recognition," in Proc. Int. Conf. Spoken Language Process., vol. 3. 2000, pp. 20-23.
19. G. Gravier, S. Axelrod, G. Potamianos, and C. Neti, "Maximum entropy and MCE based HMM stream weight estimation for audio-visual ASR," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., vol. 1. May 2002, pp. 853-856.
20. A. Garg, G. Potamianos, C. Neti, and T. S. Huang, "Frame-dependent multi-stream reliability indicators for audio-visual speech recognition," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., vol. 1. Apr. 2003, pp. 24-27.
21. R. Seymour, D. Stewart, and J. Ming, "Audio-visual integration for robust speech recognition using maximum weighted stream posteriors," in Proc. Interspeech, 2007, pp. 654-657.
22. J. Schalkwyk, D. Beeferman, F. Beaufays, B. Byrne, C. Chelba, M. Cohen, M. Kamvar, and B. Strope, "'Your word is my command': Google search by voice: A case study," in Advances in Speech Recognition: Mobile Environments, Call Centers and Clinics, 2010, ch. 4, pp. 61-90.
23. R. Seymour, J. Ming, and D. Stewart, "A new posterior based audiovisual integration method for robust speech recognition," in Proc. Interspeech-Eurospeech, Sep. 2005, pp. 1229-1232. (Pubitemid 43908290)
24. J. Ming, J. Lin, and F. J. Smith, "A posterior union model with applications to robust speech and speaker recognition," EURASIP J. Applied Signal Process., Apr. 2006, pp. 1-12.
25. J. Lin, J. Ming, and D. Crookes, "Robust face recognition using posterior union model based neural networks," Comput. Vision, IET, vol. 3, no. 3, pp. 130-142, Sep. 2009.
26. K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, "XM2VTSDB: The extended M2VTS database," in Proc. Audio video-Based Biometric Person Authentication, Mar. 1999, pp. 72-77.
27. S. Young. (2000). The HTK Book (for HTK Version 3.0), Microsoft Corporation [Online]. Available: http://htk.eng.cam.ac.uk/docs/docs.shtml
28. M. U. R. Sanchez, "Aspects of facial biometrics for verification of personal identity," Ph.D. dissertation, Univ. Surrey, Guilford, U.K., 2000.
29. G. Potamianos, H. P. Graf, and E. Cosatto, "An image transform approach for HMM based automatic lipreading," in Proc. Int. Conf. Image Process., vol. 3. 1998, pp. 173-177.
30. R. Seymour, D. Stewart, and J. Ming, "Comparison of image transformbased features for visual speech recognition in clean and corrupted videos," EURASIP J. Image Video Process., vol. 2008, article 14, Apr. 2008.
31. D. Dean and S. Sridharan, "Dynamic visual features for audio-visual speaker verification," Comput. Speech Language, vol. 24, no. 2, pp. 136-149, 2010.
32. M. Heckmann, K. Kroschel, C. Savariaux, and F. Berthommier, "DCTbased video features for audio-visual speech recognition," in Proc. Int. Conf. Spoken Language Process., Denver, CO, USA, Sep. 2002, pp. 1925-1928.
33. D. B. Dean, T. J. Wark, and S. Sridharan. (2006). "An examination of audio-visual fused HMMS for speaker recognition," in Proc. 2nd Workshop Multimodal User Authentication, Toulouse, France [Online]. Available: http://eprints.qut.edu.au/5343/