Robust audio-visual speech recognition based on late integration

IEEE Transactions on Multimedia

Volumn 10, Issue 5, 2008, Pages 767-779

  Lee, Jong Seok ^a,b   Park, Cheol Hoon ^a,b  

^a IEEE

^b Korea Advanced Institute of Science and Technology (KAIST)   (South Korea)

Author keywords

Audio visual speech recognition; Hidden Markov model; Interframe correlation; Late integration; Neural network; Robustness; Stochastic optimization

Indexed keywords

ACOUSTICS; ARTIFICIAL INTELLIGENCE; AUDIO ACOUSTICS; HIDDEN MARKOV MODELS; MARKOV PROCESSES; NEURAL NETWORKS; SPEECH; SPEECH ANALYSIS; STOCHASTIC MODELS;

AUDIO VISUAL SPEECH RECOGNITION (AVSR); CONVENTIONAL SYSTEMS; DYNAMIC CHARACTERISTICS; INTEGRATION SCHEMES; ISOLATED WORD RECOGNITION; MARKOV MODELLING; NOISE CONDITIONS; NOISY ENVIRONMENTS; PRIORI KNOWLEDGE; ROBUST RECOGNITION; SPEECH RECOGNIZERS; STOCHASTIC OPTIMIZATIONS; VISUAL SIGNALS;

SPEECH RECOGNITION;

EID: 47649103796     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2008.922789     Document Type: Article

References (41)

1. E. D. Petajan, "Automatic lipreading to enhance speech recognition," in Proc. Global Telecommunications Conf., Atlanta, GA, Nov. 1984, pp. 265-272.
2. C. C. Chibelushi, F. Deravi, and J. S. D. Mason, "A review of speechbased bimodal recognition," IEEE Trans. Multimedia, vol. 4, no. 1, pp. 23-37, Mar. 2002.
3. B. V. Dasarathy, "Sensor fusion potential exploitation: Innovative archi-tectures and illustrative applications," Proc. IEEE, vol. 85, pp. 24-38, Jan. 1997.
4. D. W. Massaro, Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry. Hillsdale, NJ: Lawrence Erlbaum, 1987.
5. J.-S. Lee and C. H. Park, "Training hidden Markov models by hybrid simulated annealing for visual speech recognition," in Proc. IEEE Int. Conf. Systems, Man, Cybernetics, Taipei, Taiwan, R.O.C., Oct. 2006, pp. 198-202.
6. X. Huang, A. Acero, and H.-W. Hon, Spoken Language Processing: A Guide to Theory, Algorithm, and System Development., Upper Saddle River, NJ: Prentice Hall, 2001.
7. R. Drullman, J. M. Festen, and R. Plomp, "Effect of temporal envelope smearing on speech reception," J. Acoust. Soc. Amer., vol. 95, no. 2, pp. 1053-1064, Feb. 1994.
8. T. Arai and S. Greenberg, "Speech intelligibility in the presence of cross-channel spectral asynchrony," in Proc. ICASSP, Seattle, WA, 1998, vol. 2, pp. 933-936.
9. S. Furai, "Speaker-independent isolated word recognition using dynamic features of speech spectrum," IEEE Trans. Acoust., Speech, Signal Process., vol. 34, no. 1, pp. 52-59, Feb. 1986.
10. H. Hermansky and N. Morgan. "RASTA processing of speech," IEEE Trans. Speech Audio Processing, vol. 2. no. 4, pp. 578-589, 1994.
11. C. Bartels and J. Bilmes. "Focused state transition information in ASR," in Proc. Workshop on Automatic Speech Recognition and Understanding, San Juan, PR, Nov. 2005, pp. 191-196.
12. H. H. Szu and R. L. Hartley, "Fast simulated annealing," Phys. Lett. A, vol. 122, no. 3-4, pp. 157-162, June 1987.
13. D. Paul, "Training of HMM recognizers by simulated annealing," in Proc. ICASSP, Tampa, FL, Mar. 1985, pp. 13-16.
14. S. R. Eddy, "Multiple alignment using hidden Markov models," in Proc. Int. Conf. Intelligent Systems for Molecular Biology, Menlo Park, CA, 1995, pp. 114-120.
15. D. Nam, J.-S. Lee, and C. H. Park, "n-dimensional Cauchy neighbor generation for the fast simulated annealing," IEICE Trans. Inf. Syst., vol. E87-D, no. 11, pp. 2499-2502, Nov. 2004.
16. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, "Equation of state calculations by fast computing machines," J. Chem. Phys., vol. 21, no. 6, pp. 1087-1092, 1953.
17. J.-S. Lee, "Audio-Visual Speech Recognition: Stochastic Optimization of Hidden Markov Models, Modeling of Interframe Correlations and Integration With Neural Networks," Ph.D. dissertation, Dept. Elect. Eng. Comput. Science, KAIST, Daejeon, Korea, 2006.
18. D. V. Ouellette, "Schur complements and statistics," Linear Algebra Appl., vol. 36, pp. 187-295, Mar. 1981.
19. A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd ed. New York: McGraw-Hill, 1991.
20. T. H. Applebaum and B. A. Hanson, "Regression features for recognition of speech in quiet and in noise," in Proc. ICASSP, Toronto, ON, Canada, Apr. 1991, vol. 2, pp. 985-988.
21. Optimization Toolbox User's Guide. Natick, MA: The Mathworks, Inc., 2005, The Mathworks.
22. A. D. Belegundu and T. R. Chandrupatla, Optimization Concepts and Applications in Engineering. Upper Saddle River, NJ: Prentice-Hall, 1999.
23. L. A. Ross, D. Saint-Amour, V. M. Leavitt, D. C. Javitt, and J. J. Foxe, "Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments," Cerebral Cortex vol. 17, no. 5, pp. 1147-1153, 2007.
24. P. Arnold and F. Hill, "Bisensory augmentation: A speechreading advantage when speech is clearly audible and intact," Brit. J. Psychol., vol. 92, pp. 339-355, 2001.
25. C. Benoît, , M. M. Taylor, F. Nel, and D. Bouwhuis, Eds., "The intrinsic bimodality of speech communication and the synthesis of talking faces," in The Structure of Multimodal Dialogue II. Amsterdam, The Netherlands: John Benjamins, 2000, pp. 485-502.
26. B. Conrey and D. B. Pisoni, "Auditory-visual speech perception and synchrony detection for speech and nonspeech signals," J. Acoust. Soc. Amer., vol. 119, no. 6, pp. 4065-4073, June 2006.
27. M. Heckmann, F. Berthommier, and K. Kroschel, "Noise adaptive stream weighting in audio-visual speech recognition," EURASIP J. Appl. Signal Process., vol. 11, pp. 1260-1273, 2002.
28. E. Marcheret, V. Libal, and G. Potamianos, "Dynamic stream weight modeling for audio-visual speech recognition," in Proc. ICASSP, Honolulu, HI, Apr. 2007, vol. 4, pp. 945-948.
29. A. Rogozan and P. Deléglise, "Adaptive fusion of acoustic and visual sources for automatic speech recognition," Speech Commun., vol. 26, no. 1-2, pp. 149-161, Oct. 1998.
30. T. W. Lewis and D. M. W. Powers, "Sensor fusion weighting measures in audio-visual speech recognition," in Proc. 27th Conf. Australasian Computer Science, Dunedin, New Zealand, 2004, pp. 305-314.
31. T. J. Hazen, "Visual model structures and synchrony constraints for audio-visual speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 3, pp. 1082-1089, May 2006.
32. A. Verma, T. Faruquie, C. Neti, and S. Basu, "Late integration in audiovisual continuous speech recognition," in Proc.Workshop on Automatic Speech Recognition and Understanding, Keystone, CO, Dec. 1999, pp. 71-74.
33. G. F. Meyer, J. B. Mulligan, and S. M. Wuerger, "Continuous audiovisual digit recognition using N-best decision fusion," Inform. Fusion, vol. 5, no. 2, pp. 91-101, June 2004.
34. S. Tamura, K. Iwano, and S. Furui, "A stream-weight optimization method for multi-stream HMMs based on likelihood value normalization," in Proc. ICASSP, Philadelphia, PA, Mar. 2005, vol. 1, pp. 469-472.
35. A. Adjoudani and C. Benoǐt, , D. G. Stork and M. E. Hennecke, Eds., "On the integration of auditory and visual parameters in an HMM-based ASR," in Speechreading by Humans and Machines: Models, Systems and Applications, ser. NATO ASI Series. Berlin, Germany: Springer, 1996, pp. 461-472.
36. C. M. Bishop, Neural Networks for Pattern Recognition. New York: Oxford Univ. Press, 1995.
37. S. Haykin, Neural Networks: A Comprehensive Foundation. Upper Saddle River, NJ: Prentice-Hall, 1999.
38. A. Varga and H. J. M. Steeneken, "Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems," Speech Commun., vol. 12, no. 3, pp. 247-251, 1993.
39. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, Sep. 2000.
40. M. H. Coen, "Multimodal integration - A biological view," in Proc. Int. Joint Conf. Artificial Intelligence, Seattle, WA, 2001, pp. 1417-1424.
41. D. Li, N. Dimitrova, M. Li, and I. K. Sethi, "Multimedia content processing through cross-modal association," in Proc. ACM Int. Conf. Multimedia, Berkeley, CA, Nov. 2003, pp. 604-611.