Integration strategies for audio-visual speech processing: Applied to text-dependent speaker recognition

IEEE Transactions on Multimedia

Volumn 7, Issue 3, 2005, Pages 495-506

  Lucey, Simon ^a,b   Chen, Tsuhan ^b   Sridharan, Sridha ^a   Chandran, Vinod ^a  

^a QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Audio visual speech processing (AVSP); Classifier combination; Integration strategies; Multistream hidden Markov model (HMM); Speaker recognition

Indexed keywords

ERROR ANALYSIS; GRAPH THEORY; INTEGRATION; MARKOV PROCESSES; MATHEMATICAL MODELS; SPEECH PROCESSING;

AUDIO-VISUAL SPEECH PROCESSING (AVSP); CLASSIFIER COMBINATION; INTEGRATION STRATEGIES; MULTISTREAM HIDDEN MARKOV MODELS (HMM); SPEAKER RECOGNITION;

SPEECH RECOGNITION;

EID: 20444375102     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2005.846777     Document Type: Article

References (31)

1. T. Wark, S. Sridharan, and V. Chandran, "Robust speaker verification via fusion of speech and lip modalities," in Proc. Int. Conf. Acoustics, Speech and Signal Processing (ICASSP'99), vol. 6, 1999, pp. 3061-3064.
2. T. Chen and R. Rao, "Audio-visual integration in multimodal communication," Proc. IEEE, vol. 86, no. 5, pp. 837-852, May 1998.
3. F. Lavagetto, "Converting speech into lip movements: A multimedia telephone for hard hearing people," IEEE Trans. Rehab. Eng., vol. 3, no. 1, pp. 90-102, Mar. 1995.
4. H. McGurk and J. MacDonald, "Hearing lips and seeing voices," Nature, pp. 746-748, Dec. 1976.
5. Speechreading by Humans and Machines, vol. 150, NATO ASI Series F: Computer and Systems Sciences, D. G. Stork and M. E. Hennecke, Eds., Springer-Verlag, New York, 1996.
6. C. C. Chibelushi, F. Deravi, and J. S. D. Mason, "A review of speech-based bimodal recognition," IEEE Trans. Multimedia, vol. 4, no. 1, pp. 23-37, Mar. 2002.
7. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, Sep. 2000.
8. G. Potamianos and H. P. Graf, "Discriminative training of HMM stream exponents for audio-visual speech recognition," in Proc. Int. Conf. Acoustics, Speech and Signal Processing (ICASSP'98), vol. 6, 1998, pp. 3733-3736.
9. A. P. Varga and R. K. Moore, "Hidden Markov model decomposition of speech and noise," in Proc. Int. Conf. Acoustics, Speech and Signal Processing (ICASSP'90), vol. 2, 1990, pp. 845-848.
10. J. Kittler, M. Hatef, R. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Anal. Machine Intell., vol. 20, no. 3, pp. 226-239, Mar. 1998.
11. K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed. ork: Academic, 1990.
12. J. Kittler, "Combining classifiers: A theoretical framework," Pattern Anal. and Applicat., vol. 1, no. 1, pp. 18-27, 1998.
13. J. R. Movellan and P. Mineiro, "Modularity and Catastrophic Fusion: A Bayesian Approach with Applications to Audio-Visual Speech Recognition," USCD, Dept. Cognitive Sci., San Diego, CA, Tech. Rep. 97.01, 1997.
14. D. Mansour and B. H. Juang, "A family of distortion measures based upon projection operation for robust speech recognition," IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 11, pp. 1659-1671, Nov. 1989.
15. M. S. Kamel and N. M. Wanas, "Data dependence in combining classifiers," in Multiple Classifier Systems, T. Windeatt and F. Roli, Eds., 2003, pp. 1-14.
16. A. Adjoudani and C. Benoit, "Audio-visual speech recognition compared across two architectures," in Proc. European Conf. Speech Communication and Technology (Eurospeech'95), 1995, pp. 1563-1566.
17. S. Cox, I. Matthews, and J. A. Bangham, "Combining noise compensation with visual information in speech recognition," in Auditory-Visual Speech Processing (AVSP'97), Rhodes, Greece, 1997.
18. C. C. Chibelushi, J. S. Mason, and F. Deravi, "Integration of acoustic and visual speech for speaker recognition," in Proc. European Conf. Speech Communication and Technology (Eurospeech'93), 1993, pp. 157-160.
19. M. McGrath and Q. Summerfield, "Intermodal timing relations and audio-visual speech recognition," J. Acoust. Soc. Amer., vol. 77, no. 2, pp. 678-685, Feb. 1985.
20. J. Luettin, G. Potamianos, and C. Neti, "Asynchronous stream modeling for large vocabulary audio-visual speech recognition," in Proc. Int. Conf. Acoustics, Speech and Signal Processing (ICASSP'01), vol. 1, 2001, pp. 169-172.
21. A. Martin, G. Doddington, T. Kamm, M. Ordowski, and P. Przybocki, "The DET curve in assessment of detection task performance," in Proc. European Conf. Speech Communication and Technology (Eurospeech'97), vol. 4, 1997, pp. 1895-1898.
22. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed, New York: Wiley, 2001.
23. S. Pigeon and L. Vandendorpe, "The M2VTS multimodal face database," in Proc. Int. Conf. Audio and Video-based Biometric Person Authentication (AVBPA'97), Crans-Montana, Switzerland, Mar. 1997.
24. P. Jourlin, J. Luettin, D. Genoud, and H. Wassner, "Acoustic-labial speaker verification," Pattern Recognit. Lett., vol. 18:9, pp. 853-858, 1997.
25. S. Lucey, V. Chandran, and S. Sridharan, "Improved facial-feature detection for AVSP via unsupervised clustering and discriminant analysis," EURASIP J. Appl. Signal Process., no. 3, pp. 264-275, 2003.
26. S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland, The HTK Book (for HTK Version 2.2). Cambridge, U.K.: Entropic Ltd., 1999.
27. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
28. M. J. Tomlinson, M. J. Russell, and N. M. Brooke, "Integrating audio and visual information to provide highly robust speech recognition," in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP '96), 1996, pp. 821-824.
29. S. Lucey, V. Chandran, and S. Sridharan, "A link between cepstral shrinking and the weighted product rule in audio-visual speech recognition," in Proc. Int. Conf. Spoken Language Processing (ICSLP'02), 2002, pp. 1961-1964.
30. S. Lucey, S. Sridharan, and V. Chandran, "An investigation of HMM classifier combination strategies for improved audio-visual speech recognition," in Proc. European Conf. Speech Communication and Technology (Eurospeech'01), 2001, pp. 1185-1188.
31. K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, "XM2VTSDB: The extended M2VTS database," in Proc. Int. Conf. Audio and Video-Based Biometric Person Authentication (AVBPA'99), 1999, pp. 72-77.