The fusion of visual lip movements and mixed speech signals for robust speech separation

Information Fusion

Volumn 5, Issue 2, 2004, Pages 103-117

  Aarabi, Parham ^a   Mungamuru, Bob ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Audiovisual information fusion; Audiovisual signal separation; Blind speech separation; Independent component analysis; Robust speech recognition

Indexed keywords

ACOUSTIC SIGNAL PROCESSING; ALGORITHMS; CLASSIFICATION (OF INFORMATION); COMPUTER SIMULATION; CORRELATION METHODS; DATABASE SYSTEMS; ESTIMATION; FEATURE EXTRACTION; GAUSSIAN NOISE (ELECTRONIC); INDEPENDENT COMPONENT ANALYSIS; SIGNAL TO NOISE RATIO; VECTORS; VIDEO SIGNAL PROCESSING;

AUDIOVISUAL INFORMATION FUSION; BLIND SPEECH SEPARATION; ROBUST SPEECH RECOGNITION;

SPEECH PROCESSING;

EID: 1842854565     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2003.10.006     Document Type: Article

References (32)

1. Aarabi P., Zaky S. Multi-modal sound localization using audiovisual information Fusion. Information Fusion. 3(2):2001;209-223.
2. P. Aarabi, Multi-sense artificial awareness, M.A.Sc. Thesis, Department of Electrical and Computer Engineering, University of Toronto, June 1999.
3. P. Aarabi, S. Zaky, Integrated vision and sound localization, in: Proceedings of the 3rd International Conference on Information Fusion, Paris, France, July 2000.
4. Grant K.W., Ardell L.H., Kuhl P.K., Sparks D.W. The contribution of fundamental frequency, amplitude envelope, and voicing duration cues to speechreading in normal-hearing subjects. Journal of the Acoustical Society of America. 77(2):1985;671-677.
5. Luettin J., Thacker N.A. Speechreading using probabilistic models. Computer Vision and Image Understanding. 65(2):1997;163-178.
6. J. Luettin, N.A. Thacker, S.W. Beet, Visual speech recognition using active shape models and hidden Markov models, in: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Atlanta, GA, May 1996.
7. J. Luettin, S. Dupont, Continuous audio-visual speech recognition, in: Proceedings of the 5th European Conference on Computer Vision, 1998.
8. C. Neti, P. deCuetos, A. Senior, Audio-visual intent-to-speak detection for human-computer interaction, in: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Istanbul, Turkey, 5-9 June 2000.
9. Rogozan A. Discriminative learning of visual data for audiovisual speech recognition. International Journal on Artificial Intelligence Tools. 8(1):1999;43-52.
10. Rogozan A., Deléglise P. Adaptive fusion of acoustic and visual sources for automatic speech recognition. Speech Communication. 26(1-2):1998;149-161.
11. Movellan J.R. Visual speech recognition with stochastic networks. Tesauro G., Toruetzky D., Leen T. Advances in Neural Information Processing Systems. vol. 7:1995;MIT Press, Cambridge.
12. J. Baldwin, T. Martin, M. Saeed, Automatic computer lip-reading using fuzzy set theory, in: Proceedings of AVSP 99, Santa Cruz, CA, 1999.
13. S. Basu, N. Oliver, A. Pentland, 3D modeling and tracking of human lip motion, in: IEEE International Conference on Computer Vision, 1998.
14. Coianiz T., Torresani L., Capril B. 2D deformable models for visual speech analysis. Stork D.G., Hennecke M.E. Speechreading by Humans and Machines. NATO ASI Series, Series F: Computer and Systems Sciences. vol. 150:1996;391-398 Springer Verlag, Berlin.
15. Ramos Sanchez M.U., Matas J., Kittler J. Statistical chromaticity models for lip tracking with B-splines. Proceedings of the First International Conference on Audio- and Video-based Biometric Person Authentication. Lecture Notes in Computer Science. 1997;69-76 Springer Verlag.
16. Erber N.P., De Flippo C.L. Voice-mouth synthesis of tactual/visual perception of /pa, ba, ma/. Journal of the Acoustical Society of America. 64:1978;1015-1019.
17. Green K.P., Miller J.L. On the role of visual rate information in phonetic perception. Perception and Psychophysics. 38(3):1985;269-276.
18. E.D. Petajan, Automatic lipreading to enhance speech recognition, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1985, pp. 40-47.
19. Silsbee P.L., Bovik A.C. Computer lipreading for improved accuracy in automatic speech recognition. IEEE Transactions on Speech and Audio Processing. 4(5):1996;337-351.
20. C. Breglar, S.M. Omohundro, Nonlinear manifold learning for visual speech recognition, in: IEEE International Conference on Computer Vision, 1995, pp. 494-499.
21. M.J. Tomlinson, M.J. Russell, N.M. Brooke, Integrating audio and visual information to provide highly robust speech recognition, in: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, 1996, pp. 821-824.
22. Yuhas B.P., Goldstein M.H., Sejnowski T.J., Jenkins R.E. Neural network models of sensory integration for improved vowel recognition. Proceedings of the IEEE. 78(10):1990;1658-1668.
23. Mase K., Pentland A. Automatic lipreading by optical flow analysis. Systems and Computers in Japan. 22(6):1991.
24. M.S. Brandstein, H. Silverman, A robust method for speech signal time-delay estimation in reverberant rooms, in: Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, Atlanta, Georgia, May 1996.
25. Bell A., Sejnowski T. An information-maximization approach to blind separation and blind deconvolution. Neural Computation. 7(7):1995;1129-1159.
26. K. Torkkola, Blind separation of delayed sources based on information maximization, ICASSP, May 1996.
27. S. Amari, A. Cichocki, H. Yang, A new learning algorithm for blind signal separation, Advances in Neural Information Processing Systems, vol. 8, 1996.
28. Lee T.W., Bell A.J., Orglmeister R. Blind source separation of real world signals. Proc. ICNN. 1997.
29. P. Aarabi, N.H. Khameneh, Robust speech separation using visually constructed speech signals, in: Proceedings of Sensor Fusion: Architectures, Algorithms, and Applications VI (AeroSense'01), Orlando, FL, April 2002.
30. T. Darrell, J. Fisher, P. Viola, B. Freeman, Audio-visual segmentation and the cocktail party effect, in: Proceedings of the International Conference on Multimodal Interfaces, Beijing, October 2000.
31. P. Aarabi, Genetic sensor selection enhanced independent component analysis and its applications to robust speech recognition, in: Proceedings of the 5th IEEE Workshop on Nonlinear Signal and Image Processing (NSIP '01), Baltimore, MD, June 2001.
32. R.-W. Liu, Hui Luo, Direct blind separation of independent non-Gaussian signals with dynamic channels, in: Proceedings of 5th IEEE International Workshop on Cellular Neural Networks and their Applications, London, England, 1998, pp. 34-38.