Developing an audio-visual speech source separation algorithm

Speech Communication

Volumn 44, Issue 1-4 SPEC. ISS., 2004, Pages 113-125

  Sodoyer, David ^a   Girin, Laurent ^a   Jutten, Christian ^a   Schwartz, Jean Luc ^a  

^a GRENOBLE INSTITUTE OF TECHNOLOGY   (France)

Author keywords

Audio visual coherence; Audio visual joint probability; Blind source separation; Spectral information; Speech enhancement

Indexed keywords

AUDIO-VISUAL COHERENCE; AUDIO-VISUAL JOINT PROBABILITY; SPEECH ENHANCEMENT;

ACOUSTIC NOISE; ALGORITHMS; BLIND SOURCE SEPARATION; PROBABILITY; SENSOR DATA FUSION; SIGNAL PROCESSING;

SPEECH PROCESSING;

EID: 10444247388     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2004.10.002     Document Type: Article

References (30)

1. Amari, S.-L., 1998. Natural gradient works efficiently in learning. Neural Comput. 10, 251-276.
2. Benoît, C., Lallouache, M.T., Mohamadi, T., Abry, C., 1992. A set of visual French visemes for visual speech synthesis. In: Bailly, G. et al. (Eds.), Talking Machines. Elsevier, Amsterdam, pp. 485-504.
3. Bernstein, L.E., Benoît, C., 1996. For speech perception by humans or machines, three senses are better than one. In: Proc. ICSLP'96, pp. 1477-1480.
4. Bernstein, L.E., Takayanagi, S., Auer E.T., Jr., 2004. Enhanced auditory detection with AV speech: Perceptual evidence for speech and non-speech mechanisms. This volume.
5. Berthommier, F., 2003, Audiovisual speech enhancement based on the association between speech envelope and video feature. In: Proc. Eurospeech'03, Geneva, pp. 1045-1048.
6. Berthommier, F., 2004. A phonetically neutral model of the low-level audiovisual interaction. This volume.
7. Cardoso, J.F., Souloumiac, A., 1993. Blind beamforming for non-Gaussian signals. IEE Proc. - F 140, 362-370.
8. Cardoso, J.F., Laheld, B., 1996. Equivariant adaptative source separation. IEEE Trans. SP 44, 3017-3030.
9. Deligne, S., Potamianos, G., Neti, C., 2002. Audio-visual speech enhancement with AVCDCN (AudioVisual Code-book Dependent Cepstral Normalization). In: Proc. ICSLP'2002, pp. 1449-1452.
10. Dempster, A.P., Laird, N.M., Rubin, D.B., 1977. Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. 39, 1-38.
11. Girin, L., Feng, G., Schwartz, J.-L., 1997. Can the visual input make the audio signal pop out in noise? A first study of the enhancement of noisy VCV acoustic sequences by audiovisual fusion. In: Proc. AVSP'97, pp. 37-40.
12. Girin, L., Schwartz, J.L., Feng, G., 2001. Audio-visual enhancement of speech in noise. J. Acoust. Soc. Am. 109, 3007-3020.
13. Goecke, R., Potamianos, G., Neti, C., 2002. Noisy audio feature enhancement using audio-visual speech data. In: Proc. Internat. Conf, Acoust., Speech, Signal Process., pp. 2025-2028.
14. Grant, K.W., Seitz, P., 2000. The use of visible speech cues for improving auditory detection of spoken sentences. J. Acoust. Soc. Am. 108, 1197-1208.
15. Grant, K.W., van Wassenhove, V., Poeppel, D., 2004. Detection of auditory and auditory-visual synchrony. This volume.
16. Hyvärinen, A., 1999. Fast and robust fixed-point algorithms for Independent Component Analysis. IEEE Trans. Neural Networks 10, 626-634.
17. Jutten, C., Herault, J., 1991. Blind separation of sources. Part I: An adaptive algorithm based on a neuromimetic architecture. Signal Process. 24, 1-10.
18. Kim, J., Davis, C., 2001. Visible speech cues and auditory detection of spoken sentences: an effect of degree of correlation between acoustic and visual properties. In: Proc. AVSP'2001, pp. 127-131.
19. Kim, J., Davis, C., 2004. Testing the cuing hypothesis for the AV speech detection advantage. This volume.
20. Lallouache, M.T., 1990. Un poste 'visage-parole'. Acquisition et traitement de contours labiaux. In: Proc. XVIII JEPs, Montréal, pp. 282-286.
21. Nakadai, K., Matsuura, D., Okuno, H.G., Tsujino, H., 2004. Improvement of three simultaneous speech recognition by using AV integration and scattering theory for humanoid. This volume.
22. Okuno, H.G., Nakadai, K., Lourens, T., Kitano, H., 2001. Separating three simultaneous speeches with two microphones by integrating auditory and visual processing. In: Proc. Eurospeech 2001, pp. 2643-2646.
23. Petajan, E.D., 1984. Automatic Lipreading to Enhance Speech Recognition. Doct. Thesis, University of Illinois.
24. Schwartz, J.L., Berthommier, F., Savariaux, C., 2002. Audio-visual scene analysis: Evidence for a "very-early" integration process in audio-visual speech perception. In: Proc. ICSLP'2002, pp. 1937-1940.
25. Schwartz, J.L., Berthommier, F., Savariaux, C., 2004. Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition 93, B69-B78.
26. Sodoyer, D., Schwartz, J.L., Girin, L., Klinkisch, J., Jutten, C., 2002. Separation of audio-visual speech sources. Eurasip JASP 2002, 1164-1173.
27. Sodoyer, D., Girin, L., Jutten, C., Schwarte, J.L., 2003. Extracting an AV speech source from a mixture of signals. In: Proc. Eurospeech 2003, pp. 1393-1396.
28. Sumby, W.H., Pollack, I., 1954. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26, 212-215.
29. Taleb, A., Jutten, C., 1999. Source separation in postnonlinear mixtures. IEEE Trans. SP 10, 2807-2820.
30. Yehia, H., Rubin, P., Vatikiotis-Bateson, E., 1998. Quantitative association of vocal-tract and facial behavior. Speech Communication 26, 23-43.