Mixing audiovisual speech processing and blind source separation for the extraction of speech signals from convolutive mixtures

IEEE Transactions on Audio, Speech and Language Processing

Volumn 15, Issue 1, 2007, Pages 96-108

  Rivet, Bertrand ^a   Girin, Laurent ^a   Jutten, Christian ^a  

^a Ecole Nationale DElectronique et de Radiolectricit   (France)

Author keywords

Audiovisual coherence; Blind source separation; Convolutive mixture; Speech enhancement; Statistical modeling

Indexed keywords

ADDITIVE MIXTURES; AUDIO AND VISUAL INFORMATIONS; AUDIO-VISUAL SPEECH PROCESSING; AUDIOVISUAL COHERENCE; BLIND SEPARATIONS; COMPLEX MIXTURES; CONVOLUTIVE MIXTURE; EXTRACTION SYSTEMS; FREQUENCY CHANNELS; FREQUENCY SEPARATIONS; NOISY ENVIRONMENTS; NOVEL ALGORITHMS; SCALE FACTOR AMBIGUITIES; SPEECH SIGNALS; STANDARD SOURCES; STATISTICAL MODELING; STATISTICAL MODELS; STATISTICAL TOOLS; VISUAL SIGNALS;

APARTMENT HOUSES; MIXTURES; SEPARATION; SIGNAL ANALYSIS; SPEECH ANALYSIS; SPEECH ENHANCEMENT; VISUAL COMMUNICATION;

BLIND SOURCE SEPARATION;

EID: 34447100075     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2006.872619     Document Type: Article

References (50)

1. L. E. Bernstein and C. Benoît, "For speech perception by humans or machines, three senses are better than one," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), 1996, pp. 1477-1480.
2. W. Sumby and I. Pollack, "Visual contribution to speech intelligibility in noise," J. Acoust. Soc. Amer., vol. 26, pp. 212-215, 1954.
3. Q. Summerfield, "Some preliminaries to a comprehensive account of audio-visual speech perception," in Hearing by Eye: The Psychology of Lipreading, B. Dodd and R. Campbell, Eds. Mahwah, NJ: Lawrence Erlbaum, 1987, pp. 3-51.
4. J. Robert-Ribes, J.-L. Schwartz, T. Lallouache, and P. Escudier, "Complementarity and synergy in bimodal speech: Auditory, visual, and audio-visual identification of French oral vowels in noise," J. Acoust. Soc. Amer., vol. 103, no. 6, pp. 3677-3689, 1998.
5. E. D. Petajan, "Automatic lipreading to enhance speech recognition," Ph.D. dissertation, Univ. Illinois, Urbana, 1984.
6. G. Potamianos, C. Neti, G. Gravier,A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audio-visual speech," Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, Sep. 2003.
7. K. Grant and P. Seitz, "The use of visible speech cues for improving auditory detection of spoken sentences," J. Acoust. Soc. Amer., vol. 108, pp. 1197-1208, 2000.
8. J. Kim and D. Chris, "Investigating the audio-visual speech detection advantage," Speech Commun., vol. 44, no. 1-4, pp. 19-30, 2004.
9. L. E. Bernstein, E. T. J. Auer, and S. Takayanagi, "Auditory speech detection in noise enhanced by lipreading," Speech Commun., vol. 44, no. 1-4, pp. 5-18, 2004.
10. J.-L. Schwartz, F. Berthommier, and C. Savariaux, "Audio-visual scene analysis; evidence for a "very-early" integration process in audio-visual speech perception," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), 2002, pp. 1937-1940.
11. L. Girin, J.-L. Schwartz, and G. Feng, "Audio-visual enhancement of speech in noise," J. Acoust. Soc. Amer., vol. 109, no. 6, pp. 3007-3020, Jun. 2001.
12. S. Deligne, G. Potamianos, and C. Neti, "Audio-visual speech enhancement with AVCDCN (AudioVisual Codebook Dependent Cepstral Normalization)," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), 2002, pp. 1449-1452.
13. R. Goecke, G. Potamianos, and C. Neti, "Noisy audio feature enhancement using audio-visual speech data," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Process. (ICASSP), Orlando, FL, May 2002, pp. 2025-2028.
14. L. Girin, A. Allard, and J.-L. Schwartz, "Speech signals separation: A new approach exploiting the coherence of audio and visual speech," in IEEE Int. Workshop Multimedia Signal Process. (MMSP), Cannes, France, 2001.
15. D. Sodoyer, J.-L. Schwartz, L. Girin, J. Klinkisch, and C. Jutten, "Separation of audio-visual speech sources: a new approach exploiting the audiovisual coherence of speech stimuli," EURASIP J. Appl. Signal Process., vol. 2002, no. 11, pp. 1165-1173, 2002.
16. D. Sodoyer, L. Girin, C. Jutten, and J.-L. Schwartz, "Developing an audio-visual speech source separation algorithm," Speech Commun., vol. 44, no. 1-4, pp. 113-125, Oct. 2004.
17. J.-F. Cardoso, "Blind signal separation: statistical principles," Proc. IEEE, vol. 86, no. 10, pp. 2009-2025, Oct. 1998.
18. C. Jutten and A. Taleb, "Source separation: from dusk till dawn," in Proc. Int. Conf. Independent Compon. Anal. Blind Source Separation (ICA), Helsinki, Finland, Jun. 2000, pp. 15-26.
19. A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. New York: Wiley, 2001.
20. P. Comon, "Independent component analysis, a new concept?," Signal Process., vol. 36, no. 3, pp. 287-314, Apr. 1994.
21. J. Hérault, C. Jutten, and B. Ans, "Détection de grandeurs primitives dans un message composite par une architecture de calcul neuromimétrique en apprentissage non supervisé," in Proc. GRETSI, Nice, France, May 1985, vol. 2, pp. 1017-1020.
22. C. Jutten and J. Hérault, "Blind separation of sources. Part I: An adaptive algorithm based on a neuromimetic architecture," Signal Process., vol. 24, no. 1, pp. 1-10, Jul. 1991.
23. J.-F. Cardoso and A. Souloumiac, "Blind beamforming for non Gaussian signals," Proc. Inst. Elect. Eng. F, vol. 140, no. 6, pp. 362-370, Dec. 1993.
24. A. Hyvarinen, "Fast and robust fixed-point algorithms for independent component analysis," IEEE Trans. Neural Netw., vol. 10, no. 3, pp. 626-634, May 1999.
25. A. Bell and T. Sejnowski, "An information-maximization approach to blind source separation and blind deconvolution," Neural Comput., vol. 7, pp. 1129-1159, 1995.
26. L. De Lathauwer, D. Callaerts, B. De Moor, and J. Vandewalle, "Fetal electrocardiogram extraction by source subspace separation," in Proc. IEEE Workshop HOS, Girona, Spain, Jun. 12-14, 1995, pp. 134-138.
27. V. Zarzoso and A. K. Nandi, "Noninvasive fetal electrocardiogram extraction: Blind source separation versus adaptative noise cancellation," IEEE Trans. Biomed. Eng., vol. 48, no. 1, pp. 12-18, Jan. 2001.
28. R. Dansereau, "Co-channel audiovisual speech separation using spectral matching constraints," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Montréal, QC, Canada, 2004.
29. S. Rajaram, A. V. Nefian, and T. S. Huang, "Bayesian separation of audio-visual speech sources," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Montréal, Canada, 2004, pp. 645-648.
30. V. Capdevielle, C. Servière, and J.-L. Lacoume, "Blind separation of wide-band sources in the frequency domain," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Detroit, MI, May 1995, pp. 2080-2083.
31. L. Para and C. Spence, "Convolutive blind separation of non stationary sources," IEEE Trans. Speech Audio Process., vol. 8, no. 3, pp. 320-327, May 2000.
32. A. Dapena, M. F. Bugallo, and L. Castedo, "Separation of convolutive mixtures of temporally-white signals: A novel frequency-domain approach," in Proc. Int. Conf. Independent Compon. Anal. Blind Source Separation (ICA), San Diego, CA, Dec. 2001, pp. 315-320.
33. D.-T. Pham, C. Servière, and H. Boumaraf, "Blind separation of convolutive audio mixtures using nonstationary," in Proc. Int. Conf. Independent Compon. Anal. Blind Source Separation (ICA), Nara, Japan, Apr. 2003, pp. 981-986.
34. B. Rivet, L. Girin, C. Jutten, and J.-L. Schwartz, "Using audiovisual speech processing to improve the robustness of the separation of convolutive speech mixtures," in IEEE Int. Workshop Multimedia Signal Process. (MMSP), Sienna, Italy, Oct. 2004, pp. 47-50.
35. B. Rivet, L. Girin, and C. Jutten, "Solving the indeterminations of blind source separation of convolutive speech mixtures," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Philadelphia, PA, Mar. 2005, pp. 533-536.
36. -, "Log-Rayleigh distribution:Asimple and efficient statistical representation of log-spectral coefficients," IEEE Trans. Audio, Speech, Lang. Process., 2006, submitted for publication.
37. H.-L. Nguyen-Thi and C. Jutten, "Blind source separation for convolutive mixtures," Signal Process., vol. 45, pp. 209-229, 1995.
38. D.-T. Pham, "Joint approximate diagonalization of positive definite matrices," SIAM J. Matrix Anal. Appl., vol. 22, no. 4, pp. 1136-1152, 2001.
39. N. Murata, S. Ikeda, and A. Ziehe, "An approach to blind source separation based on temporal structure of speech signals," Neurocomput., vol. 41, no. 1-4, pp. 1-24, Oct. 2001.
40. B. Le Goff, T. Guiard-Marigny, and C. Benoît, "Read my lips. . . and my jaw! How intelligible are the components of a speaker's face?," in Proc. Euro. Conf. Speech Communication Technology, Madrid, Spain, 1995, pp. 291-294.
41. -, "Analysis-synthesis and intelligibility of a talking face," in Progress in Speech SynthesisJ. Van Santen, R. Sproat, J. Olive, and J. Hirschberg, Eds. New York: Springer-Verlag, 1996, pp. 235-244.
42. F. Elisei, M. Odisio, G. Bailly, and P. Badin, "Creating and controlling video-realistic talking heads," in Proc. Audio-Visual Speech Processing Workshop (AVSP), Aalborg, Denmark, 2001, pp. 90-97.
43. T. Lallouache, "Un poste visage-parole. Acquisition et traitement des contours labiaux," in Proc. Journées d'Etude sur la Parole (JEP) (in French), Montréal, QC, Canada, 1990, pp. 282-286.
44. B. Picinbono, "Second-order complex random vectors and normal distributions," IEEE Trans. Signal Process., vol. 44, no. 10, pp. 2637-2640, Oct. 1996.
45. F. D. Neeser and J. L. Massey, "Proper complex random processes with applications to information theory," IEEE Trans. Inf. Theory, vol. 39, no. 4, pp. 1293-1302, Jul. 1993.
46. L. Benaroya, "Séparation de plusieurs sources sonores avec un seul microphone," Ph.D. dissertation, Traitement du signal, Univ. Rennes 1, Rennes, France, Jun. 2003.
47. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative association of vocal-tract and facial behavior," Speech Commun., vol. 26, no. 1, pp. 23-43, 1998.
48. D. Ormoneit and V. Tresp, "Averaging, maximum penalized likelihood and Bayesian estimation for improving Gaussian mixture probability density estimates," IEEE Trans. Neural Netw., vol. 9, no. 4, pp. 639-650, Jul. 1998.
49. H. Snoussi and A. Mohammad-Djafari, "Penalized maximum likelihood for multivariate Gaussian mixture," in Proc. Bayesian Inference and Maximum Entropy Methods, R. L. Fry, Ed. MaxEnt Workshops, Aug. 2001, pp. 36-46.
50. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc. Ser. B., vol. 39, pp. 1-38, 1977.