Statistical lip-appearance models trained automatically using audio information

Eurasip Journal on Applied Signal Processing

Volumn 2002, Issue 11, 2002, Pages 1202-1212

  Daubias, Philippe ^a,b   Deléglise, Paul ^a  

^a UNIVERSITÉ DU MAINE   (France)

^b Lab d'Info Gr Image Modelisation   (France)

Author keywords

Artificial neural networks; Audio visual corpora; Automatic lip region labeling; Dynamic time warping; Lip appearance model; Lip shape model

Indexed keywords

ACOUSTIC SIGNAL PROCESSING; AUDIO ACOUSTICS; FEATURE EXTRACTION; IMAGE PROCESSING; NEURAL NETWORKS; SPEECH RECOGNITION; VIDEO SIGNAL PROCESSING;

AUDIO-VISUAL SIGNALS; AUTOMATIC SPEECH RECOGNITION (ASR);

SPEECH PROCESSING;

EID: 0036875015     PISSN: 11108657     EISSN: None     Source Type: Journal    
DOI: 10.1155/S1110865702206186     Document Type: Article

References (36)

1. A. Morris, A. Hagen, H. Glotin, and H. Bourlard, "Multistream adaptive evidence combination for noise robust ASR," Speech Communication Journal, vol. 34, no. 1-2, pp. 25-40, 2001.
2. I. Zitouni, K. Smaïli, and J.-P. Haton, "Beyond the conventional statistical language models: the variable-length sequences approach," in Proc. 6th International Conference on Spoken Language Processing (ICSLP), vol. 3, pp. 962-965, Beijing, China, October 2000.
3. H. McGurk and J. McDonald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, December 1976.
4. C. Bregler, H. Hild, S. Manke, and A. Waibel, "Improving connected letter recognition by lipreading," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 1, pp. 557-560, Minneapolis, Minn, USA, April 1993.
5. P. Duchnowski, M. Hunke, D. Büsching, U. Meier, and A. Waibel, "Toward movement-invariant automatic lip-reading and speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 1, pp. 109-112, Detroit, Mich, USA, May 1995.
6. R. Kaucic, B. Dalton, and A. Blake, "Real-time lip tracking for audio-visual speech recognition applications," in Proc. 4th European Conference on Computer Vision (ECCV), vol. 2, pp. 376-387, Cambridge, UK, April 1996.
7. J. Luettin, N. A. Thacker, and S. W. Beet, "Active shape models for visual speech feature extraction," in Speechreading by Humans and Machines: Models, Systems, and Applications, D. G. Stork and M. E. Hennecke, Eds., vol. 150 of NATO Advanced Science Institutes, pp. 383-390, Springer-Verlag, New York, NY, USA, 1996.
8. C. Neti, G. Potamianos, J. Luettin, et al., "Audio-visual speech recognition," Tech. Rep. Workshop 2000, Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, Md, USA, October 2000.
9. A. Rogozan and P. Deléglise, "Adaptive fusion of acoustic and visual sources for automatic speech recognition," Speech Communication Journal, vol. 26, no. 1-2, pp. 149-161, 1998.
10. R. Auckenthaler, J. Brand, J. S. D. Mason, F. Deravi, and C. C. Chibelushi, "Lip signatures for automatic person recognition," in Proc. 2nd International Conference on Audio- and Video-Based Biometric Person Authentication (AVBPA), pp. 142-147, Washington, DC, USA, March 1999.
11. P. Jourlin, J. Luettin, D. Genoud, and H. Wassner, "Acoustic-labial speaker verification," in Proc. 1st International Conference on Audio- and Video-Based Biometric Person Authentification (AVBPA), J. Bigün, G. Chollet, and G. Borgefors, Eds., pp. 319-326, Springer-Verlag, Crans-Montana, Switzerland, March 1997.
12. L. Girin, L. Varin, G. Feng, and J.-L. Schwartz, "A signal processing system for having the sound "pop-out" in noise thanks to the image of the speaker's lips: New advances using multilayer perceptrons," in Proc. 5th International Conference on Spoken Language Processing (ICSLP), vol. 4, pp. 1451-1454, Sydney, Australia, December 1998.
13. U. Meier, R. Stiefelhagen, J. Yang, and A. Waibel, "Towards unrestricted lip reading," in Proc. 2nd International Conference on Multimodal Interfaces (ICMI), Hong Kong, 1999.
14. M. Liévin and F. Luthon, "Unsupervised lip segmentation under natural conditions," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing (ICASSP), vol. 6, pp. 3065-3068, Phoenix, Ariz, USA, March 1999.
15. C. Benoît, T. Guiard-Marigny, B. Le Goff, and A. Adjoudani, "Which components of the face do humans and machines best speechread?," in Speechreading by Humans and Machines: Models, Systems, and Applications, D. G. Stork and M. E. Hennecke, Eds., vol. 150 of NATO Advanced Science Institutes, pp. 315-328, Springer-Verlag, New York, NY, USA, 1996.
16. C. Benoît, "On the production and perception of audio-visual speech by man and machine," in Multimedia Communications and Video Coding, Y. Wang, S. Panwar, S.-P. Kim, and H. L. Bertoni, Eds., Plenum, New York, NY, USA, October 1995.
17. G. Potamianos, H. P. Graf, and E. Cosatto, "An image transform approach for HMM based automatic lipreading," in Proc. IEEE International Conference on Image Processing (ICIP), vol. 3, pp. 173-177, Chicago, Ill, USA, October 1998.
18. M. E. Hennecke, K. V. Prasad, and D. G. Stork, "Using deformable templates to infer visual speech dynamics," in 28th Annual Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, Calif, USA, November 1994.
19. S. Horbelt and J.-L. Dugelay, "Active contours for lipreading: combining snakes with templates," in 15th GRETSI Symposium Signal and Image Processing, pp. 717-720, Juan-les-Pins, France, September 1995.
20. R. Kober, J. Schiffers, and K. Schmidt, "Model-based versus knowledge-guided representation of non-rigid objects: A case study," in Proc. IEEE International Conference on Image Processing, vol. 1, pp. 973-977, IEEE Computer Society Press, Los Alamitos, Calif, USA, 1994.
21. B. Leroy, I. L. Herlin, and L. D. Cohen, "Face identification by deformation measure," in Proc. IEEE International Conference on Pattern Recognition (ICPR), vol. 3, pp. 633-637, Vienna, Austria, August 1996.
22. K. F. Lai, C. W. Ngo, and S. Chan, "Tracking of deformable contours by synthesis and match," in Proc. IEEE International Conference on Pattern Recognition (ICPR), vol. 1, pp. 657-661, Vienna, Austria, August 1996.
23. L. Revéret and C. Benoît, "A new 3D lip model for analysis and synthesis of lip motion in speech production," in Proc. Auditory-Visual Speech Processing (AVSP), pp. 207-212, Terrigal, Australia, December 1998.
24. S. Basu, N. Oliver, and A. Pentland, "3D modeling and tracking of human lip motion," in Proc. IEEE International Conference on Computer Vision (ICCV), pp. 337-343, Bombay, India, January 1998.
25. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, 2000.
26. I. Matthews, T. Cootes, S. Cox, R. Harvey, and J. A. Bangham, "Lipreading using shape, shading and scale," in Proc. Auditory-Visual Speech Processing (AVSP), pp. 73-78, Terrigal, Australia, December 1998.
27. T. Coianiz, L. Torresani, and B. Caprile, "2D deformable models for visual speech analysis," in Speechreading by Humans and Machines: Models, Systems, and Applications, D. G. Stork and M. E. Hennecke, Eds., vol. 150 of NATO Advanced Science Institutes, pp. 391-398, Springer-Verlag, New York, NY, USA, 1996.
28. J. C. Wojdel and L. J. M. Rothkrantz, "Using aerial and geometric features in automatic lip-reading," in Proc. 7th European Conference on Speech Communication and Technology (Eurospeech), vol. 4, pp. 2463-2466, Aalborg, Denmark, September 2001.
29. X. Zhang, C. C. Broun, R. M. Mersereau, and M. Clements, "Automatic speechreading with applications to human-computer interfaces," EURASIP Journal on Applied Signal Processing, vol. 2002, no. 11, pp. 1228-1247, 2002.
30. R. Kaucic and A. Blake, "Accurate, real-time, unadorned lip tracking," in Proc. IEEE International Conference on Computer Vision (ICCV), pp. 370-375, Bombay, India, January 1998.
31. J.-M. Odobez, Estimation, détection et segmentation du mouvement: une approche robuste et Markovienne, Ph.D. thesis, Université de Rennes I, December 1994.
32. J. A. Nelder and R. Mead, "A simplex method for function minimization," Computing Journal, vol. 7, no. 4, pp. 308-313, 1965.
33. H. Sakoe and S. Chiba, "Dynamic programming algorithm optimization for spoken word recognition," IEEE Trans. Acoustics, Speech, and Signal Processing, vol. 26, no. 1, pp. 43-49, 1978.
34. P. Daubias, "Utilisation de l'information acoustique pour aligner deux séquences de parole audiovisuelle," in Proc. 4th Rencontres Jeunes Chercheurs en Parole, pp. 74-77, Mons, Belgium, September 2001.
35. R. Descout, J.-F. Sérignat, O. Cervantes, and R. Carré, "BD-SONS: Une base de donnés des sons du français," in Proc. 12th International Congress on Acoustics (ICA), Toronto, Canada, 1986.
36. P. Daubias and P. Deléglise, "Lip-reading based on a fully automatic statistical model," in Proc. 7th International Conference on Spoken Language Processing (ICSLP), vol. 1, pp. 209-212, Denver, Col, USA, September 2002.