Discriminative analysis of lip motion features for speaker identification and speech-reading

IEEE Transactions on Image Processing

Volumn 15, Issue 10, 2006, Pages 2879-2891

  Çetingül, H Ertan ^a   Yücel, Yemez ^a   Erzin, Engin ^a   Tekalp, A Murat ^a  

^a KOÇ UNIVERSITY   (Turkey)

Author keywords

Bayesian discriminative feature selection; Lip motion; Speaker identification; Speech recognition; Temporal discriminative feature selection

Indexed keywords

COMPUTATIONAL GEOMETRY; FEATURE EXTRACTION; MARKOV PROCESSES; MATHEMATICAL MODELS;

BAYESIAN DISCRIMINATIVE FEATURE SELECTION; DISCRIMINATIVE ANALYSIS; FEATURE SELECTION; LIP MOTION FEATURES; SPEAKER IDENTIFICATION; TEMPORAL DISCRIMINATIVE FEATURE SELECTION;

SPEECH RECOGNITION;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; AUTOMATIC SPEECH RECOGNITION; BIOMETRY; COMPUTER ASSISTED DIAGNOSIS; DISCRIMINANT ANALYSIS; HISTOLOGY; HUMAN; IMAGE ENHANCEMENT; INFORMATION RETRIEVAL; LIP; LIP READING; METHODOLOGY; MOVEMENT (PHYSIOLOGY); PHYSIOLOGY; SPEECH;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BIOMETRY; DISCRIMINANT ANALYSIS; HUMANS; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; INFORMATION STORAGE AND RETRIEVAL; LIP; LIPREADING; MOVEMENT; PATTERN RECOGNITION, AUTOMATED; SPEECH; SPEECH RECOGNITION SOFTWARE;

EID: 33749187783     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2006.877528     Document Type: Article

References (42)

1. C. Bregler and Y. Konig, "Eigenlips for robust speech recognition," in Proc. IEEE Conf. Acoustics, Speech and Signal Processing, 1994, pp. 669-672.
2. M. J. Tomlinson, M. J. Russell, and N. M. Brooke, "Integrating audio and visual information to provide highly robust speech recognition," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 1996, vol. II, pp. 821-824.
3. G. Potamianos, C. Neti, G. Gravier,A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audio-visual speech," Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, Sep. 2003.
4. S. W. Foo, Y. Lian, and L. Dong, "Recognition of visual speech elements using adaptively boosted hidden Markov models," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 5, pp. 693-705, May 2004.
5. T. Chen, "Audiovisual speech processing," IEEE Signal Process. Mag., vol. 18, pp. 9-21, Jan. 2001.
6. S. L. Wang, W. H. Lau, S. H. Leung, and H. Yan, "A real-time automatic lipreading system," in Proc. 2004 Int. Symp. Circuits and Systems, 2004, vol. 2, pp. 101-104.
7. L. G. D. Silveira, J. Facon, and D. L. Borges, "Visual speech recognition: A solution from feature extraction towords classification," in Proc. XVI Brazilian Symp. Computer Graphics and Image Processing, 2003, pp. 399-405.
8. M. N. Kaynak, Q. Zhi, A. D. Cheok, K. Sengupta, Z. Jian, and K.C. Chung, "Analysis of lip geometric features for audio-visual speech recognition," IEEE Trans. Syst., Man, Cybern. A, Syst. Humans, vol. 34, no. 4, pp. 564-570, Jul. 2004.
9. X. Zhang, C. C. Broun, R. M. Mersereau, and M. A. Clements, "Automatic speechreading with applications to human-computer interfaces," EURASIP J. Appl. Signal Process., pp. 1228-1247, 2002.
10. J. F. G. Perez, A. F. Frangi, E. L. Solano, and K. Lukas, "Lip reading for robust speech recognition on embedded devices," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2005, vol. I, pp. 473-476.
11. I. Matthews, T. F. Cootes, J. A. Bangham, S. Cox, and R. Harvey, "Extraction of visual features for lipreading," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 2, pp. 198-213, Feb. 2002.
12. P. S. Aleksic, J. J. Williams, Z. Wu, and A. K. Katsaggelos, "Audiovisual speech recognition using MPEG-4 compliant visual features," EURASIP J. Appl. Signal Process., pp. 1213-1227, 2002.
13. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, Sep. 2000.
14. E. Erzin, Y. Yemez, and A. M. Tekalp, "Multimodal speaker identification using an adaptive classifier cascade based on modality reliability," IEEE Trans. Multimedia, vol. 7, no. 5, pp. 840-852, Oct. 2005.
15. N. A. Fox and R. B. Reilly, "Robust multi-modal person identification with tolerance of facial expression," in IEEE Int. Conf. Systems, Man and Cybernetics, 2004, vol. 1, pp. 580-585.
16. C. C. Broun, X. Zhang, R. M. Mersereau, and M. Clements, "Automatic speechreading with application to speaker verification," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2002, vol. I, pp. 685-688.
17. L. L. Mok, W. H. Lau, S. H. Leung, S. L. Wang, and H. Yan, "Lip features selection with application to person authentication," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2004, vol. III, pp. 397-400.
18. T.Wark and S. Sridharan, "Adaptive fusion of speech and lip information for robust speaker identification," Digital Signal Process., vol. 11, no. 3, pp. 169-186, Jul. 2001.
19. P. Jourlin, J. Luettin, D. Genoud, and H. Wassner, "Acoustic-labial speaker verification," Pattern Recognit. Lett., vol. 18, no. 9, pp. 853-858, 1997.
20. B. Fröba, C. Rothe, and C. Küblbeck, "Evaluation of sensor calibration in a biometric person recognition framework based on sensor fusion," in Proc. 4th IEEE Int. Conf. Automatic Face and Gesture Recognition, Mar. 2000, pp. 512-517.
21. R. W. Frischholz and U. Dieckmann, "Bioid: A multimodal biometric identification system," IEEE Computer, vol. 33, no. 2, pp. 64-68, Feb. 2000.
22. H. E.Çetingül, Y. Yemez, E. Erzin, and A. M. Tekalp, "Discriminative lip-motion features for biometric speaker identification," in Proc. Int. Conf. on Image Processing, Oct. 2004, pp. 2023-2026.
23. H. E.Çetingül, "Robust lip-motion features for speaker identification," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, Mar. 2005, vol. I, pp. 509-512.
24. J. P. Campbell, "Speaker recognition: A tutorial," Proc. IEEE, vol. 85, no. 9, pp. 1437-1462, Sep. 1997.
25. D. A. Reynolds, "Speaker identification and verification using gaussian mixture speaker models," Speech Commun., vol. 17, pp. 91-108, 1995.
26. J.-M. Odobez and P. Bouthemy, "Robust multiresolution estimation of parametric motion models," J. Vis. Commun. Image Represent., vol. 6, no. 4, pp. 348-365, Dec. 1995.
27. Y. Yemez, B. Sankur, and E. Anarim, "A quadratic motion-based object-oriented video codec," Signal Process.: Image Commun., vol. 15, pp. 729-766, 2000.
28. A. Puri, X. Chen, and A. Luthra, "Video coding using the H.264/MPEG-4 AVC compression standard," Signal Process.: Imag Commun., vol. 19, pp. 793-849, 2004.
29. M. Sadeghi, J. Kittler, and K. Messer, "Modelling and segmentation of lip area in face images," IEE Proc. Vis. Image Signal Process., vol. 149, no. 3, pp. 179-184, Jun. 2002.
30. S.-H. Leung, S.-L. Wang, and W.-H. Lau, "Lip Iimage segmentation using fuzzy clustering incorporating an elliptic shape function," IEEE Trans. Image Process., vol. 13, no. 1, pp. 51-62, Jan. 2004.
31. T. Wakasugi, M. Nishiura, and K. Fukui, "Robust lip contour extraction using separability of multi-dimensional distributions," in Proc. 6th IEEE Int. Conf. Automatic Face and Gesture Recognition, May 2004, pp. 415-420.
32. S. L. Wang, W. H. Lau, S. H. Leung, and A. W. C. Liew, "Lip segmentation with the presence of beards," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2004, vol. III, pp. 529-532.
33. N. Eveno, A. Caplier, and P.-Y. Coulon, "Accurate and quasi-automatic Lip tracking," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 5, pp. 706-715, May 2004.
34. X. Zhang, R. M. Mersereau, M. A. Clements, and C. C. Broun, "Visual speech feature extraction for improved speech recognition," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2002, pp. 1993-1996.
35. A. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, and K. Murphy, "A couple hmm for audio-visual speech recognition," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2002, pp. 2013-2016.
36. J. Luettin, N. Thacker, and S. Beet, "Statistical lip modelling for visual speech recognition," in Proc. 8th Eur. Signal Processing Conf., 1996, pp. 10-13.
37. B. Lucas and T. Kanade, "An iterative image restoration technique with an application to stereo vision," in Proc. DARPA IU Workshop, 1981, pp. 121-130.
38. C. C. Chibelushi, J. S. Mason, and F. Deravi, "Integration of acoustic and visual speech for speaker recognition," in Proc. 3rd Eur. Conf. Speech Communication and Technology, 1993, vol. 1, pp. 157-160.
39. T. J. Wark, S. Sridharan, and V. Chandran, "An approach to statistical lip modeling for speaker identification via chromatic feature extraction," in Proc. Int. Conf. Pattern Recognition, 1998, vol. 1, pp. 123-125.
40. X. Huang, A. Acero, and H.-W. Hon, Spoken Language Processing: A Guide to Theory, Algorithm, and System Development. Englewood Cliffs: Prentice-Hall, 2001.
41. A. M. Martinez and A. C. Kak, "PCA versus LDA," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 228-233, Feb. 2001.
42. E. Erzin, Y. Yemez, and A. M. Tekalp, DSP in Mobile and Vehicular Systems. New York: Springer Verlag, 2005, ch. Joint Audio-Video Processing for Robust Biometric Speaker Identification in Car.