Audio-visual and visual-only speech and speaker recognition: Issues about theory, system design, and implementation

Visual Speech Recognition: Lip Segmentation and Mapping

Volumn , Issue , 2009, Pages 1-38

  Shiell, Derek J ^a   Terry, Louis H ^a   Aleksic, Petar S ^b   Katsaggelos, Aggelos K ^c  

^a NORTHWESTERN UNIVERSITY   (United States)

^b GOOGLE INC   (United States)

^c Northwestern University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84870243209     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4018/978-1-60566-186-5.ch001     Document Type: Chapter

References (109)

1. The AAM-API. (2008). Retrieved November, 2007, from http://www2.imm.dtu.dk/~aam/aamapi/
2. Aleksic, P. S., & Katsaggelos, A. K. (2003a, December). An audio-visual person identification and verification system using FAPs as visual features. Workshop on Multimedia User Authentication (MMUA), (pp. 80-84), Santa Barbara, CA.
3. Aleksic, P. S., & Katsaggelos, A. K. (2003b, July). Product HMMs for Audio-Visual Continuous Speech Recognition Using Facial Animation Parameters. In Proceedings of IEEE Int. Conf. on Multimedia & Expo (ICME), Vol. 2, (pp. 481-484), Baltimore, MD.
4. Aleksic, P. S., & Katsaggelos, A. K. (2004). Speech-to-video synthesis using MPEG-4 compliant visual features. IEEE Trans. CSVT, Special Issue Audio Video Analysis for Multimedia Interactive Services, 682-692.
5. Aleksic, P. S., & Katsaggelos, A. K. (2006). Audio-Visual Biometrics. IEEE Proceedings, 94(11), 2025-2044.
6. Aleksic, P. S., Potamianos, G., & Katsaggelos, A. K. (2005). Exploiting visual information in automatic speech processing. In Handbook of Image and Video Processing (pp. 1263 -1289): Academic Press.
7. Aleksic, P. S., Williams, J. J., Wu, Z., & Katsaggelos, A. K. (2002). Audio-visual speech recognition using mpeg-4 compliant visual features. EURASIP Journal on Applied Signal Processing, 1213-1227.
8. Baker, S., Gross, R., & Matthews, I. (2003a). Lucas-kanade 20 years on: a unifying framework: Part 2: Carnegie Mellon University Robotics Institute.
9. Baker, S., Gross, R., & Matthews, I. (2003b). Lucas-kanade 20 years on: a unifying framework: Part 3: Carnegie Mellon University Robotics Institute.
10. Baker, S., & Matthews, I. (2004). Lucas-kanade 20 years on: A unifying framework. Int. J. Comput. Vision, 56(3), 221-255.
11. Barbosa, A. V., & Yehia, H. C. (2001). Measuring the relation between speech acoustics and 2-D facial motion. Paper presented at the Int. Conf. Acoustics, Speech Signal Processing.
12. Barczak, A. L. C. (2004). Evaluation of a Boosted Cascade of Haar-Like Features in the Presence of Partial Occlusions and Shadows for Real Time Face Detection. In PRICAI 2004: Trends in Artificial Intelligence, 3157, 969-970. Berlin, Germany: Springer.
13. Barker, J. P., & Berthommier, F. (1999). Estimation of speech acoustics from visual speech features: A comparison of linear and non-linear models. Paper presented at the Int. Conf. Auditory Visual Speech Processing.
14. Ben-Yacoub, S., Abdeljaoued, Y., & Mayoraz, E. (1999). Fusion of face and speech data for person identity verification. IEEE Trans. Neural Networks, 10, 1065-1074.
15. Bengio, S., & Mariethoz, J. (2004). A statistical significance test for person authentication. Paper presented at the Speaker and Language Recognition Workshop (Odyssey).
16. Bengio, S., Mariethoz, J., & Keller, M. (2005). The expected performance curve. Paper presented at the Int. Conf. Machine Learning, Workshop ROC Analysis Machine Learning.
17. Bernstein, L. E. (1991). Lipreading Corpus V-VI: Disc 3. Gallaudet University, Washington, D.C.
18. Biffiger, R. (2005). Audio-Visual Automatic Isolated Digits Recognition. Northwestern University, Evanston.
19. Blanz, V., Grother, P., Phillips, P. J., & Vetter, T. (2005). Face recognition based on frontal views generated from non-frontal images. Paper presented at the Computer Vision Pattern Recognition.
20. Bowyer, K. W., Chang, K., & Flynn, P. (2006). A survey of approaches and challenges in 3-D and multi-modal 3-D face recognition. Computer Vision Image Understanding, 101(1), 1-15.
21. Brunelli, R., & Falavigna, D. (1995). Person identification using multiple cues. IEEE Trans. Pattern Anal. Machine Intell., 10, 955-965.
22. Campbell, J. P. (1997). Speaker recognition: A tutorial. Proceedings of the IEEE, 85(9), 1437-1462.
23. Campbell, R., Dodd, B., & Burnham, D. (Eds.). (1998). Hearing by Eye II: Advances in the Psychology of Speechreading and Auditory Visual Speech. Hove, U.K.: Pyschology Press.
24. Chang, K. I., Bowyer, K. W., & Flynn, P. J. (2005). An evaluaion of multimodal 2D + 3D face biometrics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(4), 619-6124.
25. Chaudhari, U. V., & Ramaswamy, G. N. (2003). Information fusion and decision cascading for audio-visual speaker recognition based on time-varying stream reliability prediction. Paper presented at the Int. Conf. Multimedia Expo.
26. Chaudhari, U. V., Ramaswamy, G. N., Potamianos, G., & Neti, C. (2003). Audio-visual speaker recognition using time-varying stream reliability prediction. Paper presented at the Int. Conf. Acoustics, Speech Signal Processing, Hong Kong, China.
27. Chen, T. (2001). Audiovisual speech processing. IEEE Signal Processing Mag., 18, 9-21.
28. Chibelushi, C. C., Deravi, F., & Mason, J. S. (1993). Voice and facial image integration for speaker recognition. Paper presented at the IEEE Int. Symp. Multimedia Technologies Future Appl., South-ampton, U.K.
29. Chibelushi, C. C., Deravi, F., & Mason, J. S. (1997). Audio-visual person recognition: An evaluation of data fusion strategies. Paper presented at the Eur. Conf. Security Detection, London, U.K.
30. Chibelushi, C. C., Deravi, F., & Mason, J. S. (2002). A review of speech-based bimodal recognition. IEEE Trans. Multimedia, 4(1), 23-37.
31. Chibelushi, C. C., Gandon, S., Mason, J. S. D., Deravi, F., & Johnston, R. D. (1996). Design issues for a digital audio-visual integrated database. Paper presented at the Integrated Audio-Visual Processing for Recognition, Synthesis and Communication (Digest No: 1996/213), IEE Colloquium on.
32. Cootes, T. (2008). Modelling and Search Software. Retrieved November, 2007, from http://www.isbe. man.ac.uk/~bim/software/am_tools_doc/index.html
33. Cootes, T., Edwards, G., & Taylor, C. (1998). A comparitive evaluation of active appearance models algorithms. Paper presented at the British Machine Vision Conference.
34. Cootes, T., Edwards, G., & Taylor, C. (2001). Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 681-685.
35. Deller, J. R., Jr., Proakis, J. G., & Hansen, J. H. L. (1993). Discrete-Time Processing of Speech Signals. Englewood Cliffs, NJ: Macmillan.
36. Duda, R. O., Hart, P. E., & Stork, D. G. (2001). Pattern Classification. Hoboken, NJ: Wiley.
37. Dupont, S., & Luettin, J. (2000). Audio-visual speech modeling for continuous speech recognition. IEEE Trans. Multimedia, 2(3), 141-151.
38. Flanagan, J. L. (1965). Speech Analysis, Synthesis, and Perception. Berlin, Germany: Springer-Verlag.
39. Fox, N. A., Gross, R., de Chazal, P., Cohn, J. F., & Reilly, R. B. (2003). Person identification using automatic integration of speech, lip, and face experts. Paper presented at the ACM SIGMM 2003 Multimedia Biometrics Methods and Applications Workshop (WBMA'03), Berkley, CA.
40. Fox, N. A., O'Mullane, B., & Reilly, R. B. (2005). The realistic multi-modal VALID database and visual speaker identification comparison experiments. Paper presented at the 5th International Conference on Audio- and Video-Based Biometric Person Authentication.
41. Goldschen, A. J., Garcia, O. N., & Petajan, E. D. (1996). Rationale for phoneme-viseme mapping and feature selection in visual speech recognition. In D. G. Stork & M. E. Hennecke (Eds.), Speechreading by Humans and Machines (pp. 505-515). Berlin, Germany: Springer.
42. Gordan, M., Kotropoulos, C., & Pitas, I. (2002). A support vector machine-based dynamic network for visual speech recognition applications. EURASIP J. Appl. Signal Processing, 2002(11), 1248-1259.
43. Gravier, G., Potamianos, G., & Neti, C. (2002). Asynchrony modeling for audio-visual speech recognition. Paper presented at the Human Language Techn. Conf.
44. Gross, R., Matthews, I., & Baker, S. (2004). Constructing and fitting active appearance models with occlusion. Paper presented at the IEEE Workshop on Face Processing in Video.
45. Gross, R., Matthews, I., & Baker, S. (2006). Active appearance models with occlusion. Image and Vision Computing, 24, 593-604.
46. Haykin, S. (2002). Adaptive Filter Theory: 4th Edition. Upper Saddle River, NJ: Prentice Hall.
47. Hazen, T. J., Saenko, K., La, C.-H., & Glass, J. (2004). A segment-based audio-visual speech recognizer: Data collection, development and initial experiments. Paper presented at the International Conference on Multimodal Interfaces.
48. Hjelmas, E., & Low, B. K. (2001). Face detection: A survey. Computer Vision Image Understanding, 83(3), 236-274.
49. Hong, L., & Jain, A. (1998). Integrating faces and fingerprints for personal identification. IEEE Trans. Pattern Anal. Machine Intell., 20, 1295-1307.
50. HTK Speech Recognition Toolkit. (2008). Retrieved November, 2007, from http://htk.eng.cam.ac.uk/
51. Hu, C., Xiao, J., Matthews, I., Baker, S., Cohn, J., & Kanade, T. (2004). Fitting a single active appearance model simultaneously to multiple images. Paper presented at the British Machine Vision Conference.
52. Hua, G., & Y.Wu. (2006). Sequential mean field variational analysis of structured deformable shapes. Computer Vision and Image Understanding, 101, 87-99.
53. Jain, A. K., Ross, A., & Prabhakar, S. (2004). An introduction to biometric recognition. IEEE Trans. Circuits Systems Video Technol., 14(1), 4-20.
54. Jankowski, C., Kalyanswamy, A., Basson, S., & Spitz, J. (1990). NTIMIT: A Phonetically Balanced Continuous Speech Telephone Bandwidth Speech Database. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 1, 109-112.
55. Jiang, J., Alwan, A., Keating, P. A., E. T. Auer, J., & Bernstein, L. E. (2002). On the relationship between face movements, tongue movements, and speech acoustics. EURASIP J. Appl. Signal Processing, 2002(11), 1174-1188.
56. Jourlin, P., Luettin, J., Genoud, D., & Wassner, H. (1997). Integrating acoustic and labial information for speaker identification and verification. Paper presented at the 5th Eur. Conf. Speech Communication Technology, Rhodes, Greece.
57. Kass, M., Witkin, A., & Terzopoulos, D. (1988). Snakes: Active contour models. Int. J. Comput. Vision, 4(4), 321-331.
58. Kaucic, R., Dalton, B., & Blake, A. (1996). Real-time lip tracking for audio-visual speech recognition applications. Paper presented at the European Conference on Computer Vision.
59. Kong, S. G., Heo, J., Abidi, B. R., Paik, J., & Abidi, M. A. (2005). Recent advances in visual and infrared face recognition - A review. Computer Vision Image Understanding, 97(1), 103-135.
60. Koterba, S., Baker, S., Matthews, I., Hu, C., Xiao, J., Cohn, J., et al. (2005). Multi-view aam fitting and camera calibration. Paper presented at the Tenth IEEE International Conference on Computer Vision
61. Lee, B., Hasegawa-Johnson, M., Goudeseune, C., Kamdar, S., Borys, S., Liu, M., et al. (2004). AVICAR: Audio-visual speech corpus in a car environment. Paper presented at the Conf. Spoken Language.
62. Lienhart, R., & Maydt, J. (2002). An Extended Set of Haar-like Features for Rapid Object Detection. IEEE ICIP, 1, 900-903.
63. Lippmann, R. (1997). Speech perception by humans and machines. Speech Communication, 22, 1-15.
64. Luettin, J. (1997). Visual speech and speaker recognition. Unpublished Ph.D. dissertation, University of Sheffield, Sheffield, U.K.
65. Luettin, J., Thacker, N., & Beet, S. (1996). Speaker identification by lipreading. Paper presented at the Int. Conf. Speech and Language Processing.
66. Marcheret, E., Libal, V., & Potamianos, G. (2007). Dynamic stream weight modeling for audio-visual speech recognition. Paper presented at the Int. Conf. Acoust. Speech Signal Process.
67. Matthews, I., & Baker, S. (2004). Active appearance models revisited. Int. J. Comput. Vision, 60(2), 135-164.
68. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
69. Messer, K., Matas, J., Kittler, J., Luettin, J., & Maitre, G. (1999). XM2VTSDB: The extended M2VTS database. Paper presented at the 2nd Int. Conf. Audio- and Video-Based Biometric Person Authentication.
70. Movellan, J. R. (1995). Visual speech recognition with stochastic networks. In G. Tesauro, D. Toruetzky & T. Leen (Eds.), Advances in Neural Information Processing Systems (Vol. 7). Cambridge, MA: MIT Press.
71. Nakamura, S.(2001). Fusion of Audio-Visual Information for Integrated Speech Processing. Paper presented at the Audio- and Video-Based Biometric Person Authentication (AVBPA).
72. Neely, K. K. (1956). Effect of visual factors on the intelligibility of speech. J. Acoustic. Soc. Amer., 28, 1275.
73. Nefian, A., Liang, L., Pi, X., Liu, X., & Murphy, K. (2002). Dynamic bayesian networks for audio-visual speech recognition. EURASIP J. Appl. Signal Processing, 11, 1274-1288.
74. Neti, C., Potamianos, G., Luettin, J., Matthews, I., Glotin, H., Vergyri, D., et al. (2000). Audio-visual speech recognition, Technical Report. Johns Hopkins Univesity, Baltimore.
75. Open Computer Vision Library. (2008). Retrieved November, 2007, from http://sourceforge.net/projects/opencvlibrary/
76. Patterson, E. K., Gurbuz, S., Tufekci, Z., & Gowdy, J. N. (2002). CUAVE: A new audio-visual database for multimodal human-computer interface research. Paper presented at the Int. Conf. Acoustics, Speech and Signal Processing.
77. Petajan, E. (1985). Automatic lipreading to enhance speech recognition. Paper presented at the IEEE Conference on CVPR.
78. Pigeon, S., & Vandendorpe, L. (1997). The M2VTS multimodal face database (release 1.00). Paper presented at the 1st Int. Conf. Audio- and Video-Based Biometric Person Authentication.
79. Pitsikalis, V., Katsamanis, A., Papandreou, G., & Maragos, P. (2006). Adaptive Multimodal Fusion by Uncertainty Compensation. Paper presented at the INTERSPEECH 2006.
80. Popovici, V., Thiran, J., Bailly-Bailliere, E., Bengio, S., Bimbot, F., Hamouz, M., et al. (2003). The BANCA Database and Evaluation Protocol. Paper presented at the 4th International Conference on Audio- and Video-Based Biometric Person Authentication.
81. Potamianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. W. (2003). Recent advances in the automatic recognition of audio-visual speech. Proceedings of the IEEE, 91, 1306-12326.
82. Potamianos, G., Neti, C., Luettin, J., & Matthews, I. (2004). Audio-visual automatic speech recognition: An overview. In G. Bailly, E. Vatikiotis-Bateson & P. Perrier (Eds.), Issues in Visual and Audio-Visual Speech Processing: MIT Press.
83. Rabiner, L., & Juang, B.-H. (1993). Fundamentals of Speech Recognition. Englewood Cliffs: Prentice Hall.
84. Ross, A., & Jain, A. (2003). Information fusion in biometrics. Pattern Recogn. Lett., 24, 2215-2125.
85. Sanderson, C., Bengio, S., & Gao, Y. (2006). On transforming statistical models for non-frontal face verification. Pattern Recognition, 29(2), 288-302.
86. Sanderson, C., & Paliwal, K. K. (2003). Noise compensation in a person verification system using face and multiple speech features. Pattern Recognition, 36(2), 293-302.
87. Sanderson, C., & Paliwal, K. K. (2004). Identity verification using speech and face information. Digital Signal Processing, 14(5), 449-480.
88. Sargin, M. E., Erzin, E., Yemez, Y., & Tekalp, A. M. (2006, May). Multimodal speaker identification using canonical correlation analysis. Paper presented at the Int. Conf. Acoustics, Speech Signal Processing, Toulouse, France.
89. Shiell, D. J., Terry, L. H., Aleksic, P. S., & Katsaggelos, A. K. (2007, September). An Automated System for Visual Biometrics. Paper presented at the Forty-Fifth Annual Allerton Conference on Communication, Control, and Computing, Urbana-Champaign, IL.
90. Summerfield, Q. (1979). Use of visual information in phonetic perception. Phonetica, 36, 314-331.
91. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In R. Campbell & B. Dodd (Eds.), Hearing by Eye: The Psychology of Lip-Reading (pp. 3-51). London, U.K.: Lawrence Erlbaum.
92. Summerfield, Q. (1992). Lipreading and audio-visual speech perception. Philosophical Transactions: Biological Sciences, 335(1273), 71-78.
93. Tamura, S., Iwano, K., & Furui, S. (2005). A Stream-Weight Optimization Method for Multi--Stream HMMs Based on Likelihood Value Normalization. Int. Conf. Acoustics, Speech and Signal Processing (ICASSP '05), 1, 469-472.
94. Unknown. (1999). Robust speaker verification via asynchronous fusion of speech and lip information. Paper presented at the 2nd Int. Conf. Audio- and Video-Based Biometric Person Authentication, Washington, D. C.
95. Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. Paper presented at the IEEE Conf. on Computer Vision and Pattern Recognition.
96. Wark, T., Sridharan, S., & Chandran, V. (1999). Robust speaker verification via asynchronous fusion of speech and lip information. Paper presented at the 2nd Int. Conf. Audio- and Video-Based Biometric Person Authentication, Washington, D. C.
97. Wark, T., Sridharan, S., & Chandran, V. (2000). The use of temporal speech and lip information for multi-modal speaker identification via multi-stream HMMs. Paper presented at the Int. Conf. Acoustics, Speech Signal Processing, Istanbul, Turkey.
98. Williams, J. J., Rutledge, J. C., Garstecki, D. C., & Katsaggelos, A. K. (1998). Frame rate and viseme analysis for multimedia applications. VLSI Signal Processing Systems, 23(1/2), 7-23.
99. Wu, Z., Aleksic, P. S., & Katsaggelos, A. K. (2002, October). Lip tracking for MPEG-4 facial animation. Paper presented at the Int. Conf. on Multimodal Interfaces, Pittsburgh, PA.
100. Wu, Z., Aleksic, P. S., & Katsaggelos, A. K. (2004, May). Inner lip feature extraction for MPEG-4 facial animation. Paper presented at the Int. Conf. Acoust., Speech, Signal Processing, Montreal, Canada.
101. Xiao, J., Baker, S., Matthews, I., & Kanade, T. (2004). Real-time combined 2D+3D active appearance models. Paper presented at the IEEE Conference on Computer Vision and Pattern Recognition.
102. Yang, M.-H., Kriegman, D., & Ahuja, N. (2002). Detecting faces in images: A survey. IEEE Trans. Pattern Anal. Machine Intell., 24(1), 34-58.
103. Yehia, H. C., Kuratate, T., & Vatikiotis-Bateson, E. (1999). Using speech acoustics to drive facial motion. Paper presented at the 14th Int. Congr. Phonetic Sciences.
104. Yehia, H. C., Rubin, P., & Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26(1-2), 23-43.
105. Young, S. (2008). The ATK Real-Time API for HTK. Retrieved November, 2007, from http://mi.eng. cam.ac.uk/research/dialogue/atk_home
106. Young, S., Evermann, G., Hain, T., Kershaw, D., Moore, G., Odell, J., et al. (2005). The HTK Book. London, U.K.: Entropic.
107. Yuan, J., Wu, Y., & Yang, M. (2007). Discovery of Collocation Patterns: from Visual Words to Visual Phrases. Paper presented at the IEEE Conf. on Computer Vision and Pattern Recognition.
108. Yuille, A. L., Hallinan, P. W., & Cohen, D. S. (1992). Feature extraction from faces using deformable templates. Int. J. Comput. Vision, 8(2), 99-111.
109. Zhao, W.-Y., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003). Face recognition: A literature survey. ACM Computing Survey, 35(4), 399-458.