Audio-visual speech recognition using red exclusion and neural networks

Journal of Research and Practice in Information Technology

Volumn 35, Issue 1, 2003, Pages 41-64

  Lewis, Trent W ^a   Powers, David M W ^a  

^a FLINDERS UNIVERSITY   (Australia)

Author keywords

Audio Visual Speech Recognition; Feature Extraction; Neural Networks; Sensor Fusion

Indexed keywords

ACOUSTIC NOISE; LINGUISTICS; NEURAL NETWORKS; SENSOR DATA FUSION;

AUDIO-VISUAL SPEECH RECOGNITION;

SPEECH RECOGNITION;

EID: 0041624571     PISSN: 1443458X     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (54)

1. ADJOUDANI, A. and BENOIT, C. (1996): On the integration of auditory and visual parameters in an HMM-based ASR, in STORK and HENNECKE (1996), 461-471.
2. ANGELOPOULOU, E., MOLANA, R., and DANIILIDIS, K. (2001): Multispectral color modeling. Technical Report MS-CIS-01-22, University of Pennsylvania, CIS.
3. BASU, M. and HO, T.K. (1999): The learning behavior of single neuron classifiers on linearly separable or nonseparable input. In Proceedings of the 1999 International Joint Conference on Neural Networks, Washington, D.C.
4. BREGLER, C., MANKE, S., HILD, H., and WAIBEL, A. (1993): Bimodal sensor integration on the example of "speechreading". Proceedings of the IEEE International Conference on Neural Networks, 667-671.
5. BREGLER, C., OMOHUNDRO, S.M., SHI, J., and KONIG, Y. (1996): Towards a robust speechreading dialog system. In STORK and HENNECKE (1996), 410-423.
6. BROOKES, M. (2000): VOICEBOX: Speech Processing Toolbox for MATLAB. World Wide Web, http://www.ee.ic.ac.uk/ hp/staff/dmb/voicebox/voicebox.html.
7. CHARNIAK, E. (1993): Statistical language learning. MIT Press, Cambridge, MA.
8. CHELAPPA, R., WILSON, C., and SIROHEY, S. (1995): Human and machine recognition of faces: A survey, in Proceedings of the IEEE, 83(5): 705-739.
9. COHEN, M., WALKER, R., and MASSARO, D. (1996): Perception of synthetic visual speech. In STORK and HENNECKE (1996), 153-168.
10. COIANIZ, T., TORRESANI, L. and CAPRILE, B. (1996): 2d deformable models for visual speech analysis. In STORK and HENNECKE (1996), 391-398.
11. DAVIS, S. and MERMELSTEIN, P. (1990): Comparision of parametric representations for monosyllabic word recognition in continuously spoken sentences. In WAIBEL, A. and LEE, K., editors, Readings in Speech Recognition, 64-74. Morgan Kaufmann Publishers Inc., San Mateo, CA.
12. DEMUTH, H. and BEALE, M. (1998): Neural Network Toolbox: User's Guide. The MathWorks, http://www.mathworks.com.
13. DODD, B. and CAMPBELL, R., editors (1987): Hearing by Eye: The pyschology of lip-reading. Lawrence Erlbaum Associates, Hillsdale NJ.
14. DUCHNOWSKI, P., HUNKE, P., BUSCHING, M., MEIER, U., and WAIBEL, A. (1995): Toward movement-invariant automatic lip-reading and speech recognition. In Proceedings of the International Conference of Acoustics, Speech, and Signal Processing, Detriot USA.
15. DUPONT. S. and LEUTTIN, J. (2000): Audio-visual speech modeling for continuous speech recognition. IEEE Transactions on Multimedia, 2(3):141-151.
16. FROMKIN, V., RODMAN, R., COLLINS, P., and BLAIR, D. (1996): An Introduction to Langauge. Hartcort Brace and Company, Sydney, 3rd edition.
17. GLOTIN, H., VERGYRI, D., NETI, C., POTAMIANOS, G., and LUETTIN, J. (2001): Weighting schemes for audio-visual fusion in speech recognition. In Proc. Int. Conf. Acoust, Speech Signal Process.
18. GOLDSCHEN, A., GARCIA, O., and PETAJAN, E. (1996): Rationale for phoneme-viseme mapping and feature selection in visual speech recognition. In STORK and HENNECKE (1996), 505-515.
19. GRANT, K. and SEITZ, P. (1998): The use of visible speech cues (speechreading) for directing auditory attention: Reducing temporal and spectral uncertainty in auditory detection of spoken utterances. In 16th International Congress on Acoustics.
20. GRAY, M., MOVELLAN, J., and SEJNOWSKI, T. (1997): Dynamic features for visual speechreading: A systematic comparision. In MOZER, JORDAN, and PERSCHE, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, Cambridge MA.
21. HECKMANN, M., BERTHOMMIER, F., and KROSCHEL, K. (2001a): A hybrid ANN/ HMM audio-visual speech recognition system. In Proceedings of AVSP-2001.
22. HECKMANN, M., BERTHOMMIER, F., and KROSCHEL, K. (2001b): Optimal weighting of posteriors for audio-visual speech recognition. In Proceedings of lCASSP 2001, Salt Lake City, Utah.
23. HENNECKE, M., PRASAD, K.V., and STORK, D. (1995): Automatic speech recognition using acoustic and visual signals. Technical Report CRC-TR-95-37, Ricoh Californian Research Centre.
24. HENNECKE, M., PRASAD, V., and STORK, D. (1994): Using deformable templates to infer visual speech dynamics. In 28th Annual Asimolar Conference on Signals, Systems, and Computer, Pacific Grove, CA. IEEE Computer. 2:576-582.
25. HENNECKE, M., STORK, D., and PRASAD, K.V. (1996): Visionary speech: Looking ahead to practical speech reading systems. In STORK and HENNECKE (1996), 331-350.
26. HUNKE, M. and WAIBEL, A. (1994): Face locating and tracking for human-computer interaction. In 28th Annual Asimolar Conference on Signals, Systems, and Computers, IEEE Computer Society, Pacific Grove, CA. 2: 1277-1281.
27. LEUTTIN, J. and DUPONT, S. (1998): Continuous audio-visual speech recognition. In Proceedings of the 5th European Conference on Computer Vision, 2: 657-673.
28. LEWIS, T.W. (2000). Audio visual speech recognition: Extraction, recognition, and integration
29. Lewis T.W. and POWERS, D. (2001): Lip feature extraction using red exclusion. In EADES, P. and JIN, J., editors, CRPIT: Visualisation, 2000, 2: 61-70.
30. M2VTS (2000): M2VTS Multimodel face database, release 1.0. World Wide Web, http://www.tele.ucl.ac.be/ PROJECTS/M2VTS/.
31. MASSARO, D. and STORK, D. (1998): Speech recognition and sensory integration: a 240-year old theorem helps explain how people and machines can integrate auditory and visual information to understand speech. American Scientist, 86(3): 236-245.
32. MCGURK, H. and MACDONALD, J. (1976): Hearing lips and seeing voices. Nature, 264:746-748.
33. MEIER, U., HURST, W., and DUCHNOWSKI, P. (1996): Adaptive bimodal sensor fusion for automatic speechreading. In Proceedings of the International Conference of Acoustics, Speech, and Signal Processing, 2: 833-837.
34. MEIER, U., STEIFELHAGEN, R., YANG, J., and WAIBEL, A. (1999): Towards unrestricted lip reading. In Second International Conference on Multimedia Interfaces, Hong Kong, http://wemer.ir.uks.de/js.
35. MOVELLAN, J. (1995): Visual speech recognition with stochastic networks. In Tesauro, G., Toruetzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems, 7: 851-858. MIT Press, Cambridge.
36. MOVELLAN, J. and MINEIRO, P. (1998): Robust sensor fusion: Analysis and application to audio visual speech recognition. Machine Learning, 32: 85-100.
37. NETI, C., POTAMIANOS, G., LEUTTIN, J., MATTHEWS, I., GLOTIN, H., and VERGYRI, D. (2001): Large-vocabulary audio-visual speech recognition: A summary of the Johns Hopkins summer 2000 workshop. In Workshop on Multimedia Signal Processing. Special Session on Joint Audio-Visual Processing, Cannes.
38. POTAMIANOS, G. and NETI, C. (2000): Stream confidence estimation for audio-visual speech recognition. In Proceedings of the International Conference on Spoken Language Processing, 746-749, Beijing.
39. POTAMIAONOS, G. and POTAMIANOS, A. (1999): Speaker adaptation for audio-visual speech recognition. In Proceedings of EUROSPEECH (3), 1291-1294, Budapest.
40. PRASAD, K., STORK, D., and WOLFF, G. (1993): Preprocessing video images for neural learning of lipreading. Technical Report CRC-TR-93-26, Ricoh California Research Centre.
41. RABINER, L. and JUANG, B. (1993): Fundamentals of Speech Recognition. Prentice Hall, Englewood Cliffs, NJ.
42. RAO, R. and MERSEREAU, R. (1994): Lip modeling for visual speech recognition. In 28th Annual Asimolar Conference on Signals, Systems, and Computers, volume 2. IEEE Computer Society, Pacific Grove CA.
43. ROBERT-RIBES, J., PIQUEMAL, M., SCHWARTZ, J., and ESCUDIER, P. (1996): Exploiting sensor fusion and stimuli complementary in av speech recognition. In STORK and HENNECKE (1996), 194-219.
44. ROGOZAN, A. (1999): Discriminative learning of visual data for audiovisual speech recognition. International Journal of Artificial Intelligence Tools, 8(1):43-52.
45. SCHAFER, R. and RABINER, L. (1990): Digital representations of speech signals. In WAIBEL, A. and LEE, K., editors, Readings in Speech Recognition, 49-64. Morgan Kaufmann Publishers Inc., San Mateo, CA.
46. SCHIFFERDECKER, G. (1994): Finding structure in language. Master's thesis, University of Karlsruhe.
47. STIEFELHAGEN, R., YANG, J., and MEIER, U. (1997): Real time lip tracking for lipreading. In Proceedings of Eurospeech '97.
48. STORK, D. and HENNECKE, M., editors (1996): Speechreading by Man and Machine: Models, System, and Applications. NATO/Springer-Verlag, New York.
49. SUMMERFIELD, Q. (1987): Some preliminaries to a comprehensive account of audio-visual speech perception, 3-52. In DODD and CAMPBELL (1987).
50. VERMA, A., FARUQUIE, T., NETI, C., BASU, S., and SENIOR, A. (1999): Late integration in audio-visual continuous speech recognition. In Automatic Speech Recognition and Understanding.
51. VOGT, M. (1996): Fast matching of a dynamic lip model to color video sequences under regular illumination conditions. In STORK and HENNECKE (1996), 399-407.
52. WALDEN, B., PROSEK, R., MONTGOMERY, A., SCHERR, C., and JONES, C. (1977): Effect of training on the visual recognition of consonants. Journal of Speech and Hearing Research, 20:130-145.
53. WARK, T., SRIDHARAN, S., and CHANDRAN, V. (1998): An approach to statistical lip modelling for speaker identification via chromatic feature extraction. In Proceedings of the IEEE International Conference on Pattern Recognition, 123-125.
54. YANG, J. and WAIBEL, A. (1996): A real-time face tracker. In Proceedings of WACV'96, 142-147.