Continuous audio-visual digit recognition using N -best decision fusion

Information Fusion

Volumn 5, Issue 2, 2004, Pages 91-101

  Meyer, Georg F ^a   Mulligan, Jeffrey B ^b   Wuerger, Sophie M ^a  

^a UNIVERSITY OF LIVERPOOL   (United Kingdom)

^b NASA AMES RESEARCH CENTER   (United States)

Author keywords

Audio visual speech; Decision fusion; Lip reading; Speech recognition

Indexed keywords

ACOUSTIC SIGNAL PROCESSING; ALGORITHMS; DATABASE SYSTEMS; MARKOV PROCESSES; MAXIMUM LIKELIHOOD ESTIMATION; SPURIOUS SIGNAL NOISE; VIDEO SIGNAL PROCESSING; WORD PROCESSING;

AUDIO-VISUAL SPEECH; DECISION FUSION; LINEAR DISCRIMINANT ANALYSIS; LIP READING;

SPEECH RECOGNITION;

EID: 1842854571     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2003.07.001     Document Type: Article

References (37)

1. Lippmann R.P. Speech recognition by machines and humans. Speech Communication. 22:1997;1-15.
2. Massaro D.W. Perceiving Talking Faces. From Speech Perception to a Behavioral Principle. 1998;MIT Press, Cambridge, MA.
3. G. Potamianos, C. Neti, G. Iyenar, E. Helmuth, Large-vocabulary audio-visual speech recognition by machines and humans, in: Proceedings on Eurospeech, Aalborg, 2001, pp. 1899-1902.
4. P.L. Silsbee, Sensory integration in audiovisual automatic speech recognition, in: 28th Annual Asilomar Conference on Signals, Systems, and Computers, vol. 1, 1994, pp. 561-565.
5. Jain A.K., Duin R.P.W., Mao J. Statistical pattern recognition: a review. IEEE Transaction on Pattern Analysis and Machine Intelligence. 22:2000;4-37.
6. M. Heckmann, F. Berthommier, K. Kroschel, Optimal weighting of posteriors for audio-visual speech recognition, in: Proceedings on ICASSP 2001, Salt Lake, 2001, pp. 161-164.
7. Heckmann M., Berthommier F., Kroschel K. Noise adaptive weighting in audio-visual speech recognition. Journal on Applied Signal Processing. 11:2002;1260-1273.
8. M. Heckmann, T. Wild, F. Berthommier, K. Kroschel, Comparing audio- and a-posteriori-probability-based stream confidence measures for audio-visual speech recognition, in: Proceedings on Eurospeech 2001, Aalborg, 2001, pp. 1023-1026.
9. G. Potamianos, C. Neti, Stream confidence estimation for audio-visual speech recognition, in: Proceedings on ICSLP 2000, Beijing, 2000, pp. 746-749.
10. H. Glotin, D. Vergyri, C. Neti, G. Potamianos, J. Luettin, Weighting schemes for audio-visual fusion in speech recognition, in: Proceedings on ICASSP 2001, Salt Lake, 2001.
11. C. Neti, G. Potamianos, J. Luettin, I. Matthews, H. Glotin, D. Vergyri, J. Sison, A. Mashari, J. Zhou, Audio visual speech recognition, Final Workshop 2000 Report, Center for Language and Speech Processing, The Johns Hopkins University, Baltimore, 2000.
12. Massaro D.W., Cohen M.M. Evaluation and integration of visual and auditory information in speech perception. Journal of Experimental Psychology: HPP. 9:1983;751-753.
13. D.W. Massaro, Auditory visual speech processing, in: Proceedings on Eurospeech 2001, Aalborg, 2001, pp. 1153-1156.
14. Movellan J.R., Chadderon G. Channel separability in the audio-visual integration of speech: a Bayesian approach. Speechreading by Man and Machine, Models, Systems and Applications. NATO ASI Series. 1996;473-487 Springer-Verlag, Berlin.
15. Summerfield A.Q. Some preliminaries to a comprehensive account of audio-visual speech perception. Dodd B., Campbell R. Hearing by Eye, the Psychology of Lip-reading. 1987;3-51 Lawrence Erlbaum Associates, Hillsdale, NJ.
16. A.V. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, K. Murphy, A coupled HMM for audio-visual speech recognition, in: Proceedings on ICASSP 2002, vol. 2, 2002, pp. 2013-2016.
17. G. Potamianos, H.P. Graf, E. Cosatto, An image transform approach for HMM based automatic lipreading, in: Proceedings of the International Conference on Image Processing, Chicago, vol. III, 1998, pp. 173-177.
18. Dupont S., Luettin J. Audio-visual speech modelling for continuous speech recognition. IEEE Transactions on Multimedia. 2:2000;141-151.
19. I. Mathews, J.A. Bangham, R. Harvey, S. Cox, A comparison of active shape model and scale decomposition based features for visual speech recognition, in: Proceedings of the European Conference on Computer Vision, Freiburg, 1998, pp. 514-528.
20. D. Pearce, H.-G. Hirsch, The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions, in: Proceedings on ICSLP'00, Beijing, vol. 4, 2000, pp. 29-32.
21. S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, P. Woodland, The HTK Book, Revised version for HTK V 3.0, 2000, http://htk.eng.cam.ac.uk/index. shtml.
22. Rabiner L., Juang B.H. Fundamentals of Speech Recognition. 1993;Prentice Hall, Englewood Cliffs.
23. K.V. Prasad, G. Storck, G.J. Wolf, Preprocessing video images for neural learning of lipreading, Ricoh CRC Technical Report 93-26, 1993.
24. C. Bregler, Y. Konig, Eigenlips for robust speech recognition, in: Proceedings on ICASSP'94, Adelaide, 1994, pp. 669-672.
25. I. Mathews, S. Cootes, S. Cox, R. Harvey, J.A. Bangham, Lipreading using shape, shade and scale, in: Proceedings of Workshop on Audio Visual Speech Processing, Terrigal, 1998, pp. 73-78.
26. Chiou G., Hwang J.N. Lipreading from color video. IEEE Transactions on Image Processing. 6:1997;1192-1195.
27. Kass M., Witkin A., Terzopoulos D. Snakes: active contour models. Computer, Speech and Language. 5:1991;275-294.
28. Yuille A.L., Hallinan P.W., Cohen D.S. Feature extraction from faces using deformable templates. International Journal of Computer Vision. 8:1992;99-111.
29. N.M. Brooke, S.D. Scott, PCA image coding schemes and visual speech intelligibility, in: Proceedings on Institute of Acoustics, Windermere, vol. 16, 1994, pp. 123-129.
30. Dupont S., Luettin J. Audio-visual speech modeling for continuous speech recognition. IEEE Transactions on Multimedia. 2:2000;141-151.
31. P. Duchnowski, M. Hunke, D. Büsching, U. Meier, A. Waibel, Toward movement invariant automatic lip-reading and speech recognition, in: Proceedings of International Conference on Spoken Language Processing, Philadelphia, vol. 1, 1995, pp. 109-112.
32. G. Potanianos, A. Verma, C. Neti, G. Iyengar, S. Basu, A cascade image transform for speaker independent automatic speech reading, in: Proceedings of International Conference on Multimedia and Expo, vol. 2, 2000, pp. 1097-1100.
33. Brooke N.M. Talking heads and speech recognisers that can see: the computer processing of visual speech signals. Stork D.G., Hennecke M.E. Speechreading by Humans and Machines. 1996;351-371 Springer-Verlag, Berlin.
34. Tessier P., Robert-Ribes N., Schwartz J.-L., Guerin-Dugue A. Comparing models of audio-visual fusion in a noisy vowel recognition task. IEEE Transactions on SAP. 7:1999;629-642.
35. M. Heckmann, K. Kroschel, C. Savariaux, F. Berthommier, DCT-based video features for audio-visual speech recognition, in: Proceedings on ICSLP, Beijing, vol. 3, 2002, pp. 1925-1928.
36. Rogozan A. Discriminative learning of visual data for audiovisual speech recognition. International Journal on Artificial Intelligence Tools. 8:1999;43-52.
37. Bourlard H., Wellekens C.J. Links between Markov models and multilayer perceptrons. IEEE Transactions on Pattern Analysis and Machine Intelligence. 12(2):1990;1167-1178.