Energetic and informational masking effects in an audiovisual speech recognition system

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 3, 2009, Pages 446-458

  Barker, Jon ^b   Shao, Xu ^a,b  

^a NUANCE COMMUNICATIONS   (United States)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Audiovisual speech recognition; Energetic masking (EM); Iinformational masking (IM); Source separation; Speech fragment decoding

Indexed keywords

ACOUSTIC ANALYSIS; ACOUSTIC SIGNALS; AUDIO-VISUAL SPEECH; AUDIOVISUAL SPEECH RECOGNITION; ENERGETIC MASKING; ENERGETIC MASKING (EM); IINFORMATIONAL MASKING (IM); INFORMATIONAL MASKING; NONSTATIONARY NOISE; SOURCE SEPARATION; TARGET SPEECH; TWO STAGE; VISUAL CUES; VISUAL INFORMATION; VISUAL REPRESENTATIONS;

ACOUSTICS; DECODING; SIGNAL ANALYSIS; SPEECH COMMUNICATION; TARGETS; VISUAL COMMUNICATION;

SPEECH RECOGNITION;

EID: 70350437422     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2011534     Document Type: Article

References (37)

1. R. Lufti, "How much masking is informational masking?, " J. Acoust. Soc. Amer., vol. 88, pp. 2607-2610, 1990.
2. D. S. Brungart, B. D. Simpson, M. A. Ericson, and K. R. Scott, "Informational and energetic masking effects in the perception of multiple simultaneous talkers, " J. Acoust. Soc. Amer., vol. 110, no. 5, pp. 2527-2538, 2001.
3. N. I. Durlach, C. R. Mason, G. Kidd, Jr., T. L. Arbogast, H. S. Colburn, and B. G. Shinn-Cunningham, "Note on informational masking, " J. Acoust. Soc. Amer., vol. 113, no. 6, pp. 2984-2987, 2003.
4. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audiovisual speech, " Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, 2003.
5. K. Iwano, S. Furui, T. Yoshinaga, and S. Tamura, "Audio-visual speech recognition using lip movement extracted from side-face images, " in Proc. Int.Workshop on Auditory-Visual Speech Process. (AVSP), 2003, pp. 117-120.
6. P. Lucey and G. Potamianos, "Lipreading using profile versus frontal views, " in Proc. IEEE Int. Workshop Multimedia Signal Process. (MMSP'06), Victoria, BC, Canada, 2006, pp. 24-28.
7. K. Kumar, T. Chen, and R. M. Stern, "Profile view lip reading, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2007, vol. 4, pp. 429-432.
8. P. Lucey, G. Potamianos, and S. Sridharan, "An extended pose-invariant lipreading system, " in Proc. Int. Workshop Auditory-Visual Speech Process. (AVSP), Hilvarenbeek, The Netherlands, 2007.
9. G. Potamianos and C. Neti, "Audio-visual speech recognition in challenging environments, " in Proc. Eurospeech'03, Geneva, Switzerland, 2003, pp. 1293-1296.
10. R. Seymour, D. Stewart, and J. Ming, "Comparison of image transform based features for visual speech recognition in clean and corrupted video, " EURASIP J. Image Video Process., vol. 8, no. 2, 2008.
11. J.-L. Schwartz, F. Berthommier, and C. Savariaux, "Seeing to hear better: Evidence for early audio-visual interactions in speech identification, " Cognition, vol. 93, no. 2, pp. B69-B78, 2004.
12. J. Driver, "Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading, " Nature, vol. 381, no. 6577, pp. 66-68, 1996.
13. K. S. Helfer and R. L. Freyman, "The role of visual speech cues in reducing energetic and informational masking, " J. Acoust. Soc. Amer., vol. 117, no. 2, pp. 842-849, 2005.
14. F. Wightman, D. Kistler, and D. Brungart, "Informational masking of speech in children: Auditory-visual integration, " J. Acoust. Soc. Amer., vol. 119, no. 6, pp. 3940-3949, 2006.
15. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative association of vocal-tract and facial behavior, " Speech Commun., vol. 26, no. 1, pp. 23-43, 1998.
16. J. Jiang, A. Alwan, P. A. Keating, E. T. Auer Jr., and L. Bernstein, "On the relationship between face movements, tongue movements, and speech acoustics, " EURASIP J. Adv. Signal Process., vol. 2002, no. 11, pp. 1174-1188, 2002.
17. J. Barker and F. Berthommier, "Estimation of speech acoustics from visual speech features: A comparison of linear and non-linear models, " in Proc. Int. Workshop Auditory-Visual Speech Process. (AVSP), San Francisco, CA, 1999.
18. D. Sodoyer, L. Girin, G. Jutter, and J.-L. Schwartz, "Developing an audio-visual speech source separation algorithm, " Speech Commun., vol. 44, pp. 113-125, 2007.
19. B. Milner and I. Almajai, "Noisy audio speech enhancement using Wiener filters derived from visual speech, " in Proc. Int. Workshop Auditory-Visual Speech Process. (AVSP), Hilvarenbeek, The Netherlands, 2007.
20. B. Rivet, L. Girin, C. Servière, D.-T. Pham, and C. Jutten, "Audiovisual speech source separation: A regularization method based on visual voice activity detection, " in Proc. Int. Workshop Auditory-Visual Speech Process. (AVSP), Hilvarenbeek, The Netherlands, 2007.
21. R. Goecke, G. Potamianos, and C. Neti, "Noisy audio feature enhancement using audio-visual speech data, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Orlando, FL, 2002, pp. 2025-2028.
22. M. P. Cooke, "A glimpsing model of speech perception in noise, " J. Acoust. Soc. Amer., vol. 119, no. 3, pp. 1562-1573, 2006.
23. A. S. Bregman, Auditory Scene Analysis. Cambridge, MA: MIT Press, 1990.
24. P. Smaragdis, "Redundancy reduction for computational audition, a unifying approach, " Ph.D. dissertation, Mass. Inst. Technol., Cambridge, MA, 2001.
25. D.-L. Wang and G. J. Brown, Eds., Computational Auditory Scene Analysis: Principles, Algorithms, and Applications. New York: IEEE Press/Wiley- Interscience, 2007.
26. J. Barker, A. Coy, N. Ma, and M. Cooke, "Recent advances in speech fragment decoding techniques, " in Proc. Interspeech'06, Pittsburgh, PA, 2006, pp. 85-88.
27. J. P. Barker, M. P. Cooke, and D. P. W. Ellis, "Decoding speech in the presence of other sources, " Speech Commun., vol. 45, pp. 5-25, 2005.
28. M. P. Cooke, P. D. Green, L. Josifovski, and A. Vizinho, "Robust automatic speech recognition with missing and uncertain acoustic data, " Speech Commun., vol. 34, pp. 267-285, 2001.
29. M. P. Cooke and D. P. W. Ellis, "The auditory organization of speech and other sources in listeners and computational models, " Speech Commun., vol. 35, pp. 141-177, 2001.
30. M. P. Cooke, J. Barker, S. P. Cunningham, and X. Shao, "An audiovisual corpus for speech perception and automatic speech recognition, " J. Acoust. Soc. Amer., vol. 120, pp. 2421-2424, 2006.
31. M. P. Cooke, M. L. Garcia Lecumberri, and J. Barker, "The foreign language cocktail party problem: Energetic and informational masking effects in non-native speech perception, " J. Acoust. Soc. Amer., vol. 123, pp. 414-427, 2008.
32. B. R. Glasberg and B. C. J. Moore, "Derivation of auditory filter shapes from notched-noise data, " Hearing Res., vol. 47, pp. 103-138, 1990.
33. X. Shao and J. Barker, "Stream weight estimation for multistream audio-visual speech recognition in a multispeaker environment, " Speech Commun., vol. 50, no. 4, pp. 337-353, 2008.
34. G. R. Bradski, "Computer video face tracking for use in a perceptual user interface, " Intel Technol. J., Q2, 1998.
35. N. Ma, P. Green, J. Barker, and A. Coy, "Exploiting correlogram structure for robust automatic speech recognition with multiple speech sources, " Speech Commun., vol. 49, no. 12, pp. 874-891, 2007.
36. R. C. Gonzales, R. E. Woods, and S. L. Eddins, Digital Image Process.ing Using MATLAB. Upper Saddle River, NJ: Prentice-Hall, 2004.
37. G. Gravier, G. Potamianos, and C. Neti, "Asynchrony modeling for audio-visual speech recognition, " in Proc. Human Lang. Technol. Conf., San Diego, CA, 2002, pp. 24-27.