Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments

Cerebral Cortex

Volumn 17, Issue 5, 2007, Pages 1147-1153

  Ross, Lars A ^a,b   Saint Amour, Dave ^b   Leavitt, Victoria M ^b,c   Javitt, Daniel C ^a,b   Foxe, John J ^a,b,c  

^a CITY COLLEGE OF NEW YORK   (United States)

^b NATHAN S KLINE INSTITUTE FOR PSYCHIATRIC RESEARCH   (United States)

^c CITY UNIVERSITY OF NEW YORK   (United States)

Author keywords

Audiovisual; Crossmodal; Inverse effectiveness; Lip reading; Multisensory; Speech perception; Speech reading

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; AUDITORY SYSTEM; COMPREHENSION; CONTROLLED STUDY; FEMALE; HUMAN; HUMAN EXPERIMENT; MALE; NOISE; NORMAL HUMAN; PRIORITY JOURNAL; RECOGNITION; SENSORY STIMULATION; SIGNAL NOISE RATIO; SPEECH PERCEPTION; VISION;

ADOLESCENT; ADULT; ARTIFACTS; AUDITORY PERCEPTION; COGNITION; FACIAL EXPRESSION; FEMALE; HUMANS; MALE; MIDDLE AGED; SPEECH PERCEPTION; TASK PERFORMANCE AND ANALYSIS; VISUAL PERCEPTION;

EID: 34247172408     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhl024     Document Type: Article

References (34)

1. Bernstein LE, Auer ET, Moore JK. 2004. Audiovisual speech binding: convergence or association? In: Calvert GA, Spence C, Stein BE, editors. The handbook of multisensory processes. Cambridge, MA: MIT Press. p 203-223.
2. Binnie CA, Montgomery A, Jackson PL. 1974. Auditory and visual contributions to the perception of consonants. J Speech Hear Res 17:616.
3. Callan DE, Callan AM, Kroos C, Vatikiotis-Bateson E. 2001. Multimodal contribution to speech perception revealed by independent component analysis: a single-sweep EEG case study. Brain Res Cogn Brain Res 10:349-353.
4. Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E. 2003. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213-2218.
5. Calvert G, Lewis WL. 2004. Hemodynamic studies of audiovisual interactions. In: Calvert GA, Spence C, Stein BE, editors. The handbook of multisensory processes. Cambridge, MA: MIT Press. p 203-223.
6. Calvert GA, Campbell R. 2003. Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 15:57-70.
7. Calvert GA, Campbell R, Brammer MJ. 2000. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 1011:649-657.
8. Erber NP. 1969. Interaction of audition and vision in the recognition of oral speech stimuli. J Speech Hear Res 12:423-425.
9. Erber NP. 1971. Auditory and audiovisual reception of words in low-frequency noise by children with normal hearing and by children with impaired hearing. J Speech Hear Res 143:496-512.
10. Erber NP. 1975. Auditory-visual perception in speech. J Speech Hear Dis 40:481-492.
11. Foxe JJ, Morocz IA, Higgins BA, Murray MA, Javitt DC, Schroeder CE. 2000. Multisensory auditory-somatosensory interactions in early cortical processing. Brain Res Cogn Brain Res 10:77-83.
12. Foxe JJ, Schroeder CE. 2005. The case for a feedforward component in multisensory integration mechanisms. Neuroreport 16:419-423.
13. Grant KW, Braida LD. 1991. Evaluating the articulation index for auditory-visual input. J Acoust Soc Am 89:2952-2960.
14. Grant KW, Seitz PF. 2000. The use of visible speech cues for improving auditory detection of spoken sentences. J Acoust Soc Am 108:1197-1208.
15. Grant KW, Walden BE. 1996. Evaluating the articulation index for auditory-visual consonant recognition. J Acoust Soc Am 100:2415-2424.
16. Kucera H, Francis WN. 1967. Computational analysis of present-day American English. Providence, RI: Brown University Press.
17. MacLeod A, Summerfield Q. 1987. Quantifying the contribution of vision to speech perception in noise. Br J Audiol 21:131-141.
18. Massaro DW. 1987. Speech perception by ear and eye: a paradigm for psychological inquiry. Hillsdale, NJ: Erlbaum.
19. McCormick B. 1979. Audio-visual discrimination of speech. Clin Otolaryngol Allied Sci 45:355-361.
20. McGurk H, MacDonald JW. 1976. Hearing lips and seeing voices. Nature 264:746-748.
21. Meredith MA, Stein BE. 1986. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res Cogn Brain Res 369:350-354.
22. Molholm S, Ritter W, Javitt DC, Foxe JJ. 2004. Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 144:452-465.
23. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ. 2002. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:121-134.
24. Munhall KG, Jones JA, Callan DE, Kuratate T, Vatikiotis-Bateson E. 2004. Visual prosody and speech intelligibility: head movement improves auditory speech perception. Psychol Sci 152:133-137.
25. Murray MM, Molholm S, Michel CM, Ritter W, Heslenfeld DJ, Schroeder CE, Javitt DC, Foxe JJ. 2005. Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 157:963-974.
26. Neely KK. 1956. Effect of visual factors on the intelligibility of speech. J Acoust Soc Am 26:212.
27. O'Neill JJ. 1954. Contributions of the visual component of oral symbols to speech comprehension. J Speech Hear Dis 19:429.
28. Schroeder CE, Foxe JJ. 2002. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res 14:195-207.
29. Schroeder CE, Foxe JJ. 2004. Multisensory convergence in early cortical processing. In: Calvert GA, Spence C, Stein BE, editors. The handbook of multisensory processes. Cambridge, MA: MIT Press. p 295-309.
30. Schroeder CE, Molholm S, Lakatos P, Ritter W, Foxe JJ. 2004. Human-simian correspondence in the early cortical processing of multisensory cues. Cogn Process 53:140-151.
31. Spence C, Driver J. 2000. Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism. Neuroreport 11:2057-2061.
32. Stein BE, Huneycutt WS, Meredith MA. 1988. Neurons and behavior: the same rules of multisensory integration. Brain Res Cogn Brain Res 448:355-358.
33. Stein BE, Meredith MA. 1993. The merging of the senses. Cambridge, MA: MIT Press.
34. Sumby WH, Pollack I. 1954. Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212-215.