Mechanisms of enhancing visual-speech recognition by prior auditory information

NeuroImage

Volumn 65, Issue , 2013, Pages 109-118

  Blank, Helen ^a   von Kriegstein, Katharina ^a  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

Author keywords

FMRI; Lip reading; Multisensory; Predictive coding; Speech reading

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; AUDITORY SYSTEM FUNCTION; BRAIN FUNCTION; BRAIN REGION; CONTROLLED STUDY; ELECTROENCEPHALOGRAM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; MALE; NORMAL HUMAN; PATTERN RECOGNITION; PRIORITY JOURNAL; SIGNAL TRANSDUCTION; SPEECH DISCRIMINATION; SUPERIOR TEMPORAL SULCUS; TASK PERFORMANCE; VISUAL MEMORY; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ADULT; BRAIN; BRAIN MAPPING; FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; PHOTIC STIMULATION; RECOGNITION (PSYCHOLOGY); SPEECH PERCEPTION; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84868250566     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2012.09.047     Document Type: Article

References (58)

1. Alegria J., Charlier B.L., Mattys S. The role of lip-reading and cued speech in the processing of phonological information in French-educated deaf children. Eur. J. Cogn. Psychol. 1999, 11:451-472.
2. Altieri N.A., Pisoni D.B., Townsend J.T. Some normative data on lip-reading skills. J. Acoust. Soc. Am. 2011, 130:1-4.
3. Amedi A., von Kriegstein K., van Atteveldt N.M., Beauchamp M.S., Naumer M.J. Functional imaging of human crossmodal identification and object recognition. Exp. Brain Res. 2005, 166:559-571.
4. Arnal L.H., Morillon B., Kell C.A., Giraud A.L. Dual neural routing of visual facilitation in speech processing. J. Neurosci. 2009, 29:13445-13453.
5. Arnal L.H., Wyart V., Giraud A.-L. Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nat. Neurosci. 2011, 14:797-801.
6. Baart M., Vroomen J. Do you see what you are hearing? Cross-modal effects of speech sounds on lipreading. Neurosci. Lett. 2010, 471:100-103.
7. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat. Neurosci. 2004, 7:1190-1192.
8. Beauchamp M.S., Lee K.E., Argall B.D., Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 2004, 41:809-823.
9. Beauchamp M.S., Nath A.R., Pasalar S. fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J. Neurosci. 2010, 30:2414-2417.
10. Besle J., Fischer C., Bidet-Caulet A., Lecaignard F., Bertrand O., Giard M.H. Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J. Neurosci. 2008, 28:14301-14310.
11. Brown S., Ngan E., Liotti M. A larynx area in the human motor cortex. Cereb. Cortex 2008, 18:837-845.
12. Buchsbaum B.R., Hickok G., Humphries C. Role of left posterior superior temporal gyrus in phonological processing for speech perception and production. Cognit. Sci. 2001, 25:663-678.
13. Calvert G.A., Campbell R. Reading speech from still and moving faces: the neural substrates of visible speech. J. Cogn. Neurosci. 2003, 15:57-70.
14. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C., McGuire P.K., Woodruff P.W., Iversen S.D., David A.S. Activation of auditory cortex during silent lipreading. Science 1997, 276:593-596.
15. Campbell R., MacSweeney M., Surguladze S., Calvert G., McGuire P., Suckling J., Brammer M.J., David A.S. Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cogn. Brain Res. 2001, 12:233-243.
16. Chainay H., Krainik A., Tanguy M.L., Gerardin E., Le Bihan D., Lehericy S. Foot, face and hand representation in the human supplementary motor area. Neuroreport 2004, 15:765-769.
17. Dahl C.D., Logothetis N.K., Kayser C. Spatial organization of multisensory responses in temporal association cortex. J. Neurosci. 2009, 29:11924-11932.
18. DeWitt I., Rauschecker J.P. Phoneme and word recognition in the auditory ventral stream. Proc. Natl. Acad. Sci. 2012, 109:E505-E514.
19. Driver J., Noesselt T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 2008, 57:11-23.
20. Eickhoff S.B., Stephan K.E., Mohlberg H., Grefkes C., Fink G.R., Amunts K., Zilles K. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 2005, 25:1325-1335.
21. Friston K., Kiebel S. Predictive coding under the free-energy principle. Philos. Trans. R. Soc. B-Biol. Sci. 2009, 364:1211-1221.
22. Friston K.J., Buechel C., Fink G.R., Morris J., Rolls E., Dolan R.J. Psychophysiological and modulatory interactions in neuroimaging. Neuroimage 1997, 6:218-229.
23. Friston K.J., Zarahn E., Josephs O., Henson R.N.A., Dale A.M. Stochastic designs in event-related fMRI. Neuroimage 1999, 10:607-619.
24. Friston K.J., Rotshtein P., Geng J.J., Sterzer P., Henson R.N. A critique of functional localisers. Neuroimage 2006, 30:1077-1087.
25. Statistical Parametric Mapping: The Analysis of Functional Brain Images 2007, Academic Press. K. Friston, A. Ashburner, S. Kiebel, T. Nichols, W. Penny (Eds.).
26. Ghazanfar A.A., Maier J.X., Hoffman K.L., Logothetis N.K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 2005, 25:5004-5012.
27. Giraud A.L., Price C.J., Graham J.M., Truy E., Frackowiak R.S. Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron 2001, 30:657-663.
28. Gregory J.F. An investigation of speechreading with and without cued speech. Am. Ann. Deaf 1987, 132:393-398.
29. Hall D.A., Fussell C., Summerfield A.Q. Reading fluent speech from talking faces: typical brain networks and individual differences. J. Cogn. Neurosci. 2005, 17:939-953.
30. Hasson U., Skipper J.I., Nusbaum H.C., Small S.L. Abstract coding of audiovisual speech: beyond sensory representation. Neuron 2007, 56:1116-1126.
31. Haxby J.V., Hoffman E.A., Gobbini M.I. The distributed human neural system for face perception. Trends Cogn. Sci. 2000, 4:223-233.
32. Hein G., Knight R.T. Superior temporal sulcus-it's my area: or is it?. J. Cogn. Neurosci. 2008, 20:2125-2136.
33. Hickok G., Poeppel D. The cortical organization of speech processing. Nat. Rev. Neurosci. 2007, 8:393-402.
34. Hocking J., Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cereb. Cortex 2008, 18:2439-2449.
35. Nath A.R., Beauchamp M.S. Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. J. Neurosci. 2011, 31:1704-1714.
36. Nishitani N., Hari R. Viewing lip forms: cortical dynamics. Neuron 2002, 36:1211-1220.
37. Noesselt T., Tyll S., Boehler C.N., Budinger E., Heinze H.J., Driver J. Sound-induced enhancement of low-intensity vision: multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptual enhancement of visual detection sensitivity. J. Neurosci. 2010, 30:13609-13623.
38. Noppeney U., Ostwald D., Werner S. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J. Neurosci. 2010, 30:7434-7446.
39. Obleser J., Wise R.J.S., Dresner M.A., Scott S.K. Functional integration across brain regions improves speech perception under adverse listening conditions. J. Neurosci. 2007, 27:2283-2289.
40. Okada K., Hickok G. Two cortical mechanisms support the integration of visual and auditory speech: a hypothesis and preliminary data. Neurosci. Lett. 2009, 452:219-223.
41. Oldfield R.C. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 1971, 9:97-113.
42. O'Reilly J.X., Woolrich M.W., Behrens T.E.J., Smith S.M., Johansen-Berg H. Tools of the trade: psychophysiological interactions and functional connectivity. Soc. Cogn. Affect. Neurosci. 2012, 7:604-609.
43. Puce A., Allison T., Bentin S., Gore J.C., McCarthy G. Temporal cortex activation in humans viewing eye and mouth movements. J. Neurosci. 1998, 18:2188-2199.
44. Rao R.P., Ballard D.H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 1999, 2:79-87.
45. Ross L.A., Saint-Amour D., Leavitt V.M., Javitt D.C., Foxe J.J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environment. Cereb. Cortex 2007, 17:1147-1153.
46. Rouger J., Lagleyre S., Fraysse B., Deneve S., Deguine O., Barone P. Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc. Natl. Acad. Sci. U. S. A. 2007, 104:7295-7300.
47. Sherbecoe R.L., Studebaker G.A. Supplementary formulas and tables for calculating and interconverting speech recognition scores in transformed arcsine units. Int. J. Audiol. 2004, 43:554.
48. Skipper J.I., Nusbaum H.C., Small S.L. Listening to talking faces: motor cortical activation during speech perception. Neuroimage 2005, 25:76-89.
49. Skipper J.I., van Wassenhove V., Nusbaum H.C., Small S.L. Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb. Cortex 2007, 17:2387-2399.
50. Stevenson R.A., James T.W. Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 2009, 44:1210-1223.
51. Strelnikov K., Rouger J., Lagleyre S., Fraysse B., Deguine O., Barone P. Improvement in speech-reading ability by auditory training: evidence from gender differences in normally hearing, deaf and cochlear implanted subjects. Neuropsychologia 2009, 47:972-979.
52. Studebaker G.A. A rationalized Arcsine transform. J. Speech Hear. Res. 1985, 28:455-462.
53. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 1954, 26:212-215.
54. Summerfield Q. Lipreading and audiovisual speech-perception. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1992, 335:71-78.
55. Vaden K.I., Muftuler L.T., Hickok G. Phonological repetition-suppression in bilateral superior temporal sulci. Neuroimage 2010, 49:1018-1023.
56. van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. U. S. A. 2005, 102:1181-1186.
57. von Kriegstein K., Dogan O., Gruter M., Giraud A.L., Kell C.A., Gruter T., Kleinschmidt A., Kiebel S.J. Simulation of talking faces in the human brain improves auditory speech recognition. Proc. Natl. Acad. Sci. U. S. A. 2008, 105:6747-6752.
58. Wilson S.M., Saygin A.P., Sereno M.I., Iacoboni M. Listening to speech activates motor areas involved in speech production. Nat. Neurosci. 2004, 7:701-702.