Speech comprehension aided by multiple modalities: Behavioural and neural interactions

Neuropsychologia

Volumn 50, Issue 5, 2012, Pages 762-776

  McGettigan, Carolyn ^a   Faulkner, Andrew ^b   Altarelli, Irene ^b,c   Obleser, Jonas ^d   Baverstock, Harriet ^e   Scott, Sophie K ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c PSL RESEARCH UNIVERSITY   (France)

^d MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^e UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Auditory cortex; FMRI; Individual differences; Speech

Indexed keywords

ACCURACY; ADULT; AMYGDALOID NUCLEUS; ARTICLE; AUDITORY MEMORY; BRAIN FUNCTION; BRAIN REGION; CAUDATE NUCLEUS; CONTROLLED STUDY; FEMALE; FRONTAL CORTEX; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUSIFORM GYRUS; HUMAN; HUMAN EXPERIMENT; LANGUAGE PROCESSING; LINGUAL GYRUS; MALE; PRECUNEUS; PREDICTION; SCORING SYSTEM; SPEECH DISCRIMINATION; STIMULUS RESPONSE; TEMPORAL CORTEX; VISUAL INFORMATION;

ACOUSTIC STIMULATION; ADOLESCENT; ADULT; ANALYSIS OF VARIANCE; BRAIN MAPPING; COMPREHENSION; FEMALE; FUNCTIONAL LATERALITY; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LINGUISTICS; MAGNETIC RESONANCE IMAGING; MALE; NORMAL DISTRIBUTION; OXYGEN; PATTERN RECOGNITION, VISUAL; SPEECH; SPEECH PERCEPTION; TEMPORAL LOBE; YOUNG ADULT;

EID: 84858447247     PISSN: 00283932     EISSN: 18733514     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2012.01.010     Document Type: Article

References (117)

1. Adank P., Devlin J.T. On-line plasticity in spoken sentence comprehension: Adapting to time-compressed speech. NeuroImage 2010, 49:1124-1132.
2. Alario F.X., Chainay H., Lehericy S., Cohen L. The role of the supplementary motor area (SMA) in word production. Brain Research 2006, 1076:129-143.
3. Awad M., Warren J.E., Scott S.K., Turkheimer F.E., Wise R.J.S. A common system for the comprehension and production of narrative speech. Journal of Neuroscience 2007, 27:11455-11464.
4. Balsamo L.M., Xu B., Gaillard W.D. Language lateralization and the role of the fusiform gyrus in semantic processing in young children. NeuroImage 2006, 31:1306-1314.
5. Beauchamp M.S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 2005, 3:93-113.
6. Beauchamp M.S., Lee K.E., Argall B.D., Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 2004, 41:809-823.
7. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., Martin A. Unravelling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neuroscience 2004, 7:1190-1192.
8. Beauchamp M.S., Nath A.R., Pasalar S. fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. Journal of Neuroscience 2010, 30:2414-2417.
9. Bernstein L.E., Auer E.T., Takayanagi S. Auditory speech detection in noise enhanced by lipreading. Speech Communication 2004, 44:5-18.
10. Bernstein L.E., Auer E.T., Wagner M., Ponton C.W. Spatio-temporal dynamics of audiovisual speech processing. NeuroImage 2008, 39:423-435.
11. Bernstein L.E., Lub Z., Jianga J. Quantified acoustic optical speech signal incongruity identifies cortical sites of audiovisual speech processing. Brain Research 2008, 1242:172-184.
12. Bernstein L.E., Jiang J., Pantazis D., Lu Z.L., Joshi A. Visual phonetic processing localized using speech and nonspeech face gestures in video and point-light displays. Human Brain Mapping 2011, 32:1660-1676.
13. Binder J.R., Frost J.A., Hammeke T.A., Cox R.W., Rao S.M., Prieto T. Human brain language areas identified by functional magnetic resonance imaging. Journal of Neuroscience 1997, 17:353-362.
14. Bishop C.W., Miller L.M. A multisensory cortical network for understanding speech in noise. Journal of Cognitive Neuroscience 2009, 21:1790-1804.
15. Blank S.C., Scott S.K., Murphy K., Warburton E., Wise R.J.S. Speech production: Wernicke, Broca and beyond. Brain 2002, 125:1829-1838.
16. Boersma, P., & Weenink, D. (2008). Praat: Doing phonetics by computer. Downloaded from: http://www.fon.hum.uva.nl/praat.
17. Brainard D.H. The psychophysics toolbox. Spatial Vision 1997, 10:433-436.
18. Brefczynski-Lewis J., Lowitszch S., Parsons M., Lemieux S., Puce A. Audiovisual non-verbal dynamic faces elicit converging fMRI and ERP responses. Brain Topography 2009, 21:193-206.
19. Brett M., Anton J., Valabregue R., Poline J. Region of interest analysis using an SPM toolbox. Presented at the 8th international conference on functional mapping of the human brain 2002.
20. Büchel C., Price C., Frackowiak R.S.J., Friston K. Different activation patterns in the visual cortex of late and congenitally blind subjects. Brain 1998, 121:409-419.
21. Buckner R.L., Andrews-Hanna J.R., Schacter D.L. The brain's default network: Anatomy, function, and relevance to disease. Annals of the New York Academy of Sciences 2008, 1124:1-38.
22. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. NeuroReport 2003, 14:2213-2218.
23. Calvert G.A. Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 2001, 11:1110-1123.
24. Calvert G.A., Campbell R. Reading speech from still and moving faces: The neural substrates of visible speech. Journal of Cognitive Neuroscience 2003, 15:57-70.
25. Calvert G.A., Campbell R., Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 2000, 10:649-657.
26. Chee M.W.L., O'Craven K.M., Bergida R., Rosen B.R., Savoy R.L. Auditory and visual word processing studied with fMRI. Human Brain Mapping 1999, 7:15-28.
27. Crinion J., Turner R., Grogan A., Hanakawa T., Noppeney U., Devlin J.T., et al. Language control in the bilingual brain. Science 2006, 312:1537-1540.
28. Davis M.H., Johnsrude I.S. Hierarchical processing in spoken language comprehension. Journal of Neuroscience 2003, 23:3423-3431.
29. Davis M.H., Johnsrude I.S. Hearing speech sounds: Top-down influences on the interface between audition and speech perception. Hearing Research 2007, 229:132-147.
30. Dick A.S., Solodkin A., Small S.L. Neural development of networks for audiovisual speech comprehension. Brain and Language 2010, 114:101-114.
31. Dien J., Frishkoff G.A., Cerbone A., Tucker D.M. Parametric analysis of event-related potentials in semantic comprehension: Evidence for parallel brain mechanisms. Cognitive Brain Research 2003, 15:137-153.
32. Dubno J.R., Ahlstrom J.B., Horwitz A.R. Use of context by young and aged adults with normal hearing. Journal of the Acoustical Society of America 2000, 107:538-546.
33. Edmister W.B., Talavage T.M., Ledden P.J., Weisskoff R.M. Improved auditory cortex imaging using clustered volume acquisitions. Human Brain Mapping 1999, 7:89-97.
34. Eickhoff S.B., Stephan K.E., Mohlberg H., Grefkes C., Fink G.R., Amunts K., et al. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage 2005, 25:1325-1335.
35. Eisner F., McGettigan C., Faulkner A., Rosen S., Scott S.K. Inferior frontal gyrus activation predicts individual differences in perceptual learning of cochlear-implant simulations. Journal of Neuroscience 2010, 30:7179-7186.
36. Friederici A.D. What's in control of language?. Nature Neuroscience 2006, 9:991-992.
37. Ghazanfar A.A., Maier J.X., Hoffman K.L., Logothetis N.K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience 2005, 25:5004-5012.
38. Girin L., Schwartz J.L., Feng G. Audio-visual enhancement of speech in noise. Journal of the Acoustical Society of America 2001, 109:3007-3020.
39. Goodale M.A., Milner A.D. Separate visual pathways for perception and action. Trends in Neurosciences 1992, 15:20-25.
40. Grant K.W., Seitz P.F. The recognition of isolated words and words in sentences: Individual variability in the use of sentence context. Journal of the Acoustical Society of America 2000, 107:1000-1011.
41. Grant K.W., Seitz P.F. The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America 2000, 108:1197-1208.
42. Green A., Straube B., Weis S., Jansen A., Willmes K., Konrad K., et al. Neural integration of iconic and unrelated coverbal gestures: A functional MRI study. Human Brain Mapping 2009, 30:3309-3324.
43. Greenwood D.D. A cochlear frequency-position function for several species - 29 years later. Journal of the Acoustical Society of America 1990, 87:2592-2605.
44. Hall D.A., Haggard M.P., Akeroyd M.A., Palmer A.R., Summerfield A.Q., Elliott M.R., et al. Sparse temporal sampling in auditory fMRI. Human Brain Mapping 1999, 3:213-223.
45. Hall D.A., Fussell C., Summerfield A.Q. Reading fluent speech from talking faces: Typical brain networks and individual differences. Journal of Cognitive Neuroscience 2005, 17:939-953.
46. Hasson U., Skipper J.I., Nusbaum H.C., Small S.L. Abstract coding of audiovisual speech: Beyond sensory representation. Neuron 2007, 56:1116-1126.
47. Haxby J.V., Hoffman E.A., Gobbini M.I. The distributed human neural system for face perception. Trends in Cognitive Sciences 2000, 4:223-233.
48. Hazan V., Kim J., Chen Y. Audiovisual perception in adverse conditions: Language, speaker and listener effects. Speech Communication 2010, 52:996-1009.
49. Helfer K.S., Freyman R.L. The role of visual speech cues in reducing energetic and informational masking. Journal of the Acoustical Society of America 2005, 117:842-849.
50. Hickok G., Poeppel D. Opinion - The cortical organization of speech processing. Nature Reviews Neuroscience 2007, 8:393-402.
51. Holle H., Obleser J., Rueschmeyer S.A., Gunter T.C. Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. NeuroImage 2010, 49:875-884.
52. James T.W., Stevenson R.A. The use of fMRI to assess multisensory integration. Frontiers in the neural basis of multisensory processes 2011, Taylor & Francis, London. M.H. Wallace, M.M. Murray (Eds.).
53. Jones J.A., Callan D.E. Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport 2003, 14:1129-1133.
54. Kalikow D.N., Stevens K.N., Elliott L.L. Development of a test of speech-intelligibility in noise using sentence materials with controlled word predictability. Journal of the Acoustical Society of America 1977, 61:1337-1351.
55. Kanwisher N. Functional specificity in the human brain: A window into the functional architecture of the mind. Proceedings of the National Academy of Sciences United States of America 2010, 107:11163-11170.
56. Kassuba T., Klinge C., Holig C., Menz M.M., Ptito M., Roder B., et al. The left fusiform gyrus hosts trisensory representations of manipulable objects. NeuroImage 2011, 56:1566-1577.
57. Kawase T., Yamaguchi K., Ogawa T., Suzuki K., Suzuki M., Itoh M., et al. Recruitment of fusiform face area associated with listening to degraded speech sounds in auditory-visual speech perception: A PET study. Neuroscience Letters 2005, 382:254-258.
58. Keil J., Muller N., Ihssen N., Weisz N. On the variability of the McGurk effect: Audiovisual integration depends on prestimulus brain states. Cerebral Cortex 2012, 22:221-231.
59. Kim J., Davis C. Investigating the audio-visual speech detection advantage. Speech Communication 2004, 44:19-30.
60. Kim J., Davis C., Groot C. Speech identification in noise: Contribution of temporal, spectral, and visual speech cues. Journal of the Acoustical Society of America 2009, 126:3246-3257.
61. Kircher T., Straube B., Leube D., Weis S., Sachs O., Wilmes K., et al. Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures. Neuropsychologia 2009, 47:169-179.
62. Kuperberg G.R., McGuire P.K., Bullmore E.T., Brammer M.J., Rabe-Hesketh S., Wright I.C., et al. Common and distinct neural substrates for pragmatic, semantic, and syntactic processing of spoken sentences: An fMRI Study. Journal of Cognitive Neuroscience 2000, 12:321-341.
63. Laurienti P.J., Perrault T.J., Stanford T.R., Wallace M.T., Stein B.E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Experimental Brain Research 2005, 166:289-297.
64. Leech R., Kamourieh S., Beckmann C.F., Sharp D.J. Fractionating the default mode network: Distinct contributions of the ventral and dorsal posterior cingulate cortex to cognitive control. Journal of Neuroscience 2011, 31:3217-3224.
65. Lin P., Hasson U., Jovicich J., Robinson S. A neuronal basis for task-negative responses in the human brain. Cerebral Cortex 2011, 21:821-830.
66. Lotto A.J., Hickok G.S., Holt L.L. Reflections on mirror neurons and speech perception. Trends in Cognitive Sciences 2009, 13:110-114.
67. Love S.A., Pollick F.E., Latinus M. Cerebral correlates and statistical criteria of cross-modal face and voice integration. Seeing and Perceiving 2011, 24:351-367.
68. Luders H., Lesser R.P., Hahn J., et al. Basal temporal language area demonstrated by electrical stimulation. Neurology 1986, 36:505-510.
69. Ma W.J., Zhou X., Ross L.A., Foxe J.J., Parra L.C. Lip-reading aids word recognition most in moderate noise: A Bayesian explanation using high-dimensional feature space. PLoS One 2009, 4. doi:10.1371/journal.pone.0004638.
70. MacSweeney M., Woll B., Campbell R., McGuire P.K., David A.S., Williams S.C.R., et al. Neural systems underlying British Sign Language and audio-visual English processing in native users. Brain 2002, 125:1583-1593.
71. McGettigan C., Evans S., Rosen S., Agnew Z.K., Shah P., Scott S.K. An application of univariate and multivariate approaches in fMRI to quantifying the hemispheric lateralization of acoustic and linguistic processes. Journal of Cognitive Neuroscience 2011, [Epub ahead of print].
72. Miller L.M., D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 2005, 25:5884-5893.
73. Miller G.A., Isard S. Some perceptual consequences of linguistic rules. Journal of Verbal Learning and Verbal Behaviour 1963, 2:217-228.
74. Narain C., Scott S.K., Wise R.J.S., Rosen S., Leff A., Iversen S.D., et al. Defining a left-lateralized response specific to intelligible speech using fMRI. Cerebral Cortex 2003, 13:1362-1368.
75. Nath A.R., Beauchamp M.S. Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience 2011, 31:1704-1714.
76. Nath A.R., Beauchamp M.S. A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage 2012, 59:781-787.
77. Nath A.R., Fava E.E., Beauchamp M.S. Neural correlates of interindividual differences in children's audiovisual speech perception. Journal of Neuroscience 2011, 31:1963-1971.
78. Obleser J., Kotz S.A. Expectancy constraints in degraded speech modulate the language comprehension network. Cerebral Cortex 2010, 20:633-640.
79. Obleser J., Kotz S.A. Multiple brain signatures of integration in the comprehension of degraded speech. NeuroImage 2011, 55:713-723.
80. Obleser J., Wise R.J.S., Dresner M.A., Scott S.K. Functional integration across brain regions improves speech perception under adverse listening conditions. Journal of Neuroscience 2007, 27:2283-2289.
81. Obleser J., Eisner F., Kotz S.A. Bilateral speech comprehension reflects differential sensitivity to spectral and temporal features. Journal of Neuroscience 2008, 28:8116-8123.
82. Ojanen V., Mottonen R., Pekkola J., Jaaskelainen I.P., Joensuu R., Autti T., et al. Processing of audiovisual speech in Broca's area. NeuroImage 2005, 25:333-338.
83. Okada K., Rong F., Venezia J., Matchin W., Hsieh I., Saberi K., et al. Hierarchical organization of human auditory cortex: Evidence from acoustic invariance in the response to intelligible speech. Cerebral Cortex 2010, 20:2486-2495.
84. Osnes B., Hugdahl K., Specht K. Effective connectivity analysis demonstrates involvement of premotor cortex during speech perception. NeuroImage 2011, 54:2437-2445.
85. Pekkola J., Laasonen M., Ojanen V., Autti T., Jaaskelainen I.P., Kujala T., et al. Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: An fMRI study at 3T. NeuroImage 2006, 29:797-807.
86. Pichora-Fuller M.K., Schneider B.A., Daneman M. How young and old adults listen to and remember speech in noise. Journal of the Acoustical Society of America 1995, 97:593-608.
87. Reale R.A., Calvert G.A., Thesen T., Jenison R.L., Kawasaki H., Oya H., et al. Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 2007, 145:162-184.
88. Rosen S., Wise R.J.S., Chadha S., Conway E.J., Scott S.K. Hemispheric asmmetries in speech perception: Sense, nonsense and modulations. PLoS One 2011, 6:e24672. [Epub].
89. Ross L.A., Saint-Amour D., Leavitt V.M., Javitt D.C., Foxe J.J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environment. Cerebral Cortex 2007, 17:1147-1153.
90. Sato M., Tremblay P., Gracco V.L. A mediating role of the premotor cortex in phoneme segmentation. Brain and Language 2009, 111:1-7.
91. Schwartz J.L., Berthommier F., Savariaux C. Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition 2004, 93:B69-B78.
92. Scott S.K., Wise R.J.S. The functional neuroanatomy of prelexical processing in speech perception. Cognition 2004, 92:13-45.
93. Scott S.K., Blank C.C., Rosen S., Wise R.J.S. Identification of a pathway for intelligible speech in the left temporal lobe. Brain 2000, 123:2400-2406.
94. Scott S.K., Rosen S., Wickham L., Wise R.J.S. A positron emission tomography study of the neural basis of informational and energetic masking efforts in speech perception. Journal of the Acoustical Society of America 2004, 115:813-821.
95. Scott S.K., Rosen S., Lang H., Wise R.J.S. Neural correlates of intelligibility in speech investigated with noise vocoded speech - A positron emission tomography study. Journal of the Acoustical Society of America 2006, 120:1075-1083.
96. Scott S.K., McGettigan C., Eisner F. A little more conversation, a little less action - Candidate roles for the motor cortex in speech perception. Nature Reviews Neuroscience 2009, 10:295-302.
97. Scott, S.K., Rosen, S., Spitsyna, G., Faulkner, A., Neville, L., Wise, R.J.S. (2002). The neural basis of cross modal enhancement in speech perception: A pet study. Society for Neuroscience Abstract Viewer and Itinerary Planner, Abstract No. 354.17.
98. Sekiyama K., Kanno I., Miura S., Sugita Y. Auditory-visual speech perception examined by fMRl and PET. Neuroscience Research 2003, 47:277-287.
99. Senkowski D., Saint-Amour D., Hoefle M., Foxe J.J. Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. NeuroImage 2011, 56:2200-2208.
100. Shannon R.V., Zeng F.G., Kamath V., Wygonski J., Ekelid M. Speech recognition with primarily temporal cues. Science 1995, 270:303-304.
101. Sharp D.J., Scott S.K., Wise R.J.S. Retrieving meaning after temporal lobe infarction: The role of the basal language area. Annals of Neurology 2004, 56:836-846.
102. Skipper J.I., Nusbaum H.C., Small S.L. Listening to talking faces: Motor cortical activation during speech perception. NeuroImage 2005, 25:76-89.
103. Slotnick S.D., Moo L.R., Segal J.B., Hart J. Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Cognitive Brain Research 2003, 17:75-82.
104. Spitsyna G., Warren J.E., Scott S.K., Turkheimer F.E., Wise R.J.S. Converging language streams in the human temporal lobe. Journal of Neuroscience 2006, 26:7328-7336.
105. Stevenson R.A., James T.W. Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. NeuroImage 2009, 44:1210-1223.
106. Stevenson R.A., Kim S., James T.W. An additive-factors design to disambiguate neuronal and areal convergence: Measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Experimental Brain Research 2009, 198:183-194.
107. Stevenson R.A., Altieri N.A., Kim S., Pisoni D.B., James T.W. Neural processing of asynchronous audiovisual speech perception. NeuroImage 2010, 49:3308-3318.
108. Stickney G.S., Assmann P.F. Acoustic and linguistic factors in the perception of bandpass-filtered speech. Journal of the Acoustical Society of America 2001, 109:1157-1165.
109. Straube B., Green A., Weis S., Chatterjee A., Kircher T. Memory effects of speech and gesture binding: Cortical and hippocampal activation in relation to subsequent memory performance. Journal of Cognitive Neuroscience 2009, 21:821-836.
110. Straube B., Green A., Bromberger B., Kircher T. The differentiation of iconic and metaphoric gestures: Common and unique integration processes. Human Brain Mapping 2011, 32:520-533.
111. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 1954, 26:212-215.
112. Thomas S.M., Pilling M. Benefits of facial and textual information in understanding of vocoded speech. Auditory-visual speech processing 2007 (AVSP2007) 2007, Accessed from: http://www.isca-speech.org/archive.
113. von Kriegstein K., Eger E., Kleinschmidt A., Giraud A.L. Modulation of neural responses to speech by directing attention to voices or verbal content. Cognitive Brain Research 2003, 17:48-55.
114. Werner S., Noppeney U. Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex 2009, 20:1829-1842.
115. Wright T.M., Pelphrey K.A., Allison T., McKeown M.J., McCarthy G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex 2003, 13:1034-1043.
116. Wyk B.C.V., Ramsay G.J., Hudac C.M., Jones W., Lin D., Klin A., et al. Cortical integration of audio-visual speech and non-speech stimuli. Brain and Cognition 2010, 74:97-106.
117. Yoncheva Y.N., Zevin J.D., Maurer U., McCandliss B.D. Auditory selective attention to speech modulates activity in the visual word form area. Cerebral Cortex 2010, 20:622-632.