Audiovisual integration during speech comprehension: An fMRI study comparing ROI-based and whole brain analyses

Human Brain Mapping

Volumn 30, Issue 7, 2009, Pages 1990-1999

  Szycik, Gregor R ^a,b   Jansma, Henk ^a   Münte, Thomas F ^a,c  

^a OTTO VON GUERICKE UNIVERSITY   (Germany)

^b HANNOVER MEDICAL SCHOOL   (Germany)

^c Center for Behavioral Brain Sciences   (Germany)

Author keywords

Audiovisual; fMRI; Multimodal processing; Speech

Indexed keywords

ADULT; ARTICLE; BRAIN DEPTH STIMULATION; COMPREHENSION; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEMISPHERE; HUMAN; HUMAN EXPERIMENT; INTERMETHOD COMPARISON; MALE; NEUROANATOMY; NEUROIMAGING; NORMAL HUMAN; PRIORITY JOURNAL; SPEECH;

ADULT; AUDITORY PERCEPTION; BRAIN; BRAIN MAPPING; COMPREHENSION; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LINEAR MODELS; MAGNETIC RESONANCE IMAGING; MALE; SPEECH; SPEECH PERCEPTION; TIME FACTORS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 67649660240     PISSN: 10659471     EISSN: None     Source Type: Journal    
DOI: 10.1002/hbm.20640     Document Type: Article

References (82)

1. Amunts K, Schleicher A, Burgel U, Mohlberg H, Uylings HB, Zilles K (1999): Broca's region revisited: Cytoarchitecture and intersubject variability. J Comp Neurol 412:319-341. (Pubitemid 29395368)
2. Arbib MA (2005): From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behav Brain Sci 28:105-124. (Pubitemid 41309443)
3. Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M (2006): Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Curr Biol 16:1818-1823. (Pubitemid 44354143)
4. Baayen RH, Piepenbrock R, Gulikers L (1995): The CELEX Lexical Database [CD-ROM]. Philadelphia: University of Pennsylvania, Linguistic Data Consortium.
5. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI (2005): Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J Cogn Neurosci 17:377-391.
6. Baumgart F, Kaulisch T, Tempelmann C, Gaschler-Markefski B, Tegeler C, Schindler F, Stiller D, Scheich H (1998): Electrodynamic headphones and woofers for application in magnetic resonance imaging scanners. Med Phys 25:2068-2070. (Pubitemid 28485231)
7. Beauchamp MS (2005): See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15:145-153. (Pubitemid 41569262)
8. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a): Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190-1192. (Pubitemid 39426237)
9. Beauchamp MS, Lee KE, Argall BD, Martin A (2004b): Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809-823. (Pubitemid 38326842)
10. Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008): Spatio-temporal dynamics of audiovisual speech processing. Neuroimage 39:423-435. (Pubitemid 350102222)
11. Bookheimer S (2002): Functional MRI of language: New approaches to understanding the cortical organization of semantic processing. Annu Rev Neurosci 25:151-188. (Pubitemid 34748015)
12. Boynton GM, Engel SA, Glover GH, Heeger DJ (1996): Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci 16:4207-4221. (Pubitemid 26192416)
13. Brechmann A, Baumgart F, Scheich H (2002): Sound-level-dependent representation of frequency modulations in human auditory cortex: A low-noise fMRI study. J Neurophysiol 87:423-433. (Pubitemid 34669960)
14. Brodmann K (1909): Vergleichende Lokalisationslehre der Großhirnrinde. Leipzig: Johann Ambrosius Barth.
15. Bushara KO, Grafman J, Hallett M (2001): Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21:300-304.
16. Callan DE, Callan AM, Kroos C, Vatikiotis-Bateson E (2001): Multimodal contribution to speech perception revealed by independent component analysis: A single-sweep EEG case study. Brain Res Cogn Brain Res 10:349-353. (Pubitemid 32146593)
17. Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E (2003): Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213-2218.
18. Calvert GA (2001): Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cereb Cortex 11:1110-1123. (Pubitemid 33108238)
19. Calvert GA, Thesen T (2004): Multisensory integration: Methodological approaches and emerging principles in the human brain. J Physiol Paris 98:191-205. (Pubitemid 39345371)
20. Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997): Activation of auditory cortex during silent lipreading. Science 276:593-596. (Pubitemid 27193683)
21. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999): Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619-2623. (Pubitemid 29418147)
22. Calvert GA, Campbell R, Brammer MJ (2000): Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649-657. (Pubitemid 30354700)
23. Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001): Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427-438. (Pubitemid 32709557)
24. Cannestra AF, Bookheimer SY, Pouratian N, O'Farrell A, Sicotte N, Martin NA, Becker D, Rubino G, Toga AW (2000): Temporal and topographical characterization of language cortices using intraoperative optical intrinsic signals. Neuroimage 12:41-54. (Pubitemid 30602062)
25. Carter CS, Botvinick MM, Cohen JD (1999): The contribution of the anterior cingulate cortex to executive processes in cognition. Rev Neurosci 10:49-57. (Pubitemid 29396546)
26. Carter CS, Braver TS, Barch DM, Botvinick MM, Noll D, Cohen JD (1998): Anterior cingulate cortex, error detection, and the online monitoring of performance. Science 280:747-749. (Pubitemid 28243351)
27. Carter CS, Macdonald AM, Botvinick M, Ross LL, Stenger VA, Noll D, Cohen JD (2000): Parsing executive processes: strategic vs. evaluative functions of the anterior cingulate cortex. Proc Natl Acad Sci USA 97:1944-1948. (Pubitemid 30118548)
28. Cherry EC (1953): Some experiments on the recognition of speech, with one and two ears. J Acoust Soc Am 25:975-979.
29. Corballis MC (2003): From mouth to hand: Gesture, speech, and the evolution of right-handedness. Behav Brain Sci 26:199-208. (Pubitemid 37369402)
30. Corbetta M, Miezin FM, Dobmeyer S, Shulman GL, Petersen SE (1991): Selective and divided attention during visual discriminations of shape, color, and speed: functional anatomy by positron emission tomography. J Neurosci 11:2383-2402.
31. Devergie A, Grimault N, Berthommier F, Gaudrain E, Healy EW (2008): Effect of lip movement cues on auditory streaming of concurrent speech. J Acoust Soc Am 123:3302.
32. Frens MA, Van Opstal AJ (1995): Spatial and temporal factors determine audio-visual interactions in human saccadic eye movements. Percept Psychophys 57:802-816.
33. Galaburda A, Sanides F (1980): Cytoarchitectonic organization of the human auditory cortex. J Comp Neurol 190:597-610. (Pubitemid 10080475)
34. Gelfand JR, Bookheimer SY (2003): Dissociating neural mechanisms of temporal sequencing and processing phonemes. Neuron 38:831-842. (Pubitemid 36694788)
35. Genovese CR, Lazar NA, Nichols T (2002): Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15:870-878. (Pubitemid 34852500)
36. Goebel R, Esposito F, Formisano E (2006): Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis. Hum Brain Mapp 27:392-401. (Pubitemid 43787637)
37. Hagoort P (2005): On Broca, brain, and binding: a new framework. Trends Cogn Sci 9:416-423. (Pubitemid 41208250)
38. Heiser M, Iacoboni M, Maeda F, Marcus J, Mazziotta JC (2003): The essential role of Broca's area in imitation. Eur J Neurosci 17:1123-1128. (Pubitemid 36411611)
39. Hershenson M (1962): Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289-293.
40. Hertrich I, Mathiak K, Lutzenberger W, Menning H, Ackermann H (2007): Sequential audiovisual interactions during speech perception: A whole-head MEG study. Neuropsychologia 45: 1342-1354. (Pubitemid 46072902)
41. Jones JA, Callan DE (2003): Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport 14:1129-1133. (Pubitemid 36930260)
42. Klucharev V, Mottonen R, Sams M (2003): Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Res Cogn Brain Res 18:65-75. (Pubitemid 37500976)
43. Liberman AM, Mattingly IG (1985): The motor theory of speech perception revised. Cognition 21:1-36.
44. Macaluso E, Frith C, Driver J (2000): Selective spatial attention in vision and touch: Unimodal and multimodal mechanisms revealed by PET. J Neurophysiol 83:3062-3075. (Pubitemid 30321494)
45. Macaluso E, George N, Dolan R, Spence C, Driver J (2004): Spatial and temporal factors during processing of audiovisual speech: A PET study. Neuroimage 21:725-732 (Pubitemid 38229935)
46. McGurk H, MacDonald J (1976): Hearing lips and seeing voices. Nature 264:746-748.
47. Meister IG, Boroojerdi B, Foltys H, Sparing R, Huber W, Topper R (2003): Motor cortex hand area and speech: Implications for the development of language. Neuropsychologia 41:401-406. (Pubitemid 36143669)
48. Miller LM, D'Esposito M (2005): Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884-5893. (Pubitemid 40911419)
49. Mitchell RL, CrowTJ (2005): Right hemisphere language functions and schizophrenia: The forgotten hemisphere? Brain 128:963-978.
50. Molnar-Szakacs I, Iacoboni M, Koski L, Mazziotta JC (2005): Functional segregation within pars opercularis of the inferior frontal gyrus: Evidence from fMRI studies of imitation and action observation. Cereb Cortex 15:986-994. (Pubitemid 40848067)
51. Morosan P, Rademacher J, Schleicher A, Amunts K, Schormann T, Zilles K (2001): Human primary auditory cortex: Cytoarchitectonic subdivisions and mapping into a spatial reference system. Neuroimage 13:684-701. (Pubitemid 34184991)
52. Morrell LK (1968): Temporal characteristics of sensory interaction in choice reaction times. J Exp Psychol 77:14-18.
53. Möttönen R, Schurmann M, Sams M (2004): Time course of multisensory interactions during audiovisual speech perception in humans: A magnetoencephalographic study. Neurosci Lett 363:112-115. (Pubitemid 38692600)
54. Nishitani N, Schurmann M, Amunts K, Hari R (2005): Broca's region: From action to language. Physiology (Bethesda) 20:60-69. (Pubitemid 40363372)
55. Ochiai T, Grimault S, Scavarda D, Roch G, Hori T, Riviere D, Mangin JF, Regis J (2004): Sulcal pattern and morphology of the superior temporal sulcus. Neuroimage 22:706-719. (Pubitemid 38759433)
56. Ojanen V, Mottonen R, Pekkola J, Jaaskelainen IP, Joensuu R, Autti T, Sams M (2005): Processing of audiovisual speech in Broca's area. Neuroimage 25:333-338. (Pubitemid 40387315)
57. Pandya DN (1995): Anatomy of the auditory cortex. Rev Neurol (Paris) 151:486-494.
58. Paus T (2001): Primate anterior cingulate cortex: Where motor control, drive and cognition interface. Nat Rev Neurosci 2:417-424. (Pubitemid 33674798)
59. Paus T, Koski L, Caramanos Z, Westbury C (1998): Regional differences in the effects of task difficulty and motor output on blood flow response in the human anterior cingulate cortex: A review of 107 PET activation studies. Neuroreport 9:R37-R47. (Pubitemid 28300540)
60. Pekkola J, Laasonen M, Ojanen V, Autti T, Jaaskelainen IP, Kujala T, Sams M (2006): Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: An fMRI study at 3 T. Neuroimage 29:797-807. (Pubitemid 43121205)
61. Pekkola J, Ojanen V, Autti T, Jaaskelainen IP, Mottonen R, Tarkiainen A, Sams M (2005): Primary auditory cortex activation by visual speech: An fMRI study at 3 T. Neuroreport 16:125-128. (Pubitemid 40250383)
62. Penhune VB, Zatorre RJ, MacDonald JD, Evans AC (1996): Interhemispheric anatomical differences in human primary auditory cortex: Probabilistic mapping and volume measurement from magnetic resonance scans. Cereb Cortex 6:661-672. (Pubitemid 26337837)
63. Pulvermüller F (2005): Brain mechanisms linking language and action. Nat Rev Neurosci 6:576-582. (Pubitemid 40942824)
64. Rademacher J, Morosan P, Schormann T, Schleicher A, Werner C, Freund HJ, Zilles K (2001): Probabilistic mapping and volume measurement of human primary auditory cortex. Neuroimage 13:669-683. (Pubitemid 34184990)
65. Raij T, Uutela K, Hari R (2000): Audiovisual integration of letters in the human brain. Neuron 28:617-625.
66. Reale RA, Calvert GA, Thesen T, Jenison RL, Kawasaki H, Oya H, Howard MA, Brugge JF (2007): Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 145:162-184. (Pubitemid 46227714)
67. Ridderinkhof KR, Ullsperger M, Crone EA, Nieuwenhuis S (2004): The role of the medial frontal cortex in cognitive control. Science 306:443-447. (Pubitemid 39372436)
68. Rizzolatti G, Arbib MA (1998): Language within our grasp. Trends Neurosci 21:188-194. (Pubitemid 28191810)
69. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007): Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147-1153. (Pubitemid 46598855)
70. Sekiyama K, Kanno I, Miura S, Sugita Y (2003): Auditory-visual speech perception examined by fMRI and PET. Neurosci Res 47:277-287. (Pubitemid 37267963)
71. Sekuler R, Sekuler AB, Lau L (1997): Sounds alter visual motion perception. Nature 385:308. (Pubitemid 27053094)
72. Semple MN, Scott BH (2003): Cortical mechanisms in hearing. Curr Opin Neurobiol 13:167-173. (Pubitemid 36556273)
73. Skipper JI, Nusbaum HC, Small SL (2005): Listening to talking faces: Motor cortical activation during speech perception. Neuroimage 25:76-89. (Pubitemid 40289139)
74. Skipper JI, van Wassenhove V, Nusbaum HC, Small SL (2007): Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cereb Cortex 17:2387-2399. (Pubitemid 47517474)
75. Stein BE, Meredith MA, Huneycutt WS, McDade L (1989): Behavioural indices of multisensory integration: Orientation to visual cues is affected by auditory stimuli. J Cogn Neurosci 1:12-24. (Pubitemid 20346080)
76. Sumby WH, Pollack I (1954): Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212-215.
77. Szycik GR, Tausche P, Münte TF (2007): A novel approach to study audiovisual integration in speech perception: Localizer fMRI and sparse sampling. Brain Res. [Epub ahead of print].
78. Talairach J, Tornoux P (1988): Co-Planar Stereotactic Atlas of the Human Brain. Stuttgart: Thieme.
79. Tokimura H, Tokimura Y, Oliviero A, Asakura T, Rothwell JC (1996): Speech-induced changes in corticospinal excitability. Ann Neurol 40:628-634. (Pubitemid 26357888)
80. Tomaiuolo F, MacDonald JD, Caramanos Z, Posner G, Chiavaras M, Evans AC, Petrides M (1999): Morphology, morphometry and probability mapping of the pars opercularis of the inferior frontal gyrus: An in vivo MRI analysis. Eur J Neurosci 11:3033-3046. (Pubitemid 29431639)
81. van Atteveldt N, Formisano E, Goebel R, Blomert L (2004): Integration of letters and speech sounds in the human brain. Neuron 43:271-282. (Pubitemid 38942966)
82. Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003): Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex 13:1034-1043. (Pubitemid 37163202)