Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex

Hearing Research

Volumn 258, Issue 1-2, 2009, Pages 143-151

  Besle, Julien ^a,b,c   Bertrand, Olivier ^a,b   Giard, Marie Hélène ^a,b  

^a INSERM   (France)

^b UNIVERSITÉ LYON 1   (France)

^c UNIVERSITÉ LUMIÈRE LYON 2   (France)

Author keywords

Auditory; ERPs; Human; Intracranial; Multisensory; Speech; Visual

Indexed keywords

ARTICLE; AUDITORY CORTEX; AUDITORY DISCRIMINATION; ELECTROENCEPHALOGRAPHY; HUMAN; INFORMATION PROCESSING; MAGNETOENCEPHALOGRAPHY; NERVOUS SYSTEM ELECTROPHYSIOLOGY; PRIORITY JOURNAL; SENSORY EVALUATION; SENSORY MEMORY; SOUND DETECTION; SPEECH DISCRIMINATION; TASK PERFORMANCE; VISUAL DISCRIMINATION;

AUDITORY CORTEX; AUDITORY PERCEPTION; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; HEARING; HUMANS; MAGNETOENCEPHALOGRAPHY; MODELS, NEUROLOGICAL; PERCEPTION; SOUND; SPEECH; TIME FACTORS; VISION, OCULAR; VISUAL PERCEPTION;

EID: 70450136233     PISSN: 03785955     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.heares.2009.06.016     Document Type: Article

References (71)

1. Aoyama A., Endo H., Honda S., and Takeda T. Modulation of early auditory processing by visually based sound prediction. Brain Res. 1068 (2006) 194-204
2. Besle, J., Giard, M.H., submitted for publication. Representation of visual information in the auditory sensory memory.
3. Besle J., Fort A., and Giard M.H. Interest and validity of the additive model in electrophysiological studies of multisensory interactions. Cogn. Process. 5 (2004) 189-192
4. Besle J., Fort A., Delpuech C., and Giard M.H. Bimodal speech: Early suppressive visual effects in the human auditory cortex. Eur. J. Neurosci. 20 (2004) 2225-2234
5. Besle J., Fort A., and Giard M.H. Is the auditory sensory memory sensitive to visual information?. Exp. Brain Res. 166 (2005) 337-344
6. Besle J., Fischer C., Bidet-Caulet A., Lecaignard F., Bertrand O., and Giard M.H. Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J. Neurosci. 28 (2008) 14301-14310
7. Besle J., Caclin A., Mayet R., Bauchet F., Delpuech C., Giard M.H., and Morlet D. Audiovisual events in sensory memory. J. Psychophysiol. 21 (2007) 231-238
8. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11 (2001) 1110-1123
9. Calvert G.A., Hansen P.C., Iversen S.D., and Brammer M.J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14 (2001) 427-438
10. Colin C., Radeau M., Soquet A., and Deltenre P. Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants. Clin. Neurophysiol. 115 (2004) 1989-2000
11. Colin C., Radeau M., Soquet A., Dachy B., and Deltenre P. Electrophysiology of spatial scene analysis: the mismatch negativity (MMN) is sensitive to the ventriloquism illusion. Clin. Neurophysiol. 113 (2002) 507-518
12. Colin C., Radeau M., Soquet A., Demolin D., Colin F., and Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clin. Neurophysiol. 113 (2002) 495-506
13. Czigler I. Visual mismatch negativity: violation of nonattended environmental regularities. J. Psychophysiol. 21 (2007) 224-230
14. Davis C., Kislyuk D., Kim J., and Sams M. The effect of viewing speech on auditory speech processing is different in the left and right hemispheres. Brain Res. 1242 (2008) 151-161
15. Dittmann-Balcar A., Thienel R., and Schall U. Attention-dependent allocation of auditory processing resources as measured by mismatch negativity. Neuroreport 10 18 (1999) 3749-3753
16. Fort A., Delpuech C., Pernier J., and Giard M.H. Early auditory-visual interactions in human cortex during nonredundant target identification. Cogn. Brain Res. 14 (2002) 20-30
17. Fort A., Delpuech C., Pernier J., and Giard M.H. Dynamics of cortico-subcorical crossmodal operations involved in audio-visual object detection in humans. Cereb. Cortex 12 (2002) 1031-1039
18. Giard M.H., and Peronnet F. Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 11 (1999) 473-490
19. Giard, M.H., Besle, J., in press. Methodological considerations: electrophysiology of multisensory interactions in humans. In: Kaiser, J., Naumer, M.J. (Eds.), Multisensory Object Perception in the Primate Brain. Springer.
20. Giard M.H., Fort A., Mouchetant-Rostaing Y., and Pernier J. Neurophysiological mechanisms of auditory selective attention in humans. Front. Biosci. 5 (2000) 84-94
21. Gomes H., Bernstein R., Ritter W., Vaughan H.G., and Miller J. Storage of feature conjunctions in transient auditory memory. Psychophysiology 34 (1997) 712-716
22. Hertrich I., Mathiak K., Lutzenberger W., Menning H., and Ackermann H. Sequential audiovisual interactions during speech perception: a whole-head MEG study. Neuropsychologia 45 (2007) 1342-1354
23. Hillyard S.A., Teder-Sälejärvi W.A., and Munte T.F. Temporal dynamics of early perceptual processing. Curr. Opin. Neurobiol. 8 (1998) 202-210
24. Jaaskelainen I.P., Ojanen V., Ahveninen J., Auranen T., Levänen S., Möttönen R., Tarnanen I., and Sams M. Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans. Neuroreport 15 (2004) 2741-2744
25. Kislyuk D.S., Mottonen R., and Sams M. Visual processing affects the neural basis of auditory discrimination. J. Cogn. Neurosci. 20 (2008) 2175-2184
26. Klucharev V., Möttönen R., and Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Res. Cogn. Brain Res. 18 (2003) 65-75
27. Korzyukov O.A., Winkler I., Gumenyuk V.I., and Alho K. Processing abstract auditory features in the human auditory cortex. Neuroimage 20 (2003) 2245-2258
28. Lebib R., Papo D., de Bode S., and Baudonniere P.M. Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation. Neurosci. Lett. 341 (2003) 185-188
29. McGurk H., and McDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
30. Miki K., Watanabe S., and Kakigi R. Interaction between auditory and visual stimulus relating to the vowel sounds in the auditory cortex in humans: a magnetoencephalographic study. Neurosci. Lett. 357 (2004) 199-202
31. Miller J.O. Divided attention: evidence for coactivation with redundant signals. Cognit. Psychol. 14 (1982) 247-279
32. Miller J.O. Time course of coactivation in bimodal divided attention. Percept. Psychophys. 40 (1986) 331-343
33. Miller J.O. Channel interaction and the redundant targets effect in bimodal divided attention. J. Exp. Psychol. Hum. Percept. Perform. 17 (1991) 160-169
34. Molholm S., Ritter W., Murray M.M., Javitt D.C., Schroeder C.E., and Foxe J.J. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn. Brain Res. 14 (2002) 115-128
35. Möttönen R., Schurmann M., and Sams M. Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neurosci. Lett. 363 (2004) 112-115
36. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Cogn. Brain Res. 13 (2002) 417-425
37. Muller-Gass A., Stelmack R.M., and Campbell K.B. The effect of visual task difficulty and attentional direction on the detection of acoustic change as indexed by the Mismatch Negativity. Brain Res. 1078 (2006) 112-130
38. Musacchia G., Sams M., Nicol T., and Kraus N. Seeing speech affects acoustic information processing in the human brainstem. Exp. Brain Res. 168 (2006) 1-10
39. Näätänen R., Paavilainen P., Rinne T., and Alho K. The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin. Neurophysiol. 118 (2007) 2544-2590
40. Nyman g., Alho K., laurinen P., Paavilainen P., Radil T., Rainikainen K., Sams M., and Näätänen R. Mismatch negativity (MMN) for sequences of auditory and visual stimuli: evidence for a mechanism specific to the auditory modality. Electroencephalogr. Clin. Neurophysiol. 77 (1990) 436-444
41. Oray S., Lu Z.L., and Dawson M.E. Modification of sudden onset auditory ERP by involuntary attention to visual stimuli. Int. J. Psychophysiol. 43 (2002) 213-224
42. Otten L.J., Alain C., and Picton T.W. Effects of visual attentional load on auditory processing. Neuroreport 11 (2000) 875-880
43. Paavilainen P., Arajarvi P., and Takegata R. Preattentive detection of nonsalient contingencies between auditory features. Neuroreport 18 (2007) 159-163
44. Pekkola J., Ojanen V., Autti T., Jaaskelainen I.P., Möttönen R., Tarkiainen A., and Sams M. Primary auditory cortex activation by visual speech: an fMRI study at 3 T. Neuroreport 16 (2005) 125-128
45. Reale R.A., Calvert G.A., Thesen T., Jenison R.L., Kawasaki H., Oya H., Howard M.A., and Brugge J.F. Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 145 (2007) 162-184
46. Romei V., Murray M.M., Merabet L.B., and Thut G. Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J. Neurosci. 27 (2007) 11465-11472
47. Saarinen J., Paavilainen P., Schoger E., Tervaniemi M., and Näätänen R. Representation of abstract attributes of auditory stimuli in the human brain. Neuroreport 3 (1992) 1149-1151
48. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., and Foxe J.J. Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45 (2007) 587-597
49. Sams M., and Levänen S. A neuromagnetic study of the integration of audiovisual speech in the brain. In: Koga Y., Nagata K., and Hirita K. (Eds). Brain Topography Today (1998), Elsevier Science, Amsterdam 47-53
50. Sams M., Aulanko R., Hamalainen H., Hari R., Lounasmaa O.V., Lu S.T., and Simola J. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci. Lett. 127 (1991) 141-145
51. Schneider T.R., Debener S., Oostenveld R., and Engel A.K. Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming. Neuroimage 42 (2008) 1244-1254
52. Schröger E., and Widmann A. Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology 35 (1998) 755-759
53. Senkowski D., Molholm S., Gomez-Ramirez M., and Foxe J.J. Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: a high-density electrical mapping study. Cereb. Cortex 16 (2006) 1556-1565
54. Senkowski D., Saint-Amour D., Kelly S.P., and Foxe J.J. Multisensory processing of naturalistic objects in motion: a high-density electrical mapping and source estimation study. Neuroimage 36 (2007) 877-888
55. Senkowski D., Talsma D., Grigutsch M., Herrmann C.S., and Woldorff M.G. Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45 (2007) 561-571
56. Senkowski D., Gomez-Ramirez M., Lakatos P., Wylie G.R., Molholm S., Schroeder C.E., and Foxe J.J. Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simians. Exp. Brain. Res. 177 (2007) 184-195
57. Stein B.E., and Meredith M.A. The Merging of the Senses. first ed. (1993), The MIT Press, Cambridge, MA
58. Stekelenburg J.J., and Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. J. Cogn. Neurosci. 19 (2007) 1964-1973
59. Stekelenburg J.J., Vroomen J., and de Gelder B. Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neurosci. Lett. 357 (2004) 163-166
60. Takegata R., Brattico E., Tervaniemi M., Varyagina O., Näätänen R., and Winkler I. Preattentive representation of feature conjunctions for concurrent spatially distributed auditory objects. Brain Res. Cogn. Brain Res. 25 (2005) 169-179
61. Talsma D., and Woldorff M.G. Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J. Cogn. Neurosci. 17 (2005) 1098-1114
62. Talsma D., Doty T.J., and Woldorff M.G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?. Cereb. Cortex 17 (2007) 679-690
63. Teder-Sälejärvi W.A., McDonald J.J., Di Russo F., and Hillyard S.A. An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res. Cogn. Brain Res. 14 (2002) 106-114
64. Teder-Sälejärvi W.A., Di Russo F., McDonald J.J., and Hillyard S.A. Effects of spatial congruity on audio-visual multimodal integration. J. Cogn. Neurosci. 17 (2005) 1396-1409
65. Thomas G.J. Experimental study of the influence of vision on sound localization. J. Exp. Psychol. 28 (1941) 163-177
66. Ullsperger P., Erdmann U., Freude G., and Dehoff W. When sound and picture do not fit: mismatch negativity and sensory interaction. Int. J. Psychophysiol. 59 (2006) 3-7
67. Valtonen J., May P., Makinen V., and Tiitinen H. Visual short-term memory load affects sensory processing of irrelevant sounds in human auditory cortex. Brain Res. Cogn. Brain Res. 17 (2003) 358-367
68. van Wassenhove V., Grant K.W., and Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. USA 102 (2005) 1181-1186
69. Vaughan H.G., and Ritter W. The sources of auditory evoked responses recorded from the human scalp. Electroencephalogr. Clin. Neurophysiol. 28 (1970) 360-367
70. Widmann A., Kujala T., Tervaniemi M., Kujala A., and Schröger E. From symbols to sounds: visual symbolic information activates sound representations. Psychophysiology 41 (2004) 709-715
71. Yumoto M., Uno A., Itoh K., Karino S., Saitoh O., Kaneko Y., Yatomi Y., and Kaga K. Audiovisual phonological mismatch produces early negativity in auditory cortex. Neuroreport 16 (2005) 803-806