Auditory-visual multisensory interactions in humans: Timing, topography, directionality, and sources

Journal of Neuroscience

Volumn 30, Issue 38, 2010, Pages 12572-12580

  Cappe, Céline ^a   Thut, Gregor ^d   Romei, Vincenzo ^d,e,f   Murray, Micah M ^a,b,c,g  

^a UNIVERSITY OF LAUSANNE   (Switzerland)

^b LAUSANNE UNIVERSITY HOSPITAL   (Switzerland)

^c University of Lausanne   (Switzerland)

^d UNIVERSITY OF GLASGOW   (United Kingdom)

^e UNIVERSITY COLLEGE LONDON   (United Kingdom)

^f UNIVERSITY COLLEGE LONDON   (United Kingdom)

^g VANDERBILT UNIVERSITY MEDICAL CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY RESPONSE; BRAIN REGION; ELECTRIC FIELD; EVENT RELATED POTENTIAL; FEMALE; HUMAN; HUMAN EXPERIMENT; MALE; MOTOR ACTIVITY; NORMAL HUMAN; PRIORITY JOURNAL; STIMULUS RESPONSE; TEMPORAL LOBE; TRANSCRANIAL MAGNETIC STIMULATION; VISUAL SYSTEM FUNCTION; WAVEFORM; ADOLESCENT; AUDITORY STIMULATION; BRAIN; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; EVOKED RESPONSE; HEARING; NERVE CELL NETWORK; PHOTOSTIMULATION; PHYSIOLOGY; SIGNAL PROCESSING; VISION;

ACOUSTIC STIMULATION; ADOLESCENT; ADULT; AUDITORY PERCEPTION; BRAIN; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; EVOKED POTENTIALS; FEMALE; HUMANS; MALE; NERVE NET; PHOTIC STIMULATION; SIGNAL PROCESSING, COMPUTER-ASSISTED; VISUAL PERCEPTION;

EID: 77957191402     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.1099-10.2010     Document Type: Article

References (87)

1. Allman BL, Keniston LP, Meredith MA (2009) Not just for bimodal neurons anymore: the contribution of unimodal neurons to cortical multisensory processing. Brain Topogr 21:157-167.
2. Angelaki DE, Gu Y, DeAngelis GC (2009) Multisensory integration: psychophysics, neurophysiology, and computation. Curr Opin Neurobiol 19:452-458.
3. Ary JP, Darcey TM, Fender DH (1981) A method for locating scalp electrodes in spherical coordinates. IEEE Trans Biomed Eng 28:834-836.
4. Beauchamp MS (2005) Statistical criteria in fMRI studies of multisensory integration. Neuroinformatics 3:93-113.
5. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190-1192.
6. Besle J, Fort A, Giard MH (2004) Interest and validity of the additive model in electrophysiological studies of multisensory interactions. Cogn Process 5:189-192.
7. Besle J, Fischer C, Bidet-Caulet A, Lecaignard F, Bertrand O, Giard MH (2008) Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J Neurosci 28:14301-14310.
8. Bizley JK, King AJ (2008) Visual-auditory spatial processing in auditory cortical neurons. Brain Res 1242:24-36.
9. Bizley JK, Nodal FR, Bajo VM, Nelken I, King AJ (2007) Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb Cortex 17:2172-2189.
10. Brefczynski-Lewis J, Lowitszch S, Parsons M, Lemieux S, Puce A (2009) Audiovisual non-verbal dynamic faces elicit converging fMRI and ERP responses. Brain Topogr 21:193-206.
11. Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110-1123.
12. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during cross-modal binding. Neuroreport 10:2619-2623.
13. Cappe C, Barone P (2005) Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. Eur J Neurosci 22:2886-2902.
14. Cappe C, Morel A, Rouiller EM (2007a) Thalamocortical and the dual pattern of corticothalamic projections of the posterior parietal cortex in macaque monkeys. Neuroscience 146:1371-1387.
15. Cappe C, Loquet G, Barone P, Rouiller EM (2007b) Influences of auditory and visual stimuli on neurons in auditory cortical areas in monkeys performing an audio-visual detection task. Soc Neurosci Abstr 33:620.18.
16. Cappe C, Rouiller EM, Barone P (2009a) Multisensory anatomic pathway. Hear Res 258:28-36.
17. Cappe C, Thut G, Romei V, Murray MM (2009b) Selective integration of auditory-visual looming cues by humans. Neuropsychologia 47:1045-1052.
18. Cappe C, Morel A, Barone P, Rouiller EM (2009c) The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor integrations. Cereb Cortex 19:2025-2037.
19. Cappe C, Murray MM, Barone P, Rouiller EM (2010) Multisensory facilitation of behavior in monkeys: effects of stimulus intensity. J Cogn Neurosci 22:2850-2863.
20. Carriere BN, Royal DW, Perrault TJ, Morrison SP, Vaughan JW, Stein BE, Wallace MT (2007) Visual deprivation alters the development of cortical multisensory integration. J Neurophysiol 98:2858-2867.
21. Dahl CD, Logothetis NK, Kayser C (2009) Spatial organization of multisensory responses in temporal association cortex. J Neurosci 29:11924-11932.
22. De Lucia M, Camen C, Clarke S, Murray MM (2009) The role of actions in auditory object discrimination. Neuroimage 48:475-485.
23. Falchier A, Clavagnier S, Barone P, Kennedy H (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22:5749-5759.
24. Falchier A, Schroeder CE, Hackett TA, Lakatos P, Nascimento-Silva S, Ulbert I, Karmos G, Smiley JF (2010) Projection from visual areas V2 and prostriata to caudal auditory cortex in the monkey. Cereb Cortex 20:1529-1538.
25. Fort A, Delpuech C, Pernier J, Giard MH (2002a) Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Cereb Cortex 12:1031-1039.
26. Fort A, Delpuech C, Pernier J, Giard MH (2002b) Early auditory-visual interactions in human cortex during nonredundant target identification. Brain Res Cogn Brain Res 14:20-30.
27. Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77-83.
28. Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol 88:540-543.
29. Foxe JJ, Strugstad EC, Sehatpour P, Molholm S, Pasieka W, Schroeder CE, McCourt ME (2008) Parvocellular and magnocellular contributions to the initial generators of the visual evoked potential: high-density electrical mapping of the "C1" component. Brain Topogr 21:11-21.
30. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cogn Sci 10:278-285.
31. Ghazanfar AA, Chandrasekaran C, Logothetis NK (2008) Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. J Neurosci 28:4457-4469.
32. Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473-490.
33. Gondan M, Röder B (2006) A new method for detecting interactions between the senses in event-related potentials. Brain Res 1073-1074:389-397.
34. Gonzalez Andino SL, Murray MM, Foxe JJ, de Peralta Menendez RG (2005) How single-trial electrical neuroimaging contributes to multisensory research. Exp Brain Res 166:298-304.
35. Grave de Peralta Menendez R, Gonzalez Andino S, Lantz G, Michel CM, Landis T (2001) Noninvasive localization of electromagnetic epileptic activity: I. Method descriptions and simulations. Brain Topogr 14:131-137.
36. Grave de Peralta Menendez R, Murray MM, Michel CM, Martuzzi R, Gonzalez Andino SL (2004) Electrical neuroimaging based on biophysical constraints. Neuroimage 21:527-539.
37. Guthrie D, Buchwald JS (1991) Significance testing of difference potentials. Psychophysiology 28:240-244.
38. Hackett TA, De La Mothe LA, Ulbert I, Karmos G, Smiley J, Schroeder CE (2007) Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. J Comp Neurol 502:924-952. (Pubitemid 46806628)
39. Holmes NP (2009) The principle of inverse effectiveness in multisensory integration: some statistical considerations. Brain Topogr 21:168-176.
40. Kayser C, Logothetis NK (2009) Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Front Integr Neurosci 3:7.
41. Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. J Neurosci 27:1824-1835.
42. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18:1560-1574.
43. Kayser C, Petkov CI, Logothetis NK (2009) Multisensory interactions in primate auditory cortex: fMRI and electrophysiology. Hear Res 258:80-88.
44. Lakatos P, Chen CM, O'Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279-292.
45. Lakatos P, Karmos G, Mehta AD, Ulbert I, Schroeder CE (2008) Entrainment of neuronal oscillations as a mechanism of attentional selection. Science 320:110-113.
46. Lakatos P, O'Connell MN, Barczak A, Mills A, Javitt DC, Schroeder CE (2009) The leading sense: supramodal control of neurophysiological context by attention. Neuron 64:419-430.
47. Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE (2002) Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci 14:420-429.
48. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166:289-297.
49. Lehmann D (1987) Principles of spatial analysis. In: Handbook of electroencephalography and clinical neurophysiology, Vol 1, Methods of analysis of brain electrical and magnetic signals (Gevins AS, Remond A, eds), pp 309-405. Amsterdam: Elsevier.
50. Lehmann D, Skrandies W (1980) Reference-free identification of components of checkerboard-evoked multichannel potential fields. Electroencephalogr Clin Neurophysiol 48:609-621.
51. Maier JX, Chandrasekaran C, Ghazanfar AA (2008) Integration of bimodal looming signals through neuronal coherence in the temporal lobe. Curr Biol 18:963-968.
52. Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP, Clarke S, Meuli RA (2007) Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb Cortex 17:1672-1679.
53. Meredith MA, Stein BE (1986) Visual, auditory, and somatosensory convergence on cells in the superior colliculus results in multisensory integration. J Neurophysiol 56:640-662.
54. Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, Grave de Peralta R (2004) EEG source imaging. Clin Neurophysiol 115:2195-2222.
55. Miniussi C, Girelli M, Marzi CA (1998) Neural site of the redundant target effect electrophysiological evidence. J Cogn Neurosci 10:216-230.
56. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115-128.
57. Möttönen R, Schürmann M, Sams M (2004) Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neurosci Lett 363:112-115.
58. Murray MM, Foxe JJ, Higgins BA, Javitt DC, Schroeder CE (2001) Visuospatial neural response interactions in early cortical processing during a simple reaction time task: a high-density electrical mapping study. Neuropsychologia 39:828-844.
59. Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ (2005) Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15:963-974.
60. Murray MM, Brunet D, Michel CM (2008) Topographic ERP analyses: a step-by-step tutorial review. Brain Topography 20:249-264.
61. Musacchia G, Schroeder CE (2009) Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res 258:72-79.
62. Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97-113.
63. Penhune VB, Zatorre RJ, MacDonald JD, Evans AC (1996) Interhemispheric anatomical differences in human primary auditory cortex: probabilistic mapping and volume measurement from magnetic resonance scans. Cereb Cortex 6:661-672.
64. Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575-2586.
65. Perrin F, Pernier J, Bertrand O, Giard MH, Echallier JF (1987) Mapping of scalp potentials by surface spline interpolation. Electroencephalogr Clin Neurophysiol 66:75-81.
66. Rademacher J, Morosan P, Schormann T, Schleicher A, Werner C, Freund HJ, Zilles K (2001) Probabilistic mapping and volume measurement of human primary auditory cortex. Neuroimage 13:669-683.
67. Raij T, Ahveninen J, Lin FH, Witzel T, Jääskeläinen IP, Letham B, Israeli E, Sahyoun C, Vasios C, Stufflebeam S, Hämälä inen M, Belliveau JW (2010) Onset timing of cross-sensory activations and multisensory interactions in auditory and visual sensory cortices. Eur J Neurosci 31:1772-1782.
68. Rockland KS, Ojima H (2003) Multisensory convergence in calcarine visual areas in macaque monkey. Int J Psychophysiol 50:19-26.
69. Romei V, Murray MM, Merabet LB, Thut G (2007) Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J Neurosci 27:11465-11472.
70. Romei V, Murray MM, Cappe C, Thut G (2009) Pre-perceptual and stimulus-selective enhancement of human low-level visual cortex excitability by sounds. Curr Biol 19:1799-1805.
71. Senkowski D, Schneider TR, Foxe JJ, Engel AK (2008) Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci 31:401-409.
72. Sperdin HF, Cappe C, Foxe JJ, Murray MM (2009) Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front Integr Neurosci 3:2.
73. Sperdin HF, Cappe C, Murray MM (2010) The behavioral relevance of multisensory neural response interactions. Front Neurosci 4:9.
74. Spinelli L, Andino SG, Lantz G, Seeck M, Michel CM (2000) Electromagnetic inverse solutions in anatomically constrained spherical head models. Brain Topogr 13:115-125.
75. Stanford TR, Quessy S, Stein BE (2005) Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25:6499-6508.
76. Stein BE, Meredith MA (1993) The merging of the senses. Cambridge, MA: MIT Press.
77. Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9:255-266.
78. Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198:183-194.
79. Talairach J, Tournoux P (1988) Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system - an approach to cerebral imaging. New York: Thieme Medical.
80. Teder-Sälejärvi WA, McDonald JJ, Di Russo F, Hillyard SA (2002) An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res Cogn Brain Res 14:106-114.
81. Toepel U, Knebel JF, Hudry J, le Coutre J, Murray MM (2009) The brain tracks the energetic value in food images. Neuroimage 44:967-974.
82. Van Zaen J, Uldry L, Duchêne C, Prudat Y, Meuli RA, Murray MM, Vesin JM (2010) Adaptive tracking of EEG oscillations. J Neurosci Methods 186:97-106.
83. Vidal J, Giard MH, Roux S, Barthé lémy C, Bruneau N (2008) Cross-modal processing of auditory-visual stimuli in a no-task paradigm: a topographic event-related potential study. Clin Neurophysiol 119:763-771.
84. Wallace MT, Stein BE (2007) Early experience determines how the senses will interact. J Neurophysiol 97:921-926.
85. Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246-1266.
86. Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci U S A 101:2167-2172.
87. Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neuroscience 9:79.