Onset timing of cross-sensory activations and multisensory interactions in auditory and visual sensory cortices

European Journal of Neuroscience

Volumn 31, Issue 10, 2010, Pages 1772-1782

  Raij, Tommi ^a   Ahveninen, Jyrki ^a   Lin, Fa Hsuan ^a,b   Witzel, Thomas ^c   Jääskeläinen, Iiro P ^d   Letham, Benjamin ^a   Israeli, Emily ^c   Sahyoun, Cherif ^c   Vasios, Christos ^a   Stufflebeam, Steven ^a   Hämäläinen, Matti ^a,c   Belliveau, John W ^a  

^a MASSACHUSETTS GENERAL HOSPITAL   (United States)

^b NATIONAL TAIWAN UNIVERSITY   (Taiwan)

^c HARVARD MIT DIVISION OF HEALTH SCIENCES AND TECHNOLOGY   (United States)

^d AALTO UNIVERSITY   (Finland)

Author keywords

Audiovisual interactions; Auditory cortex; Cross modal; Human; MEG; Visual cortex

Indexed keywords

ARTICLE; AUDITORY CORTEX; AUDITORY STIMULATION; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEMISPHERE; HUMAN; LATENT PERIOD; MAGNETOENCEPHALOGRAPHY; NORMAL HUMAN; PRIORITY JOURNAL; REACTION TIME; SIGNAL NOISE RATIO; VISUAL CORTEX; VISUAL STIMULATION; ADULT; EVOKED RESPONSE; FEMALE; HEMISPHERIC DOMINANCE; MALE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHOTOSTIMULATION; PHYSIOLOGY; SOMATOSENSORY CORTEX; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; AUDITORY CORTEX; EVOKED POTENTIALS; FEMALE; FUNCTIONAL LATERALITY; HUMANS; MAGNETIC RESONANCE IMAGING; MAGNETOENCEPHALOGRAPHY; MALE; PHOTIC STIMULATION; REACTION TIME; SOMATOSENSORY CORTEX; VISUAL CORTEX; YOUNG ADULT;

EID: 77952659582     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2010.07213.x     Document Type: Article

References (81)

1. Bandettini, P., Jesmanowicz, A., Van Kylen, J., Birn, R. Hyde, J. (1998) Functional MRI of brain activation induced by scanner acoustic noise. Magn. Reson. Med., 39, 410 416.
2. Bar, M., Kassam, K., Ghuman, A., Boshyan, J., Schmid, A., Dale, A., Hämäläinen, M., Marinkovic, K., Schacter, D., Rosen, B. Halgren, E. (2006) Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA, 103, 449 454.
3. Benevento, L., Fallon, J., Davis, B. Rezak, M. (1977) Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and orbital cortex of the macaque monkey. Exp. Neurol., 57, 849 872.
4. Besle, J., Fischer, C., Bidet-Caulet, A., Lecaignard, F., Bertrand, O. Giard, M. (2008) Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J. Neurosci., 28, 14301 14310.
5. Bruce, C., Desimone, R. Gross, C. (1986) Both striate cortex and superior colliculus contribute to visual properties of neurons in superior temporal polysensory area of macaque monkey. J. Neurophysiol., 55, 1057 1075.
6. Budinger, E., Heil, P., Hess, A. Scheich, H. (2006) Multisensory processing via early cortical stages: connections of the primary auditory cortical field with other sensory systems. Neuroscience, 143, 1065 1083.
7. Burock, M. Dale, A. (2000) Estimation and detection of event-related fMRI signals with temporally correlated noise: a statistically efficient and unbiased approach. Hum. Brain Mapp., 11, 249 260.
8. Cappe, C. Barone, P. (2005) Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. Eur. J. Neurosci., 22, 2886 2902.
9. Cappe, C., Morel, A., Barone, P. Rouiller, E. (2009) The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor interplay. Cereb. Cortex, 19, 2025 2037.
10. Celesia, G. (1976) Organization of auditory cortical areas in man. Brain, 99, 403 414.
11. Clavagnier, S., Falchier, A. Kennedy, H. (2004) Long-distance feedback projections to area V1: implications for multisensory integration, spatial awareness, and visual consciousness. Cogn. Affect. Behav. Neurosci., 4, 117 126.
12. Cohen, D., Schlapfer, U., Ahlfors, S., Hamalainen, M. Halgren, E. (2002) New six-layer magnetically-shielded room for MEG. In Nowak, H., Haueisen, J., Giessler, F. Huonker, R. (Eds 13th International Conference on Biomagne. VDE Verlag
13. Collins, C., Lyon, D. Kaas, J. (2005) Distribution across cortical areas of neurons projecting to the superior colliculus in new world monkeys. Anat. Rec. A Discov. Mol. Cell. Evol. Biol., 285, 619 627.
14. Cox, R. Jesmanowicz, A. (1999) Real-time 3D image registration for functional MRI. Magn. Reson. Med., 42, 1014 1018.
15. Cusick, C. (1997) The superior temporal polysensory region in monkeys. In Rockland, K., Kaas, J. Peters, A. (Eds Cerebral Cortex Volume 12: Extrastriate Cortex in Primates. Plenum Press
16. Dale, A. (1999) Optimal experimental design for event-related fMRI. Hum. Brain Mapp., 8, 109 114.
17. Dale, A. Sereno, M. (1993) Improved localization of cortical activity by combining EEG and MEG with MRI cortical surface reconstruction: A linear approach. J. Cogn. Neurosci., 5, 162 176.
18. Dale, A., Liu, A., Fischl, B., Buckner, R., Belliveau, J., Lewine, J. Halgren, E. (2000) Dynamic statistical parametric mapping: Combining fMRI and MEG for high-resolution imaging of cortical activity. Neuron, 26, 55 67.
19. Desikan, R., Segonne, F., Fischl, B., Quinn, B., Dickerson, B., Blacker, D., Buckner, R., Dale, A., Maguire, R., Hyman, B., Albert, M. Killiany, R. (2006) An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage, 31, 968 980.
20. Falchier, A., Clavagnier, S., Barone, P. Kennedy, H. (2002) Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci., 22, 5749 5759.
21. Fischl, B., Sereno, M., Tootell, R. Dale, A. (1999) High-resolution inter-subject averaging and a coordinate system for the cortical surface. Hum. Brain Mapp., 8, 272 284.
22. Fischl, B., Salat, D., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., Montillo, A., Makris, N., Rosen, B. Dale, A. (2002) Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33, 341 355.
23. Fischl, B., van der Kouwe, A., Destrieux, C., Halgren, E., Segonne, F., Salat, D., Busa, E., Seidman, L., Goldstein, J., Kennedy, D., Caviness, V., Makris, N., Rosen, B. Dale, A. (2004) Automatically parcellating the human cerebral cortex. Cereb. Cortex, 14, 11 22.
24. Foxe, J. Schroeder, C. (2005) The case for feedforward multisensory convergence during early cortical processing. Neuroreport, 16, 419 423.
25. Foxe, J. Simpson, G. (2002) Flow of activation from V1 to frontal cortex in humans. A framework for defining "early" visual processing. Exp. Brain Res., 142, 139 150.
26. Foxe, J., Morocz, I., Murray, M., Higgins, B., Javitt, D. Schroeder, C. (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res. Cogn. Brain Res., 10, 77 83.
27. Ghazanfar, A. Schroeder, C. (2006) Is neocortex essentially multisensory? Trends Cogn. Sci., 10, 278 285.
28. Giard, M. Peronnet, F. (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci., 11, 473 490.
29. Goldenholz, D., Ahlfors, S., Hämäläinen, M., Sharon, D., Ishitobi, M., Vaina, L. Stufflebeam, S. (2009) Mapping the signal-to-noise- ratios of cortical sources in magnetoencephalography and electroencephalography. Hum. Brain Mapp., 30, 1077 1086.
30. Groh, J., Trause, A., Underhill, A., Clark, K. Inati, S. (2001) Eye position influences auditory responses in primate inferior colliculus. Neuron, 29, 509 518.
31. Gross, C. (1991) Contribution of striate cortex and the superior colliculus to visual function in area MT, the superior temporal polysensory area and the inferior temporal cortex. Neuropsychologia, 29, 497 515.
32. Hackett, T., De La Mothe, L., Ulbert, I., Karmos, G., Smiley, J. Schroeder, C. (2007) Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. J. Comp. Neurol., 502, 924 952. (Pubitemid 46806628)
33. Hämäläinen, M. Hari, R. (2002) Magnetoencephalographic characterization of dynamic brain activation. Basic principles and methods of data collection and source analysis. In Toga, A.W. Mazziotta, J.C. (Ed. Brain Mapping: the Methods. Academic Press
34. Hämäläinen, M. Ilmoniemi, R. (1984) Interpreting Measured Magnetic Fields of the Brain: Estimates of Current Distributions. Helsinki University of Technology
35. Hämäläinen, M. Ilmoniemi, R. (1994) Interpreting magnetic fields of the brain: minimum norm estimates. Med. Biol. Eng. Comput., 32, 35 42.
36. Hämäläinen, M. Sarvas, J. (1989) Realistic conductivity geometry model of the human head for interpretation of neuromagnetic data. IEEE Trans. Biomed. Eng., 36, 165 171.
37. Hillebrand, A. Barnes, G. (2002) A quantitative assessment of the sensitivity of whole-head MEG to activity in the adult human cortex. Neuroimage, 16, 638 650.
38. Jiang, W. Stein, B. (2003) Cortex controls multisensory depression in superior colliculus. J. Neurophysiol., 90, 2132 2135.
39. Karns, C. Knight, R. (2009) Intermodal auditory, visual, and tactile attention modulates early stages of neural processing. J. Cogn. Neurosci., 21, 669 683.
40. Laurienti, P., Burdette, J., Wallace, M., Yen, Y., Field, A. Stein, B. (2002) Deactivation of sensory-specific cortex by cross-modal stimuli. J. Cogn. Neurosci., 14, 420 429.
41. Lin, F., Hara, K., Solo, V., Vangel, M., Belliveau, J., Stufflebeam, S. Hämäläinen, M. (2009) Dynamic Granger-Geweke causality modeling with application to interictal spike propagation. Hum. Brain Mapp., 30, 1877 1886.
42. Liu, A., Belliveau, J. Dale, A. (1998) Spatiotemporal imaging of human brain activity using functional MRI constrained magnetoenceohalography data: Monte Carlo simulations. Proc. Natl. Acad. Sci. USA, 95, 8945 8950.
43. Macaluso, E. (2006) Multisensory processing in sensory-specific cortical areas. Neuroscientist, 12, 327 338.
44. Macaluso, E. Driver, J. (2005) Multisensory spatial interactions: a window onto functional integration in the human brain. Trends Neurosci., 28, 264 271.
45. Martuzzi, R., Murray, M., Maeder, P., Fornari, E., Thiran, J., Clarke, S., Michel, C. Meuli, R. (2006) Visuo-motor pathways in humans revealed by event-related fMRI. Exp. Brain Res., 170, 472 487.
46. Martuzzi, R., Murray, M., Michel, C., Thiran, J.-P., Maeder, P., Clarke, S. Meul, R. (2007) Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb. Cortex, 17, 1672 1679.
47. Mesulam, M.-M. (1998) From sensation to cognition. Brain, 121, 1013 1052.
48. Miller, L. D'Esposito, M. (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci., 25, 5884 5893.
49. Mishra, J., Martinez, A., Sejnowski, T. Hillyard, S. (2007) Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J. Neurosci., 27, 4120 4131.
50. Molholm, S. Foxe, J. (2005) Look 'hear', primary auditory cortex is active during lip-reading. Neuroreport, 16, 123 124.
51. Molholm, S., Ritter, W., Murray, M., Javitt, D., Schroeder, C. Foxe, J. (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res. Cogn. Brain Res., 14, 115 128.
52. Molholm, S., Ritter, W., Javitt, D. Foxe, J. (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb. Cortex, 14, 452 465.
53. de la Mothe, L., Blumell, S., Kajikawa, Y. Hackett, T. (2006a) Cortical connections of the auditory cortex in marmoset monkeys: Core and medial belt regions. J. Comp. Neurol., 496, 27 71.
54. de la Mothe, L., Blumell, S., Kajikawa, Y. Hackett, T. (2006b) Thalamic connections of the auditory cortex in marmoset monkeys: Core and medial belt regions. J. Comp. Neurol., 496, 72 96.
55. Murray, M., Molholm, S., Michel, C., Heslenfeld, D., Ritter, W., Javitt, D., Schroeder, C. Foxe, J. (2005) Grabbing your ear: rapid auditory- somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb. Cortex, 15, 963 974.
56. Musacchia, G. Schroeder, C. (2009) Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear. Res., 258, 72 79.
57. Musacchia, G., Sams, M., Nicol, T. Kraus, N. (2006) Seeing speech affects acoustic information processing in the human brainstem. Exp. Brain Res., 168, 1 10.
58. Parkkonen, L., Fujiki, N. Mäkelä, J. (2009) Sources of auditory brainstem responses revisited: contribution by magnetoencephalography. Hum. Brain Mapp., 30, 1772 1782.
59. Pekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I., Mottonen, R., Tarkiainen, A. Sams, M. (2005) Primary auditory cortex activation by visual speech: an fMRI study at 3 T. Neuroreport, 16, 125 128.
60. Poghosyan, V. Ioannides, A. (2008) Attention modulates earliest responses in the primary auditory and visual cortices. Neuron, 58, 802 813.
61. Raij, T., Uutela, K. Hari, R. (2000) Audiovisual integration of letters in the human brain. Neuron, 28, 617 625.
62. Raij, T., Karhu, J., Kičić, D., Lioumis, P., Julkunen, P., Lin, F., Ahveninen, J., Ilmoniemi, R., Mäkelä, J., Hämäläinen, M., Rosen, B. Belliveau, J. (2008) Parallel input makes the brain run faster. Neuroimage, 40, 1792 1797.
63. Rockland, K. Ojima, H. (2003) Multisensory convergence in calcarine visual areas in macaque monkey. Int. J. Psychophysiol., 50, 19 26.
64. Rockland, K. Van Hoesen, G. (1994) Direct temporal-occipital feedback connections to striate cortex (V1) in the macaque monkey. Cereb. Cortex, 4, 300 313.
65. Schoffelen, J. Gross, J. (2009) Source connectivity analysis with MEG and EEG. Hum. Brain Mapp., 30, 1857 1865.
66. Schroeder, C. Foxe, J. (2002) The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res. Cogn. Brain, 14, 187 198.
67. Schroeder, C. Foxe, J. (2005) Multisensory contributions to low-level, 'unisensory' processing. Curr. Opin. Neurobiol., 15, 454 458.
68. Schroeder, C.E., Lindsley, R.W., Specht, C., Marcovici, A., Smiley, J.F. Javitt, D.C. (2001) Somatosensory input to auditory association cortex in the macaque monkey. J. Neurophysiol., 85, 1322 1327.
69. Schroeder, C., Smiley, J., Fu, K., McGinnis, T., O'Connell, M. Hackett, T. (2003) Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. Int. J. Psychophysiol., 50, 5 17.
70. Shams, L., Kamitani, Y. Shimojo, S. (2002) Visual illusion induced by sound. Brain Res. Cogn. Brain Res., 14, 147 152.
71. Shams, L., Iwaki, S., Chawla, A. Bhattacharya, J. (2005) Early modulation of visual cortex by sound: an MEG study. Neurosci. Lett., 378, 76 81.
72. Smiley, J. Falchier, A. (2009) Multisensory connections of monkey auditory cerebral cortex. Hear. Res., 258, 37 46.
73. Smiley, J., Hackett, T., Ulbert, I., Karmas, G., Lakatos, P., Javitt, D. Schroeder, C. (2007) Multisensory convergence in auditory cortex, I. Cortical connections of the caudal superior temporal plane in macaque monkeys. J. Comp. Neurol., 502, 894 923. (Pubitemid 46806627)
74. Sperdin, H., Cappe, C., Foxe, J. Murray, M. (2009) Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front. Integr. Neurosci., 3, 2.
75. Stein, B. Meredith, M. (1993) The Merging of the Senses. MIT Press
76. Stein, B., Wallace, M., Stanford, T. Jiang, W. (2002) Cortex governs multisensory integration in the midbrain. Neuroscientist, 8, 306 314.
77. Talsma, D. Woldorff, M. (2005) Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J. Cogn. Neurosci., 17, 1098 1114.
78. Talsma, D., Doty, T. Woldorff, M. (2007) Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex, 17, 679 690.
79. Teder-Sälejärvi, W., McDonald, J., Di Russo, F. Hillyard, S. (2002) An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res. Cogn. Brain Res., 14, 106 114.
80. Wang, Y., Celebrini, S., Trotter, Y. Barone, P. (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci., 9, 79.
81. Watkins, S., Shams, L., Tanaka, S., Haynes, J. Rees, G. (2006) Sound alters activity in human V1 in association with illusory visual perception. Neuroimage, 31, 1247 1256.