Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis

NeuroImage

Volumn 52, Issue 2, 2010, Pages 617-632

  Hertz, Uri ^a,b   Amedi, Amir ^a,b  

^a HEBREW UNIVERSITY   (Israel)

^b HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY CORTEX; AUDITORY DISCRIMINATION; AUDITORY STIMULATION; CONTROLLED STUDY; CORTICAL SYNCHRONIZATION; FEMALE; FOURIER ANALYSIS; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; INFORMATION PROCESSING; MALE; NORMAL HUMAN; PRIORITY JOURNAL; SENSORY EVALUATION; STIMULUS RESPONSE; VISUAL CORTEX; VISUAL DISCRIMINATION; VISUAL STIMULATION; VOLUNTEER; AUDITORY NERVOUS SYSTEM; BRAIN; COCHLEA; HEARING; NEUROPSYCHOLOGICAL TEST; NUCLEAR MAGNETIC RESONANCE IMAGING; PHOTOSTIMULATION; PHYSIOLOGY; PROCEDURES; RETINA; SIGNAL PROCESSING; VISION; VISUAL SYSTEM; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; AUDITORY PATHWAYS; AUDITORY PERCEPTION; BRAIN; COCHLEA; FEMALE; FOURIER ANALYSIS; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; NEUROPSYCHOLOGICAL TESTS; PHOTIC STIMULATION; RETINA; SIGNAL PROCESSING, COMPUTER-ASSISTED; VISUAL PATHWAYS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 77953689092     PISSN: 10538119     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2010.04.186     Document Type: Article

References (109)

1. Aguirre G.K., Zarahn E., D'Esposito M. The variability of human, BOLD hemodynamic responses. Neuroimage 1998, 8:360-369.
2. Alain C., Arnott S.R., Hevenor S., Graham S., Grady C.L. "What" and "where" in the human auditory system. Proc. Natl. Acad. Sci. 2001, 98:12301.
3. Allison T., Puce A., McCarthy G. Social perception from visual cues: role of the STS region. Trends Cogn. Sci. 2000, 4:267-278.
4. Allman B.L., Meredith M.A. Multisensory processing in" unimodal" neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. J. Neurophysiol. 2007, 98:545.
5. Amedi A., Kriegstein K., Atteveldt N.M., Beauchamp M.S., Naumer M.J. Functional imaging of human crossmodal identification and object recognition. Exp. Brain Res. 2005, 166:559-571.
6. Amedi A., Malach R., Pascual-Leone A. Negative BOLD differentiates visual imagery and perception. Neuron 2005, 48:859-872.
7. Amedi A., Raz N., Pianka P., Malach R., Zohary E. Early 'visual' cortex activation correlates with superior verbal memory performance in the blind. Nat. Neurosci. 2003, 6:758-766.
8. Avidan G., Hasson U., Hendler T., Zohary E., Malach R. Analysis of the neuronal selectivity underlying low fMRI signals. Curr. Biol. 2002, 12:964-972.
9. Azulay H., Striem E., Amedi A. Negative BOLD in sensory cortices during verbal memory: a component in generating internal representations?. Brain Topogr. 2009, 21:221-231.
10. Barraclough N.E., Xiao D., Baker C.I., Oram M.W., Perrett D.I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 2005, 17:377-391.
11. Bavelier D., Dye M.W.G., Hauser P.C. Do deaf individuals see better?. Trends Cogn. Sci. 2006, 10:512-518.
12. Bavelier D., Neville H.J. Cross-modal plasticity: where and how?. Nat. Rev. Neurosci. 2002, 3:443-452.
13. Beauchamp M.S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 2005, 3:93-113.
14. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat. Neurosci. 2004, 7:1190-1192.
15. Beauchamp M.S., Lee K.E., Argall B.D., Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 2004, 41:809-823.
16. Beauchamp M.S., Yasar N.E., Frye R.E., Ro T. Touch, sound and vision in human superior temporal sulcus. Neuroimage 2008, 41:1011-1020.
17. Benjamini Y., Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B (Methodological) 1995, 289-300.
18. Bischoff M., Walter B., Blecker C.R., Morgen K., Vaitl D., Sammer G. Utilizing the ventriloquism-effect to investigate audio-visual binding. Neuropsychologia 2007, 45:578-586.
19. Brosch M., Selezneva E., Scheich H. Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J. Neurosci. 2005, 25:6797.
20. Buonomano D.V., Karmarkar U.R. How do we tell time?. The Neuroscientist 2002, 8:42.
21. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 2001, 11:1110-1123.
22. Calvert G.A., Campbell R., Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 2000, 10:649-657.
23. Calvert G.A., Thesen T. Multisensory integration: methodological approaches and emerging principles in the human brain. J. Physiol. Paris 2004, 98:191-205.
24. Colonius H., Diederich A. Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J. Cogn. Neurosci. 2004, 16:1000-1009.
25. Dhamala M., Assisi C.G., Jirsa V.K., Steinberg F.L., Kelso J.A. Multisensory integration for timing engages different brain networks. Neuroimage 2007, 34:764-773.
26. Dixon N.F., Spitz L. The detection of auditory visual desynchrony. Perception 1980, 9:719-721.
27. Driver J., Noesselt T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 2008, 57:11-23.
28. Engel A.K., Singer W. Temporal binding and the neural correlates of sensory awareness. Trends Cogn. Sci. 2001, 5:16-25.
29. Engel S.A., Glover G.H., Wandell B.A. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex 1997, 7:181-192.
30. Engel S.A., Rumelhart D.E., Wandell B.A., Lee A.T., Glover G.H., Chichilnisky E.J., Shadlen M.N. fMRI of human visual cortex. Nature 1994, 369:525.
31. Felleman D.J., Van Essen D.C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1991, 1:1.
32. Formisano E., Kim D.S., Di Salle F., van de Moortele P.F., Ugurbil K., Goebel R. Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 2003, 40:859-869.
33. Genovese C.R., Lazar N.A., Nichols T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 2002, 15:870-878.
34. Ghazanfar A.A., Maier J.X., Hoffman K.L., Logothetis N.K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 2005, 25:5004.
35. Ghazanfar A.A., Schroeder C.E. Is neocortex essentially multisensory?. Trends Cogn. Sci. 2006, 10:278-285.
36. Grill-Spector K., Malach R. The human visual cortex. Annu. Rev. Neurosci. 2004, 27:649-677.
37. Grossman E.D., Battelli L., Pascual-Leone A. Repetitive TMS over posterior STS disrupts perception of biological motion. Vis. Res. 2005, 45:2847-2853.
38. Hadjikhani N., Roland P.E. Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J. Neurosci. 1998, 18:1072-1084.
39. Hasson U., Harel M., Levy I., Malach R. Large-scale mirror-symmetry organization of human occipito-temporal object areas. Neuron 2003, 37:1027-1041.
40. Hein G., Doehrmann O., Muller N.G., Kaiser J., Muckli L., Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J. Neurosci. 2007, 27:7881-7887.
41. Hein G., Knight R.T. Superior temporal sulcus-it's my area: or is it?. J. Cogn. Neurosci. 2008, 20:2125-2136.
42. Hocking J., Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cereb. Cortex 2008, 18:2439-2449.
43. Holmes N.P. The principle of inverse effectiveness in multisensory integration: some statistical considerations. Brain Topogr. 2009, 21:168-176.
44. Jammalamadaka S.R., Sengupta A. Topics in Circular Statistics 2001, World Scientific Pub Co Inc.
45. Kauffman T., Thoret H., Pascual-Leone A. Braille character discrimination in blindfolded human subjects. Neuroreport 2002, 13:571.
46. Kayser C., Logothetis N.K. Do early sensory cortices integrate cross-modal information?. Brain Struct. Funct. 2007, 212:121-132.
47. Kayser C., Logothetis N.K., Panzeri S. Visual enhancement of the information representation in auditory cortex. Curr. Biol. 2010, 20:19-24.
48. Kourtzi Z., Tolias A.S., Altmann C.F., Augath M., Logothetis N.K. Integration of local features into global shapes monkey and human fMRI studies. Neuron 2003, 37:333-346.
49. Lacey S., Tal N., Amedi A., Sathian K. A putative model of multisensory object representation. Brain Topogr. 2009, 21:269-274.
50. Lakatos P., Chen C.M., O'Connell M.N., Mills A., Schroeder C.E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 2007, 53:279-292.
51. Laurienti P.J., Perrault T.J., Stanford T.R., Wallace M.T., Stein B.E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 2005, 166:289-297.
52. Levy I., Hasson U., Avidan G., Hendler T., Malach R. Center-periphery organization of human object areas. Nat. Neurosci. 2001, 4:533-539.
53. Macaluso E., Driver J. Spatial attention and crossmodal interactions between vision and touch. Neuropsychologia 2001, 39:1304-1316.
54. Macaluso E., Driver J. Multisensory spatial interactions: a window onto functional integration in the human brain. TRENDS Neurosci. 2005, 28:264-271.
55. Macaluso E., George N., Dolan R., Spence C., Driver J. Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 2004, 21:725-732.
56. Martuzzi R., Murray M.M., Michel C.M., Thiran J.P., Maeder P.P., Clarke S., Meuli R.A. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb. Cortex 2007, 17:1672-1679.
57. McGrath M., Summerfield Q. Intermodal timing relations and audio visual speech recognition by normal hearing adults. J. Acoust. Soc. Am. 1985, 77:678.
58. Meienbrock A., Naumer M.J., Doehrmann O., Singer W., Muckli L. Retinotopic effects during spatial audio-visual integration. Neuropsychologia 2007, 45:531-539.
59. Merabet L.B., Swisher J.D., McMains S.A., Halko M.A., Amedi A., Pascual-Leone A., Somers D.C. Combined activation and deactivation of visual cortex during tactile sensory processing. J. Neurophysiol. 2007, 97:1633.
60. Meredith M.A. On the neuronal basis for multisensory convergence: a brief overview. Brain Res. Cogn. Brain Res. 2002, 14:31-40.
61. Meredith M.A., Stein B.E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 1986, 56:640.
62. Miller L.M., D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 2005, 25:5884.
63. Mishra J., Martinez A., Sejnowski T.J., Hillyard S.A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J. Neurosci. 2007, 27:4120.
64. Molholm S., Ritter W., Murray M.M., Javitt D.C., Schroeder C.E., Foxe J.J. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res. Cogn. Brain Res. 2002, 14:115-128.
65. Murray M.M., Molholm S., Michel C.M., Heslenfeld D.J., Ritter W., Javitt D.C., Schroeder C.E., Foxe J.J. Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb. Cortex 2005, 15:963.
66. Naumer M.J., Doehrmann O., Muller N.G., Muckli L., Kaiser J., Hein G. Cortical plasticity of audio-visual object representations. Cereb. Cortex 2009, 19:1641-1653.
67. Noesselt T., Rieger J.W., Schoenfeld M.A., Kanowski M., Hinrichs H., Heinze H.J., Driver J. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J. Neurosci. 2007, 27:11431-11441.
68. Noppeney U., Josephs O., Hocking J., Price C.J., Friston K.J. The effect of prior visual information on recognition of speech and sounds. Cerebral cortex 2007, 18:598-609.
69. Olivetti Belardinelli M., Sestieri C., Di Matteo R., Delogu F., Del Gratta C., Ferretti A., Caulo M., Tartaro A., Romani G.L. Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn. Process. 2004, 5:167-174.
70. Pascual-Leone A., Amedi A., Fregni F., Merabet L.B. The Plastic Human Brain Cortex 2005.
71. Petkov C.I., Kayser C., Augath M., Logothetis N.K. Functional Imaging Reveals Numerous Fields in the Monkey Auditory Cortex 2006.
72. Platel H., Price C., Baron J.C., Wise R., Lambert J., Frackowiak R.S., Lechevalier B., Eustache F. The structural components of music perception. A functional anatomical study. Brain 1997, 120:229.
73. Powers A.R., Hillock A.R., Wallace M.T. Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 2009, 29:12265.
74. Rauschecker J.P. Parallel processing in the auditory cortex of primates. Audiol. Neurotol. 2000, 3:86-103.
75. Rauschecker J.P., Tian B., Hauser M. Processing of Complex Sounds in the Macaque Nonprimary Auditory Cortex. Science-New York then Washington-, 111-111 1995.
76. Rockland K.S., Ojima H. Multisensory convergence in calcarine visual areas in macaque monkey. Int. J. Psychophysiol. 2003, 50:19-26.
77. Rockland K.S., Van Hoesen G.W. Direct temporal-occipital feedback connections to striate cortex (V1) in the macaque monkey. Cereb. Cortex 1994, 4:300.
78. Röder B., Rösler F., Neville H.J. Auditory memory in congenitally blind adults: a behavioral-electrophysiological investigation. Cogn. Brain Res. 2001, 11:289-303.
79. Sadato N. How the blind" see" Braille: lessons from functional magnetic resonance imaging. The Neuroscientist 2005, 11:577.
80. Sathian K. Visual cortical activity during tactile perception in the sighted and the visually deprived. Dev. Psychobiol. 2005, 46:279-286.
81. Schroeder C.E., Foxe J. Multisensory contributions to low-level, 'unisensory' processing. Curr. Opin. Neurobiol. 2005, 15:454-458.
82. Schroeder C.E., Smiley J., Fu K.G., McGinnis T., O'Connell M.N., Hackett T.A. Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. Int. J. Psychophysiol. 2003, 50:5-17.
83. Self M.W., Zeki S. The integration of colour and motion by the human visual brain. Cereb. Cortex 2005, 15:1270.
84. Sereno M.I., Dale A.M., Reppas J.B., Kwong K.K., Belliveau J.W., Brady T.J., Rosen B.R., Tootell R.B. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 1995, 268:889-893.
85. Sereno M.I., Huang R.S. A human parietal face area contains aligned head-centered visual and tactile maps. Nat. Neurosci. 2006, 9:1337-1343.
86. Sereno M.I., Tootell R.B. From monkeys to humans: what do we now know about brain homologies?. Curr. Opin. Neurobiol. 2005, 15:135-144.
87. Shams L., Kamitani Y., Shimojo S. Illusions: what you see is what you hear. Nature 2000, 408:788.
88. Singer W., Gray C.M. Visual feature integration and the temporal correlation hypothesis. Annu. Rev. Neurosci. 1995, 18:555-586.
89. Spence C., Driver J. Crossmodal Space and Crossmodal Attention 2004, Oxford University Press, USA.
90. Stevenson R.A., Geoghegan M.L., James T.W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp. Brain Res. 2007, 179:85-95.
91. Stevenson R.A., James T.W. Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 2009, 44:1210-1223.
92. Stevenson R.A., Kim S., James T.W. An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Experimental Brain Research 2009, 198:183-194.
93. Tal N., Amedi A. Multisensory visual-tactile object related network in humans: insights gained using a novel crossmodal adaptation approach. Exp. Brain Res. 2009, 198:165-182.
94. Talairach J., Tournoux P. Co-Planar Stereotaxic Atlas of the Human Brain 1988, Thieme New York.
95. Talavage T.M., Sereno M.I., Melcher J.R., Ledden P.J., Rosen B.R., Dale A.M. Tonotopic organization in human auditory cortex revealed by progressions of frequency sensitivity. J. Neurophysiol. 2004, 91:1282-1296.
96. Taylor K.I., Moss H.E., Stamatakis E.A., Tyler L.K. Binding crossmodal object features in perirhinal cortex. Proceedings of the National Academy of Sciences of the United States of America 2006, 103:8239-8244.
97. Ungerleider L.G., Mishkin M. Two cortical visual systems. Analysis of Visual Behavior 1982, 549-586.
98. Upadhyay J., Ducros M., Knaus T.A., Lindgren K.A., Silver A., Tager-Flusberg H., Kim D.S. Function and connectivity in human primary auditory cortex: a combined fMRI and DTI study at 3Tesla. Cereb. Cortex 2007, 17:2420-2432.
99. van Atteveldt N., Formisano E., Goebel R., Blomert L. Integration of letters and speech sounds in the human brain. Neuron 2004, 43:271-282.
100. van Atteveldt N.M., Blau V.C., Blomert L., Goebel R. fMR-adaptation indicates selectivity to audiovisual content congruency in distributed clusters in human superior temporal cortex. BMC neuroscience 2010, 11:11.
101. van Atteveldt N.M., Formisano E., Blomert L., Goebel R. The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb. Cortex 2007, 17:962-974.
102. van Atteveldt N.M., Formisano E., Goebel R., Blomert L. Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 2007, 36:1345-1360.
103. Watkins S., Shams L., Josephs O., Rees G. Activity in human V1 follows multisensory perception. Neuroimage 2007, 37:572-578.
104. Watkins S., Shams L., Tanaka S., Haynes J.D., Rees G. Sound alters activity in human V1 in association with illusory visual perception. Neuroimage 2006, 31:1247-1256.
105. Wessinger C.M., VanMeter J., Tian B., Van Lare J., Pekar J., Rauschecker J.P. Hierarchical organization of the human auditory cortex revealed by functional magnetic resonance imaging. J. Cogn. Neurosci. 2001, 13:1-7.
106. Wilson S.M., Molnar-Szakacs I., Iacoboni M. Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension. Cereb. Cortex 2008, 18:230-242.
107. Winston J.S., Henson R.N.A., Fine-Goulden M.R., Dolan R.J. fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. J. Neurophysiol. 2004, 92:1830.
108. Zahar Y., Reches A., Gutfreund Y. Multisensory enhancement in the optic tectum of the barn owl: spike count and spike timing. J. Neurophysiol. 2009, 101:2380.
109. Zeki S.M. Functional specialisation in the visual cortex of the rhesus monkey. Nature 1978, 274:423-428.