Multisensory influences on auditory processing: Perspectives from fMRI and electrophysiology

The Neural Bases of Multisensory Processes

Volumn , Issue , 2011, Pages 99-113

  Kayser, Christoph ^a   Petkov, Christopher I ^b   Remedios, Ryan ^a   Logothetis, Nikos K ^a  

^a MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^b NEWCASTLE UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85059472834     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (116)

1. Adrian, E.D. 1928. The Basis of Sensations. New York, Norton.
2. Allman, B.L., and M.A. Meredith. 2007. Multisensory processing in "unimodal" neurons: Cross-modal sub-threshold auditory effects in cat extrastriate visual cortex. Journal of Neurophysiology 98:545-9.
3. Allman, B.L., R.E. Bittencourt-Navarrete, L.P. Keniston et al. 2008a. Do cross-modal projections always result in multisensory integration? Cerebral Cortex 18:2066-76.
4. Allman, B.L., L.P. Keniston, and M.A. Meredith. 2008b. Subthreshold auditory inputs to extrastriate visual neurons are responsive to parametric changes in stimulus quality: Sensory-specific versus non-specific coding. Brain Research 1242:95-101.
5. Beauchamp, M.S., B.D. Argall, J. Bodurka, J.H. Duyn, and A. Martin. 2004. Unraveling multisensory integra-tion: Patchy organization within human STS multisensory cortex. Nature Neuroscience 7:1190-2.
6. Beauchamp, M.S., N.E. Yasar, R.E. Frye, and T. Ro. 2008. Touch, sound and vision in human superior temporal sulcus. NeuroImage 41:1011-20.
7. Belin, P., and R.J. Zatorre. 2003. Adaptation to speaker's voice in right anterior temporal lobe. Neuroreport 14:2105-9.
8. Belin, P., R.J. Zatorre, P. Lafaille, P. Ahad, and B. Pike. 2000. Voice-selective areas in human auditory cortex. Nature 403:309-12.
9. Benevento, L.A., J. Fallon, B.J. Davis, and M. Rezak. 1977. Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology 57:849-72.
10. Bernstein, L.E., E.T. Auer Jr., J.K. Moore et al. 2002. Visual speech perception without primary auditory cortex activation. Neuroreport 13:311-5.
11. Bizley, J.K., and A.J. King. 2008. Visual-auditory spatial processing in auditory cortical neurons. Brain Research 1242:24-36.
12. Bizley, J.K., and K.M. Walker. 2009. Distributed sensitivity to conspecific vocalizations and implications for the auditory dual stream hypothesis. Journal of Neuroscience 29:3011-3.
13. Bizley, J.K., F.R. Nodal, V.M. Bajo, I. Nelken, and A.J. King. 2006. Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cerebral Cortex 17:2172-89.
14. Bruce, C., R. Desimone, and C.G. Gross. 1981. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology 46:369-84.
15. Calvert, G.A. 2001. Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11:1110-23.
16. Calvert, G.A., and R. Campbell. 2003. Reading speech from still and moving faces: The neural substrates of visible speech. Journal of Cognitive Neuroscience 15:57-70.
17. Calvert, G.A., M.J. Brammer, E.T. Bullmore et al. 1999. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619-23.
18. Calvert, G., C. Spence, and B.E. Stein. 2004. The Handbook of Multisensory Processes. Cambridge: MIT Press.
19. Calvert, G.A., E.T. Bullmore, M.J. Brammer et al. 1997. Activation of auditory cortex during silent lipreading. Science 276:593-6.
20. Campanella, S., and P. Belin. 2007. Integrating face and voice in person perception. Trends in Cognitive Sciences 11:535-43.
21. Cancelliere, A.E., and A. Kertesz. 1990. Lesion localization in acquired deficits of emotional expression and comprehension. Brain and Cognition 13:133-47.
22. Cappe, C., and P. Barone. 2005. Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. European Journal of Neuroscience 22:2886-902.
23. Cappe, C., G. Loquet, P. Barone, and E.M. Rouiller. 2007. Neuronal responses to visual stimuli in auditory cortical areas of monkeys performing an audio-visual detection task. European Brain and Behaviour Society. Trieste.
24. Chiry, O., E. Tardif, P.J. Magistretti, and S. Clarke. 2003. Patterns of calcium-binding proteins support parallel and hierarchical organization of human auditory areas. European Journal of Neuroscience 17:397-410.
25. Clarke, S., and F. Rivier. 1998. Compartments within human primary auditory cortex: Evidence from cyto-chrome oxidase and acetylcholinesterase staining. European Journal of Neuroscience 10:741-5.
26. Cohen, Y.E., F. Theunissen, B.E. Russ, and P. Gill. 2007. Acoustic features of rhesus vocalizations and their representation in the ventrolateral prefrontal cortex. Journal of Neurophysiology 97:1470-84.
27. Dahl, C., N. Logothetis, and C. Kayser. 2009. Spatial organization of multisensory responses in temporal asso-ciation cortex. Journal of Neuroscience 29:11924-32.
28. Dehner, L.R., L.P. Keniston, H.R. Clemo, and M.A. Meredith. 2004. Cross-modal circuitry between auditory and somatosensory areas of the cat anterior ectosylvian sulcal cortex: A 'new' inhibitory form of multi-sensory convergence. Cerebral Cortex 14:387-403.
29. Engel, S.A., D.E. Rumelhart, B.A. Wandell et al. 1994. fMRI of human visual cortex. Nature 369:525.
30. Engelien, A., D. Silbersweig, E. Stern et al. 1995. The functional anatomy of recovery from auditory agnosia. A PET study of sound categorization in a neurological patient and normal controls. Brain 118(Pt 6):1395-409.
31. Ernst, M.O., and H.H. Bülthoff. 2004. Merging the senses into a robust percept. Trends in Cognitive Science 8:162-9.
32. Falchier, A., S. Clavagnier, P. Barone, and H. Kennedy. 2002. Anatomical evidence of multimodal integration in primate striate cortex. Journal of Neuroscience 22:5749-59.
33. Formisano, E., D.S. Kim, F. Di Salle et al. 2003. Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 40:859-69.
34. Foxe, J.J., and C.E. Schroeder. 2005. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16:419-23.
35. Foxe, J.J., G.R. Wylie, A. Martinez et al. 2002. Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study. Journal of Neurophysiology 88:540-3.
36. Fullerton, B.C., and D.N. Pandya. 2007. Architectonic analysis of the auditory-related areas of the superior temporal region in human brain. Journal of Comparative Neurology 504:470-98.
37. Ghazanfar, A.A., C. Chandrasekaran, and N.K. Logothetis. 2008. Interactions between the superior tempo-ral sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. Journal of Neuroscience 28:4457-69.
38. Ghazanfar, A.A., and N.K. Logothetis. 2003. Neuroperception: Facial expressions linked to monkey calls. Nature 423:937-8.
39. Ghazanfar, A.A., and C.E. Schroeder. 2006. Is neocortex essentially multisensory? Trends in Cognitive Sciences 10:278-85.
40. Ghazanfar, A.A., J.X. Maier, K.L. Hoffman, and N.K. Logothetis. 2005. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience 25:5004-12.
41. Griffiths, T.D., A. Rees, C. Witton et al. 1997. Spatial and temporal auditory processing deficits following right hemisphere infarction. A psychophysical study. Brain 120(Pt 5):785-94.
42. Habib, M., G. Daquin, L. Milandre et al. 1995. Mutism and auditory agnosia due to bilateral insular damage-role of the insula in human communication. Neuropsychologia 33:327-39.
43. Hackett, T.A., I. Stepniewska, and J.H. Kaas. 1998. Subdivisions of auditory cortex and ipsilateral cortical connec-tions of the parabelt auditory cortex in macaque monkeys. Journal of Comparative Neurology 394:475-95.
44. Hall, D.A. 2003. Auditory pathways: Are 'what' and 'where' appropriate? Current Biology 13:R406-8.
45. Hershenson, M. 1962. Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology 63:289-93.
46. Jones, E.G., and T.P. Powell. 1970. An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain 93:793-820.
47. Juergens, E., A. Guettler, and R. Eckhorn. 1999. Visual stimulation elicits locked and induced gamma oscilla-tions in monkey intracortical- and EEG-potentials, but not in human EEG. Experimental Brain Research 129:247-59.
48. Kaas, J.H., and T.A. Hackett. 2000. Subdivisions of auditory cortex and processing streams in primates. Proceedings of the National Academy of Sciences of the United States of America 97:11793-9.
49. Kayser, C., and N.K. Logothetis. 2007. Do early sensory cortices integrate cross-modal information? Brain Structure and Function 212:121-32.
50. Kayser, C., C.I. Petkov, M. Augath, and N.K. Logothetis. 2005. Integration of touch and sound in auditory cortex. Neuron 48:373-84.
51. Kayser, C., C.I. Petkov, M. Augath, and N.K. Logothetis. 2007. Functional imaging reveals visual modulation of specific fields in auditory cortex. Journal of Neuroscience 27:1824-35.
52. Kayser, C., C.I. Petkov, and N.K. Logothetis. 2008. Visual modulation of neurons in auditory cortex. Cerebral Cortex 18:1560-74.
53. Kayser, C., N. Logothetis, and S. Panzeri. 2009a. Visual enhancement of the information representation in audi-tory cortex. Current Biology (in press).
54. Kayser, C., M.A. Montemurro, N. Logothetis, and S. Panzeri. 2009b. Spike-phase coding boosts and stabilizes the information carried by spatial and temporal spike patterns. Neuron 61:597-608.
55. Kayser, C., C.I. Petkov, and N.K. Logothetis. 2009c. Multisensory interactions in primate auditory cortex: fMRI and electrophysiology. Hearing Research (in press). doi:10.1016/j.heares.2009.02.011.
56. Kosaki, H., T. Hashikawa, J. He, and E.G. Jones. 1997. Tonotopic organization of auditory cortical fields delineated by parvalbumin immunoreactivity in macaque monkeys. Journal of Comparative Neurology 386:304-16.
57. Kotz, S.A., M. Meyer, K. Alter et al. 2003. On the lateralization of emotional prosody: An event-related func-tional MR investigation. Brain and Language 86:366-76.
58. Lakatos, P., C.M. Chen, M.N. O'Connell, A. Mills, and C.E. Schroeder. 2007. Neuronal oscillations and multi-sensory interaction in primary auditory cortex. Neuron 53:279-92.
59. Laurienti, P.J., T.J. Perrault, T.R. Stanford, M.T. Wallace, and B.E. Stein. 2005. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Experimental Brain Research 166:289-97.
60. Lauritzen, M. 2005. Reading vascular changes in brain imaging: Is dendritic calcium the key? Nature Neuroscience Reviews 6(1):77-85.
61. Lehmann, C., M. Herdener, F. Esposito et al. 2006. Differential patterns of multisensory interactions in core and belt areas of human auditory cortex. NeuroImage 31:294-300.
62. Leopold, D.A. 2009. Neuroscience: Pre-emptive blood flow. Nature 457:387-8.
63. Logothetis, N.K. 2002. The neural basis of the blood-oxygen-level-dependent functional magnetic resonance imaging signal. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 357:1003-37.
64. Logothetis, N.K. 2008. What we can do and what we cannot do with fMRI. Nature 453:869-78.
65. Logothetis, N.K., H. Guggenberger, S. Peled, and J. Pauls. 1999. Functional imaging of the monkey brain. Nature Neuroscience 2:555-62.
66. Martuzzi, R., M.M. Murray, C.M. Michel et al. 2006. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cerebral Cortex 17:1672-9.
67. McGurk, H., and J. MacDonald. 1976. Hearing lips and seeing voices. Nature 264:746-8.
68. Meredith, M.A., and B.L. Allman. 2009. Subthreshold multisensory processing in cat auditory cortex. Neuroreport 20:126-31.
69. Merzenich, M.M., and J.F. Brugge. 1973. Representation of the cochlear partition of the superior temporal plane of the macaque monkey. Brain Research 50:275-96.
70. Meyer, M., K. Alter, A.D. Friederici, G. Lohmann, and D.Y. Von Cramon. 2002. FMRI reveals brain regions mediating slow prosodic modulations in spoken sentences. Human Brain Mapping 17:73-88.
71. Mitzdorf, U. 1985. Current source-density method and application in cat cerebral cortex: Investigation of evoked potentials and EEG phenomena. Physiological Reviews 65:37-100.
72. Montessori, M. 1967. The Absorbent Mind. New York: Henry Holt & Co.
73. Morel, A., P.E. Garraghty, and J.H. Kaas. 1993. Tonotopic organization, architectonic fields, and connections of auditory cortex in macaque monkeys. Journal of Comparative Neurology 335:437-59.
74. Oakland, T., J.L. Black, G. Stanford, N.L. Nussbaum, and R.R. Balise. 1998. An evaluation of the dyslexia training program: A multisensory method for promoting reading in students with reading disabilities. Journal of Learning Disabilities 31:140-7.
75. Pekkola, J., V. Ojanen, T. Autti et al. 2005. Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Human Brain Mapping 27:471-7.
76. Perrodin, C., C. Kayser, N. Logothetis, and C. Petkov. 2009a. Visual influences on voice-selective neurons in the anterior superior-temporal plane. International Conference on Auditory Cortex. Madgeburg, Germany, 2009.
77. Perrodin, C., L. Veit, C. Kayser, N.K. Logothetis, and C.I. Petkov. 2009b. Encoding properties of neurons sensi-tive to species-specific vocalizations in the anterior temporal lobe of primates. International Conference on Auditory Cortex. Madgeburg, Germany, 2009.
78. Petkov, C.I., C. Kayser, M. Augath, and N.K. Logothetis. 2006. Functional imaging reveals numerous fields in the monkey auditory cortex. PLoS Biology 4:e215.
79. Petkov, C.I., C. Kayser, T. Steudel et al. 2008. A voice region in the monkey brain. Nature Neuroscience 11:367-74.
80. Petkov, C.I., C. Kayser, M. Augath, and N.K. Logothetis. 2009. Optimizing the imaging of the monkey auditory cortex: Sparse vs. continuous fMRI. Magnetic Resonance Imaging 27:1065-73.
81. Petrini, K., M. Russell, and F. Pollick. 2009. When knowing can replace seeing in audiovisual integration of actions. Cognition 110:432-9.
82. Rauschecker, J.P. 1998. Cortical processing of complex sounds. Current Opinion in Neurobiology 8:516-21.
83. Rauschecker, J.P., and B. Tian. 2000. Mechanisms and streams for processing of what and where in auditory cortex. Proceedings of the National Academy of Sciences of the United States of America 97:11800-6.
84. Rauschecker, J.P., and B. Tian. 2004. Processing of band-passed noise in the lateral auditory belt cortex of the rhesus monkey. Journal of Neurophysiology 91:2578-89.
85. Rauschecker, J.P., B. Tian, and M. Hauser. 1995. Processing of complex sounds in the macaque nonprimary auditory cortex. Science 268:111-4.
86. Rauschecker, J.P., B. Tian, T. Pons, and M. Mishkin. 1997. Serial and parallel processing in rhesus monkey auditory cortex. Journal of Comparative Neurology 382:89-103.
87. Recanzone, G.H., D.C. Guard, and M.L. Phan. 2000. Frequency and intensity response properties of single neu-rons in the auditory cortex of the behaving macaque monkey. Journal of Neurophysiology 83:2315-31.
88. Remedios, R., N.K. Logothetis, and C. Kayser. 2009. An auditory region in the primate insular cortex respond-ing preferentially to vocal communication sounds. Journal of Neuroscience 29:1034-45.
89. Rockland, K.S., and H. Ojima. 2003. Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiology 50:19-26.
90. Romanski, L.M., B. Tian, J. Fritz et al. 1999. Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nature Neuroscience 2:1131-6.
91. Romanski, L.M., B.B. Averbeck, and M. Diltz. 2005. Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. Journal of Neurophysiology 93:734-47.
92. Ross, L.A., D. Saint-Amour, V.M. Leavitt, D.C. Javitt, and J.J. Foxe. 2007. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex 17: 1147-53.
93. Rumsey, J.M., B. Horwitz, B.C. Donohue et al. 1997. Phonological and orthographic components of word recognition. A PET-rCBF study. Brain 120(Pt 5):739-59.
94. Schroeder, C.E., and J.J. Foxe. 2002. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Research. Cognitive Brain Research 14:187-98.
95. Schroeder, C.E., and J. Foxe. 2005. Multisensory contributions to low-level, 'unisensory' processing. Current Opinion in Neurobiology 15:454-8.
96. Schroeder, C.E., R.W. Lindsley, C. Specht et al. 2001. Somatosensory input to auditory association cortex in the macaque monkey. Journal of Neurophysiology 85:1322-7.
97. Schroeder, C.E., J. Smiley, K.G. Fu et al. 2003. Anatomical mechanisms and functional implications of multi-sensory convergence in early cortical processing. International Journal of Psychophysiology 50:5-17.
98. Schurmann, M., G. Caetano, Y. Hlushchuk, V. Jousmaki, and R. Hari. 2006. Touch activates human auditory cortex. NeuroImage 30:1325-31.
99. Shams, L., and A.R. Seitz. 2008. Benefits of multisensory learning. Trends in Cognitive Sciences 12:411-7.
100. Stein, B.E. 1998. Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Experimental Brain Research 123:124-35.
101. Stein, B.E., and T.R. Stanford. 2008. Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience 9:255-66.
102. Stein, B.E., and M.A. Meredith. 1993. Merging of the Senses. Cambridge: MIT Press.
103. Sugihara, T., M.D. Diltz, B.B. Averbeck, and L.M. Romanski. 2006. Integration of auditory and visual communi-cation information in the primate ventrolateral prefrontal cortex. Journal of Neuroscience 26:11138-47.
104. Sumby, W.H., and I. Polack. 1954. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26:212-5.
105. Talavage, T.M., M.I. Sereno, J.R. Melcher et al. 2004. Tonotopic organization in human auditory cortex revealed by progressions of frequency sensitivity. Journal of Neurophysiology 91:1282-96.
106. Tian, B., D. Reser, A. Durham, A. Kustov, and J.P. Rauschecker. 2001. Functional specialization in rhesus monkey auditory cortex. Science 292:290-3.
107. van Atteveldt, N., E. Formisano, R. Goebel, and L. Blomert. 2004. Integration of letters and speech sounds in the human brain. Neuron 43:271-82.
108. van Wassenhove, V., K.W. Grant, and D. Poeppel. 2005. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America 102:1181-6.
109. von Kriegstein, K., and A.L. Giraud. 2006. Implicit multisensory associations influence voice recognition. PLoS Biol 4:e326.
110. von Kriegstein, K., E. Eger, A. Kleinschmidt, and A.L. Giraud. 2003. Modulation of neural responses to speech by directing attention to voices or verbal content. Brain Research. Cognitive Brain Research 17:48-55.
111. von Kriegstein, K., A. Kleinschmidt, and A.L. Giraud. 2006. Voice recognition and cross-modal responses to familiar speakers' voices in prosopagnosia. Cerebral Cortex 16:1314-22.
112. Wallace, M.T., R. Ramachandran, and B.E. Stein. 2004. A revised view of sensory cortical parcellation. Proceedings of the National Academy of Sciences of the United States of America 101:2167-72.
113. Wang, X. 2000. On cortical coding of vocal communication sounds in primates. Proceedings of the National Academy of Sciences of the United States of America 97:11843-9.
114. Warnking, J., M. Dojat, A. Guerin-Dugue et al. 2002. fMRI retinotopic mapping-step by step. NeuroImage 17:1665-83.
115. Wessinger, C.M., J. Vanmeter, B. Tian et al. 2001. Hierarchical organization of the human auditory cortex revealed by functional magnetic resonance imaging. Journal of Cognitive Neuroscience 13:1-7.
116. Zatorre, R.J., A.C. Evans, and E. Meyer. 1994. Neural mechanisms underlying melodic perception and memory for pitch. Journal of Neuroscience 14:1908-19.