Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution

Cerebral Cortex

Volumn 25, Issue 8, 2015, Pages 2049-2064

  Hertz, Uri ^a,b   Amedi, Amir ^a,b  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^b HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

audiovisual integration; crossmodal effects; fMRI; multisensory processing; sensory cortices

Indexed keywords

ADULT; ARTICLE; ASSOCIATION; ATTENUATION; AUDITORY CORTEX; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; MIDDLE TEMPORAL GYRUS; NORMAL HUMAN; PRIORITY JOURNAL; SENSORY CORTEX; SENSORY EVALUATION; SENSORY STIMULATION; VISION; VISUAL CORTEX; ALGORITHM; AUDITORY STIMULATION; BRAIN; BRAIN MAPPING; HEARING; NEUROPSYCHOLOGICAL TEST; NUCLEAR MAGNETIC RESONANCE IMAGING; PHOTOSTIMULATION; PHYSIOLOGY; PROCEDURES; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; ALGORITHMS; AUDITORY PERCEPTION; BRAIN; BRAIN MAPPING; FEMALE; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; NEUROPSYCHOLOGICAL TESTS; PHOTIC STIMULATION; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84938580426     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhu010     Document Type: Article

References (96)

1. Adam R, Noppeney U. 2010. Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex. Neuroimage. 52:1592-1602.
2. Adrian ED. 1949. The Sherrington Lectures. I. Sensory integration. Liverpool (UK): University of Liverpool Press.
3. Amedi A, Malach R, Pascual-Leone A. 2005. Negative BOLD differentiates visual imagery and perception. Neuron. 48:859-872.
4. Amedi A, Stern WM, Camprodon JA, Bertolino A, Merabet L, Rotman S, Hemond C, Meijer P, Pascual-Leone A, Bermpohl F. 2007. Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat Neurosci. 10:687-689.
5. Azulay H, Striem E, Amedi A. 2009. Negative BOLD in sensory cortices during verbal memory: a component in generating internal representations? Brain Topogr. 21:221-231.
6. Bavelier D, Dye MWG, Hauser PC. 2006. Do deaf individuals see better? Trends Cogn Sci. 10:512-518.
7. Bavelier D, Green CS, Pouget A, Schrater P. 2012. Brain plasticity through the life span: learning to learn and action video games. Annu Rev Neurosci. 35:391-416.
8. Bavelier D, Neville HJ. 2002. Cross-modal plasticity: where and how? Nat Rev Neurosci. 3:443-452.
9. Beauchamp MS, Lee KE, Argall BD, Martin A. 2004. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron. 41:809-823.
10. Beauchamp MS, Pasalar S, Ro T. 2010. Neural substrates of reliabilityweighted visual-tactile multisensory integration. Front Syst Neurosci. 4:25.
11. Beauchamp MS, Yasar NE, Frye RE, Ro T. 2008. Touch, sound and vision in human superior temporal sulcus. Neuroimage. 41:1011-1020.
12. Benevento LA, Fallon J, Davis BJ, Rezak M. 1977. Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Exp Neurol. 57:849-872.
13. Bernstein LE, Lu Z-L, Jiang J. 2008. Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing. Brain Res. 1242:172-184.
14. Blakemore S-J, Bristow D, Bird G, Frith C, Ward J. 2005. Somatosensory activations during the observation of touch and a case of vision-touch synaesthesia. Brain. 128:1571-1583.
15. Boynton GM, Engel SA, Glover GH, Heeger DJ. 1996. Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci. 16:4207-4221.
16. Bressler SL, Tang W, Sylvester CM, Shulman GL, Corbetta M. 2008. Topdown control of human visual cortex by frontal and parietal cortex in anticipatory visual spatial attention. J Neurosci. 28:10056-10061.
17. Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E. 2003. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport. 14:2213-2218.
18. Callan DE, Tajima K, Callan AM, Kubo R, Masaki S, Akahane-Yamada R. 2003. Learning-induced neural plasticity associated with improved identification performance after training of a difficult second-language phonetic contrast. Neuroimage. 19:113-124.
19. Calvert GA. 2001. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex. 11:1110-1123.
20. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS. 1999. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport. 10:2619-2623.
21. Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS. 1997. Activation of auditory cortex during silent lipreading. Science. 276:593-596.
22. Calvert GA, Campbell R. 2003. Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci. 15:57-70.
23. Calvert GA, Campbell R, Brammer MJ. 2000. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol. 10:649-657.
24. Calvert GA, Thesen T. 2004. Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol Paris. 98:191-205.
25. Cohen L, Rothschild G, Mizrahi A. 2011. Multisensory integration of natural odors and sounds in the auditory cortex. Neuron. 72:357-369.
26. Cohen LG, Celnik P, Pascual-Leone A, Corwell B, Falz L, Dambrosia J, Honda M, Sadato N, Gerloff C, Catalá MD et al. 1997. Functional relevance of cross-modal plasticity in blind humans. Nature. 389:180-183.
27. Collignon O, Vandewalle G, Voss P, Albouy G, Charbonneau G, Lassonde M, Lepore F. 2011. Functional specialization for auditoryspatial processing in the occipital cortex of congenitally blind humans. Proc Natl Acad Sci USA. 108:4435-4440.
28. Corbetta M, Shulman GL. 2002. Control of goal-directed and stimulusdriven attention in the brain. Nat Rev Neurosci. 3:201-215.
29. Dehaene S, Cohen L. 2007. Cultural recycling of cortical maps. Neuron. 56:384-398.
30. Dehaene S, Cohen L, Sigman M, Vinckier F. 2005. The neural code for written words: a proposal. Trends Cogn Sci. 9:335-341.
31. Diedrichsen J, Wiestler T, Krakauer JW. 2013. Two distinct ipsilateral cortical representations for individuated finger movements. Cereb Cortex. 23:1362-1377.
32. Engel SA, Glover GH, Wandell BA. 1997. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb Cortex. 7:181-192.
33. Ernst MO, Banks MS. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature. 415:429-433.
34. Fairhall SL, Macaluso E. 2009. Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. Eur J Neurosci. 29:1247-1257.
35. Felleman DJ, Van Essen DC, Van Essen DC. 1991. Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex. 1:1-47.
36. Fetsch CR, Pouget A, DeAngelis GC, Angelaki DE. 2012. Neural correlates of reliability-based cue weighting during multisensory integration. Nat Neurosci. 15:146-154.
37. Fetsch CR, Turner AH, DeAngelis GC, Angelaki DE. 2009. Dynamic reweighting of visual and vestibular cues during self-motion perception. J Neurosci. 29:15601-15612.
38. Forman SD, Cohen JD, Fitzgerald M, Eddy WF, Mintun MA, Noll DC. 1995. Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn Reson Med. 33:636-647.
39. Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE. 2000. Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res. 10:77-83.
40. Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM. 2002. Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol. 88:540-543.
41. Friston KJ,Holmes AP, Price CJ, Büchel C, WorsleyKJ. 1999.Multisubject fMRI studies and conjunction analyses. Neuroimage. 10:385-396.
42. Friston KJ, Holmes AP, Worsley KJK, Poline JJ-BJ-P, Frith CD, Frackowiak RSJ. 1994. Statistical parametric maps in functional imaging: a general linear approach. Hum Brain Mapp. 2:189-210.
43. Friston KJ, Jezzard P, Turner R. 1994. Analysis of functional MRI timeseries. Hum Brain Mapp. 12:153-171.
44. Fu K-MG, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE. 2003. Auditory cortical neurons respond to somatosensory stimulation. J Neurosci. 23:7510-7515.
45. Gazzaley A, D'Esposito M. 2007. Top-down modulation and normal aging. Ann N Y Acad Sci. 1097:67-83.
46. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK. 2005. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci. 25:5004-5012.
47. Ghazanfar AA, Schroeder CE. 2006. Is neocortex essentially multisensory? Trends Cogn Sci. 10:278-285.
48. Hertz U, Amedi A. 2010. Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis. Neuroimage. 52:617-632.
49. Hickok G, Poeppel D. 2000. Towards a functional neuroanatomy of speech perception. Trends Cogn Sci. 4:131-138.
50. Jones EG, Powell TPS. 1970. An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain. 93:793-820.
51. Kastrup A, Baudewig J, Schnaudigel S, Huonker R, Becker L, Sohns JM, Dechent P, Klingner C, Witte OW. 2008. Behavioral correlates of negative BOLD signal changes in the primary somatosensory cortex. Neuroimage. 41:1364-1371.
52. Kayser C. 2010. The multisensory nature of unisensory cortices: a puzzle continued. Neuron. 67:178-180.
53. Kayser C, Logothetis NK. 2007. Do early sensory cortices integrate cross-modal information? Brain Struct Funct. 212:121-132.
54. Kayser C, Petkov CI, Logothetis NK. 2008. Visual modulation of neurons in auditory cortex. Cereb Cortex. 18:1560-1574.
55. Kim J-K, Zatorre RJ. 2008. Generalized learning of visual-to-auditory substitution in sighted individuals. Brain Res. 1242:263-275.
56. Kim J-K, Zatorre RJ. 2011. Tactile-auditory shape learning engages the lateral occipital complex. J Neurosci. 31:7848-7856.
57. Kuchinsky SE, Vaden KI, Keren NI, Harris KC, Ahlstrom JB, Dubno JR, Eckert MA. 2012. Word intelligibility and age predict visual cortex activity during word listening. Cereb Cortex. 22:1360-1371.
58. Lakatos P, Chen C, O'Connell MN, Mills A, Schroeder CE, Connell MNO. 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 53:279-292.
59. Laurienti PJ, Burdette JH, Wallace MT, Yen Y, Field AS, Stein BE. 2002. Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci. 14:420-429.
60. Lee H, Noppeney U. 2011. Physical and perceptual factors shape the neural mechanisms that integrate audiovisual signals in speech comprehension. J Neurosci. 31:11338-11350.
61. Magosso E, Serino A, di Pellegrino G, Ursino M. 2010. Crossmodal links between vision and touch in spatial attention: a computational modelling study. Comput Intell Neurosci. 2010:304941.
62. Matteau I, Kupers R, Ricciardi E, Pietrini P, Ptito M. 2010. Beyond visual, aural and haptic movement perception: hMT+ is activated by electrotactile motion stimulation of the tongue in sighted and in congenitally blind individuals. Brain Res Bull. 82:264-270.
63. Meijer PB. 1992. An experimental system for auditory image representations. IEEE Trans Biomed Eng. 39:112-121.
64. Merabet LB, Rizzo JF, Amedi A, Somers DC, Pascual-Leone A. 2005. What blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses. Nat Rev Neurosci. 6:71-77.
65. Meredith MA, Stein BE. 1986. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol. 56:640-662.
66. Mozolic JL, Joyner D, Hugenschmidt CE, Peiffer AM, Kraft RA, Maldjian JA, Laurienti PJ. 2008. Cross-modal deactivations during modalityspecific selective attention. BMC Neurol. 8:35.
67. Murray MM, Foxe JJ, Wylie GR. 2005. The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage. 27:473-478.
68. Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Gonzalez Andino S, Schnider A. 2004. Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage. 21:125-135.
69. Naumer MJ, Doehrmann O, Müller NG, Muckli L, Kaiser J, Hein G. 2009. Cortical plasticity of audio-visual object representations. Cereb Cortex. 19:1641-1653.
70. Noppeney U, Ostwald D, Werner S. 2010. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci. 30:7434-7446.
71. Pascual-Leone A, Amedi A, Fregni F, Merabet LB. 2005. The plastic human brain cortex. Annu Rev Neurosci. 28:377-401.
72. Pascual-Leone A, Hamilton R. 2001. The metamodal organization of the brain. Prog Brain Res. 134:427-445.
73. Ptito M, Matteau I, Gjedde A, Kupers R. 2009. Recruitment of the middle temporal area by tactile motion in congenital blindness. Neuroreport. 20:543-547.
74. Reich L, Szwed M, Cohen L, Curie M, Amedi A. 2011. A ventral visual stream reading center independent of visual experience. Curr Biol. 21:1-6.
75. Renier LA, Anurova I, De Volder AG, Carlson S, VanMeter J, Rauschecker JP. 2010. Preserved functional specialization for spatial processing in the middle occipital gyrus of the early blind. Neuron. 68:138-148.
76. Schridde U, Khubchandani M, Motelow JE, Sanganahalli BG, Hyder F, Blumenfeld H. 2008. Negative BOLD with large increases in neuronal activity. Cereb Cortex. 18:1814-1827.
77. Schroeder CE, Foxe J. 2005. Multisensory contributions to low-level, "unisensory" processing. Curr Opin Neurobiol. 15:454-458.
78. Sekuler R, Sekuler AB, Lau R. 1997. Sound alters visual motion perception. Nature. 385:308.
79. Sereno MI, Dale AM, Reppas JB, Kwong KK, Belliveau JW, Brady TJ, Rosen BR, Tootell RB. 1995. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science. 268:889-893.
80. Shams L. 2012. Early integration and Bayesian causal inference in multisensory perception. In: Murray MM, Wallace MT, editors. The neural bases ofmultisensory processes. Boca Raton (FL): CRC Press.
81. Shams L, Kamitani Y, Shimojo S. 2000. Illusions. What you see is what you hear. Nature. 408:788.
82. Sheppard JP, Raposo D, Churchland AK. 2013. Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans. J Vis. 13:1-19.
83. Shmuel A, Augath M, Oeltermann A, Logothetis NK. 2006. Negative functional MRI response correlates with decreases in neuronal activity in monkey visual area V1. Nat Neurosci. 9:569-577.
84. Stevenson RA, Kim S, James TW. 2009. An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res. 198:183-194.
85. Striem-Amit E, Cohen L, Dehaene S, Amedi A. 2012a. Reading with sounds: sensory substitution selectively activates the visual word form area in the blind. Neuron. 76:640-652.
86. Striem-Amit E, Dakwar O, Reich L, Amedi A. 2012b. The large-scale organization of "visual" streams emerges without visual experience. Cereb Cortex. 22:1698-1709.
87. Striem-Amit E, Hertz U, Amedi A. 2011. Extensive cochleotopic mapping of human auditory cortical fields obtained with phase-encoding fMRI. PLoS ONE. 6:e17832.
88. Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG. 2010. The multifaceted interplay between attention and multisensory integration. Trends Cogn Sci. 14:400-410.
89. Thelen A, Cappe C, Murray MM. 2012. Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage. 62:1478-1488.
90. Tkach JA, Chen X, Freebairn LA, Schmithorst VJ, Holland SK, Lewis BA. 2011. Neural correlates of phonological processing in speech sound disorder: a functional magnetic resonance imaging study. Brain Lang. 119:42-49.
91. Van Atteveldt NM, Formisano E, Goebel R, Blomert L, Van Atteveldt N. 2004. Integration of letters and speech sounds in the human brain. Neuron. 43:271-282.
92. Van Wassenhove V, Grant KW, Poeppel D, Van Wassenhove V. 2005. Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA. 102:1181-1186.
93. Watkins S, Shams L, Tanaka S, Haynes J-DJ, Rees G. 2006. Sound alters activity in human V1 in association with illusory visual perception. Neuroimage. 31:1247-1256.
94. Werner S, Noppeney U. 2010. Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci. 30: 2662-2675.
95. Zeharia N, Hertz U, Flash T, Amedi A. 2012. Negative blood oxygenation level dependent homunculus and somatotopic information in primary motor cortex and supplementary motor area. Proc Natl Acad Sci USA. 109:18565-18570.
96. Zeki SM. 1978. Functional specialisation in the visual cortex of the rhesus monkey. Nature. 274:423-428.