Multisensory integration of sounds and vibrotactile stimuli in processing streams for "what" and "where"

Journal of Neuroscience

Volumn 29, Issue 35, 2009, Pages 10950-10960

  Renier, Laurent A ^a,b   Anurova, Irina ^a,c   De Volder, Anne G ^b   Carlson, Synnöve ^c,d,e   VanMeter, John ^a   Rauschecker, Josef P ^a  

^a GEORGETOWN UNIVERSITY MEDICAL CENTER   (United States)

^b UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^c UNIVERSITY OF HELSINKI   (Finland)

^d AALTO UNIVERSITY   (Finland)

^e TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; BRAIN REGION; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEARING; HUMAN; HUMAN EXPERIMENT; INSULA; MALE; NORMAL HUMAN; PARIETAL LOBE; PRIORITY JOURNAL; TACTILE STIMULATION; TOUCH; VIBRATION;

ACOUSTIC STIMULATION; ADULT; AUDITORY PERCEPTION; FEMALE; FUNCTIONAL LATERALITY; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; MIDDLE AGED; PARIETAL LOBE; SOUND LOCALIZATION; TOUCH; VIBRATION;

EID: 69749113843     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.0910-09.2009     Document Type: Article

References (107)

1. Ackermann H, Riecker A (2004) The contribution of the insula to motor aspects of speech production: a review and a hypothesis. Brain Lang 89:320-328. (Pubitemid 38447179)
2. Alain C, Arnott SR, Hevenor S, Graham S, Grady CL (2001) "What" and "where" in the human auditory system. Proc Natl Acad Sci U S A 98:12301-12306.
3. Alain C, He Y, Grady C (2008) The contribution of the inferior parietal lobe to auditory spatial working memory. J Cogn Neurosci 20:285-295. (Pubitemid 351159090)
4. Altmann CF, Bledowski C, Wibral M, Kaiser J (2007) Processing of location and pattern changes of natural sounds in the human auditory cortex. Neuroimage 35:1192-1200. (Pubitemid 46507821)
5. Altmann CF, Henning M, Döring MK, Kaiser J (2008) Effects of feature-selective attention on auditory pattern and location processing. Neuroimage 41:69-79.
6. Amedi A, Malach R, Hendler T, Peled S, Zohary E (2001) Visuo-haptic object-related activation in the ventral visual pathway. Nat Neurosci 4:324-330. (Pubitemid 32194882)
7. Amedi A, Jacobson G, Hendler T, Malach R, Zohary E (2002) Convergence of visual and tactile shape processing in the human lateral occipital complex. Cereb Cortex 12:1202-1212. (Pubitemid 35191421)
8. Anurova I, Artchakov D, Korvenoja A, Ilmoniemi RJ, Aronen HJ, Carlson S (2003) Differences between auditory evoked responses recorded during spatial and nonspatial working memory tasks. Neuroimage 20:1181-1192. (Pubitemid 37268136)
9. Arnott SR, Binns MA, Grady CL, Alain C (2004) Assessing the auditory dual-pathway model in humans. Neuroimage 22:401-408.
10. Arnott SR, Grady CL, Hevenor SJ, Graham S, Alain C (2005) The functional organization of auditory working memory as revealed by fMRI. J Cogn Neurosci 17:819-831. (Pubitemid 40632502)
11. Augustine JR (1996) Circuitry and functional aspects of the insular lobe in primates including humans. Brain Res Brain Res Rev 22:229-244.
12. Bamiou DE, Musiek FE, Luxon LM (2003) The insula (Island of Reil) and its role in auditory processing. Literature review. Brain Res Brain Res Rev 42:143-154.
13. Bamiou DE, Musiek FE, Stow I, Stevens J, Cipolotti L, Brown MM, Luxon LM (2006) Auditory temporal processing deficits in patients with insular stroke. Neurology 67:614-619. (Pubitemid 44273607)
14. Banati RB, Goerres GW, Tjoa C, Aggleton JP, Grasby P (2000) The functional anatomy of visual-tactile integration in man: a study using positron emission tomography. Neuropsychologia 38:115-124.
15. Beauchamp MS, Lee KE, Argall BD, Martin A (2004a) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809-823.
16. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004b) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190-1192. (Pubitemid 39426237)
17. Belger A, Puce A, Krystal JH, Gore JC, Goldman-Rakic P, McCarthy G (1998) Dissociation of mnemonic and perceptual processes during spatial and nonspatial working memory using fMRI. Hum Brain Mapp 6:14-32. (Pubitemid 28080008)
18. Boynton GM, Engel SA, Glover GH, Heeger DJ (1996) Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci 16:4207-4221. (Pubitemid 26192416)
19. Bremmer F, Schlack A, Shah NJ, Zafiris O, Kubischik M, Hoffmann K, Zilles K, Fink GR (2001) Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys. Neuron 29:287-296. (Pubitemid 32108740)
20. Burton H, McLaren DG, Sinclair RJ (2006) Reading embossed capital letters: an fMRI study in blind and sighted individuals. Hum Brain Mapp 27:325-339.
21. Bushara KO, Weeks RA, Ishii K, Catalan MJ, Tian B, Rauschecker JP, Hallet M (1999) Modality-specific frontal and parietal areas for auditory and visual spatial localization in humans. Nat Neurosci 2:759-766. (Pubitemid 30491199)
22. Caetano G, Jousmäki V (2006) Evidence of vibrotactile input to human auditory cortex. Neuroimage 29:15-28. (Pubitemid 41828026)
23. Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110-1123. (Pubitemid 33108238)
24. Clarke S, Bellmann Thiran A, Maeder P, Adriani M, Vernet O, Regli L, Cuisenaire O, Thiran JP (2002) What and where in human audition: selective deficits following focal hemispheric lesions. Exp Brain Res 147:8-15. (Pubitemid 35193466)
25. Culham JC, Kanwisher NG (2001) Neuroimaging of cognitive functions in human parietal cortex. Curr Opin Neurobiol 11:157-163. (Pubitemid 32289401)
26. Deibert E, Kraut M, Kremen S, Hart J Jr (1999) Neural pathways in tactile object recognition. Neurology 52:1413-1417. (Pubitemid 29190533)
27. De Santis L, Spierer L, Clarke S, Murray MM (2007) Getting in touch: segregated somatosensory what and where pathways in humans revealed by electrical neuroimaging. Neuroimage 37:890-903. (Pubitemid 47248318)
28. D'Esposito M, Aguirre GK, Zarahn E, Ballard D, Shin RK, Lease J (1998) Functional MRI studies of spatial and nonspatial working memory. Brain Res Cogn Brain Res 7:1-13. (Pubitemid 28419435)
29. Downar J, Crawley AP, Mikulis DJ, Davis KD (2000) A multimodal cortical network for the detection of changes in the sensory environment. Nat Neurosci 3:277-283. (Pubitemid 30140791)
30. Drzezga A, Grimmer T, Peller M, Wermke M, Siebner H, Rauschecker JP, Schwaiger M, Kurz A (2005) Impaired cross-modal inhibition in Alzheimer disease. PLoS Med 2:e288. (Pubitemid 41549950)
31. Forster B, Eimer M (2004) The attentional selection of spatial and non-spatial attributes in touch: ERP evidence for parallel and independent processes. Biol Psychol 66:1-20. (Pubitemid 38281177)
32. Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol 88:540-543. (Pubitemid 34755375)
33. Friston KJ, Frith CD, Turner R, Frackowiak RS (1995) Characterizing evoked hemodynamics with fMRI. Neuroimage 2:157-165.
34. Fu KM, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE (2003) Auditory cortical neurons respond to somatosensory stimulation. J Neurosci 23:7510-7515. (Pubitemid 37034067)
35. Golaszewski SM, Siedentopf CM, Baldauf E, Koppelstaetter F, Eisner W, Unterrainer J, Guendisch GM, Mottaghy FM, Felber SR (2002) Functional magnetic resonance imaging of the human sensorimotor cortex using a novel vibrotactile stimulator. Neuroimage 17:421-430.
36. Goodale MA, Milner AD (1992) Separate visual pathways for perception and action. Trends Neurosci 15:20-25.
37. Hackett TA, De La Mothe LA, Ulbert I, Karmos G, Smiley J, Schroeder CE (2007) Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. J Comp Neurol 502:924-952. (Pubitemid 46806628)
38. Hadjikhani N, Roland PE (1998) Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J Neurosci 18:1072-1084. (Pubitemid 28112928)
39. Harada T, Saito DN, Kashikura K, Sato T, Yonekura Y, Honda M, Sadato N (2004) Asymmetrical neural substrates of tactile discrimination in humans: a functional magnetic resonance imaging study. J Neurosci 24:7524-7530. (Pubitemid 39166743)
40. Haxby JV, Grady CL, Horwitz B, Ungerleider LG, Mishkin M, Carson RE, Herscovitch P, Schapiro MB, Rapoport SI (1991) Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proc Natl Acad Sci U S A 88:1621-1625. (Pubitemid 21916277)
41. Hyvärinen J, Poranen A (1974) Function of parietal associative area 7 as revealed from cellular discharges in alert monkeys. Brain 97:673-692.
42. Hyvärinen J, Carlson S, Hyvärinen L (1981) Early visual deprivation alters modality of neuronal responses in area 19 of monkey cortex. Neurosci Lett 26:239-243.
43. Jager G, Postma A (2003) On the hemispheric specialization for categorical and coordinate spatial relations: a review of the current evidence. Neuropsychologia 41:504-515. (Pubitemid 36143678)
44. Kayser C, Petkov CI, Augath M, Logothetis NK (2005) Integration of touch and sound in auditory cortex. Neuron 48:373-384. (Pubitemid 41476368)
45. Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. J Neurosci 27:1824-1835. (Pubitemid 46310306)
46. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18:1560-1574. (Pubitemid 351882035)
47. Kitada R, Kito T, Saito DN, Kochiyama T, Matsumura M, Sadato N, Lederman SJ (2006) Multisensory activation of the intraparietal area when classifying grating orientation: a functional magnetic resonance imaging study. J Neurosci 26:7491-7501. (Pubitemid 44315218)
48. Lancaster JL, Woldorff MG, Parsons LM, Liotti M, Freitas CS, Rainey L, Kochunov PV, Nickerson D, Mikiten SA, Fox PT (2000) Automated Talairach Atlas labels for functional brain mapping. Hum Brain Mapp 10:120-131. (Pubitemid 30456920)
49. Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE (2002) Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci 14:420-429.
50. Lehnert G, Zimmer HD (2008a) Common coding of auditory and visual spatial information in working memory. Brain Res 1230:158-167.
51. Lehnert G, Zimmer HD (2008b) Modality and domain specific components in auditory and visual working memory tasks. Cogn Process 9:53-61. (Pubitemid 351317122)
52. Lewald J, Riederer KA, Lentz T, Meister IG (2008) Processing of sound location in human cortex. Eur J Neurosci 27:1261-1270.
53. Maeder PP, Meuli RA, Adriani M, Bellmann A, Fornari E, Thiran JP, Pittet A, Clarke S (2001) Distinct pathways involved in sound recognition and localization: a human fMRI study. Neuroimage 14:802-816. (Pubitemid 32929881)
54. Martinkauppi S, Rämä P, Aronen HJ, Korvenoja A, Carlson S (2000) Working memory of auditory localization. Cereb Cortex 10:889-898.
55. Merabet L, Thut G, Murray B, Andrews J, Hsiao S, Pascual-Leone A (2004) Feeling by sight or seeing by touch? Neuron 42:173-179.
56. Meyer M, Baumann S, Marchina S, Jancke L (2007) Hemodynamic responses in human multisensory and auditory association cortex to purely visual stimulation. BMC Neurosci 8:14. (Pubitemid 46306425)
57. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze HJ, Driver J (2007) Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J Neurosci 27:11431-11441. (Pubitemid 47621322)
58. Pascual-Leone A, Hamilton R (2001) The metamodal organization of the brain. Prog Brain Res 134:427-445.
59. Peltier S, Stilla R, Mariola E, LaConte S, Hu X, Sathian K (2007) Activity and effective connectivity of parietal and occipital cortical regions during haptic shape perception. Neuropsychologia 45:476-483. (Pubitemid 44894461)
60. Pietrini P, Furey ML, Ricciardi E, Gobbini MI, Wu WH, Cohen L, Guazzelli M, Haxby JV (2004) Beyond sensory images: object-based representation in the human ventral pathway. Proc Natl Acad Sci U S A 101:5658-5663. (Pubitemid 38481212)
61. Poremba A, Saunders RC, Crane AM, Cook M, Sokoloff L, Mishkin M (2003) Functional mapping of the primate auditory system. Science 299:568-572. (Pubitemid 36135155)
62. Prather SC, Votaw JR, Sathian K (2004) Task-specific recruitment of dorsal and ventral visual areas during tactile perception. Neuropsychologia 42:1079-1087. (Pubitemid 38456895)
63. Rämä P (2008) Domain-dependent activation during spatial and nonspatial auditory working memory. Cogn Process 9:29-34. (Pubitemid 351317117)
64. Rauschecker JP (1997) Processing of complex sounds in the auditory cortex of cat, monkey, and man. Acta Otolaryngol Suppl 532:34-38. (Pubitemid 27521378)
65. Rauschecker JP (1998a) Parallel processing in the auditory cortex of primates. Audiol Neurootol 3:86-103. (Pubitemid 28159884)
66. Rauschecker JP (1998b) Cortical processing of complex sounds. Curr Opin Neurobiol 8:516-521.
67. Rauschecker JP, Scott SK (2009) Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing. Nat Neurosci 12:718-724.
68. Rauschecker JP, Tian B (2000) Mechanisms and streams for processing of "what" and "where" in auditory cortex. Proc Natl Acad Sci U S A 97:11800-11806.
69. Rauschecker JP, Tian B, Pons T, Mishkin M (1997) Serial and parallel processing in rhesus monkey auditory cortex. J Comp Neurol 382:89-103. (Pubitemid 27194661)
70. Reed CL, Klatzky RL, Halgren E (2005) What vs where in touch: an fMRI study. Neuroimage 25:718-726.
71. Remedios R, Logothetis NK, Kayser C (2009) An auditory region in the primate insular cortex responding preferentially to vocal communication sounds. J Neurosci 29:1034-1045.
72. Renier L, Collignon O, Poirier C, Tranduy D, Vanlierde A, Bol A, Veraart C, De Volder AG (2005) Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage 26:573-580. (Pubitemid 40719835)
73. Ricciardi E, Bonino D, Gentili C, Sani L, Pietrini P, Vecchi T (2006) Neural correlates of spatial working memory in humans: a functional magnetic resonance imaging study comparing visual and tactile processes. Neuroscience 139:339-349.
74. Rodgers KM, Benison AM, Klein A, Barth DS (2008) Auditory, somatosensory, and multisensory insular cortex in the rat. Cereb Cortex 18:2941-2951.
75. Romanski LM, Goldman-Rakic PS (2002) An auditory domain in primate prefrontal cortex. Nat Neurosci 5:15-16.
76. Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP (1999) Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci 2:1131-1136. (Pubitemid 30126373)
77. Roth EC, Hellige JB (1998) Spatial processing and hemispheric asymmetry. Contributions of the transient/magnocellular visual system. J Cogn Neurosci 10:472-484.
78. Sathian K (2005) Visual cortical activity during tactile perception in the sighted and the visually deprived. Dev Psychobiol 46:279-286.
79. Schroeder CE, Foxe J (2005) Multisensory contributions to low-level, 'unisensory' processing. Curr Opin Neurobiol 15:454-458.
80. Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85:1322-1327. (Pubitemid 32209618)
81. Senkowski D, Saint-Amour D, Kelly SP, Foxe JJ (2007) Multisensory processing of naturalistic objects in motion: a high-density electrical mapping and source estimation study. Neuroimage 36:877-888. (Pubitemid 46908959)
82. Smiley JF, Hackett TA, Ulbert I, Karmas G, Lakatos P, Javitt DC, Schroeder CE (2007) Multisensory convergence in auditory cortex. I. Cortical connections of the caudal superior temporal plane in macaque monkeys. J Comp Neurol 502:894-923. (Pubitemid 46806627)
83. Smith EE, Jonides J (1998) Neuroimaging analyses of human working memory. Proc Natl Acad Sci U S A 95:12061-12068. (Pubitemid 28460539)
84. Stoesz MR, Zhang M, Weisser VD, Prather SC, Mao H, Sathian K (2003) Neural networks active during tactile form perception: common and differential activity during macrospatial and microspatial tasks. Int J Psychophysiol 50:41-49. (Pubitemid 37162870)
85. Stricanne B, Andersen RA, Mazzoni P (1996) Eye-centered, head-centered, and intermediate coding of remembered sound locations in area LIP. J Neurophysiol 76:2071-2076. (Pubitemid 26322640)
86. Talairach G, Tournoux P (1988) Co-planar stereotaxic atlas of the human brain. New York: Thieme.
87. Tanabe HC, Honda M, Sadato N (2005) Functionally segregated neural substrates for arbitrary audiovisual paired-association learning. J Neurosci 25:6409-6418. (Pubitemid 40962537)
88. Tian B, Reser D, Durham A, Kustov A, Rauschecker JP (2001) Functional specialization in rhesus monkey auditory cortex. Science 292:290-293. (Pubitemid 32295388)
89. Ungerleider LG, and Mishkin M (1982) Two cortical visual systems. In: Analysis of visual behavior (Ingle DJ, Goodale MA, Mansfield RJW, eds), pp 549-586. Cambridge, MA: MIT.
90. van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271-282. (Pubitemid 38942966)
91. van Atteveldt NM, Formisano E, Goebel R, Blomert L (2007) Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 36:1345-1360. (Pubitemid 47001726)
92. Van Boven RW, Ingeholm JE, Beauchamp MS, Bikle PC, Ungerleider LG (2005) Tactile form and location processing in the human brain. Proc Natl Acad Sci U S A 102:12601-12605. (Pubitemid 41266405)
93. Wager TD, Smith EE (2003) Neuroimaging studies of working memory: a meta-analysis. Cogn Affect Behav Neurosci 3:255-274.
94. Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci U S A 101:2167-2172. (Pubitemid 38229033)
95. Warren JD, Griffiths TD (2003) Distinct mechanisms for processing spatial sequences and pitch sequences in the human auditory brain. J Neurosci 23:5799-5804.
96. Weaver KE, Stevens AA (2007) Attention and sensory interactions within the occipital cortex in the early blind: an fMRI study. J Cogn Neurosci 19:315-330.
97. Weeks R, Horwitz B, Aziz-Sultan A, Tian B, Wessinger CM, Cohen LG, Hallett M, Rauschecker JP (2000) A positron emission tomographic study of auditory localization in the congenitally blind. J Neurosci 20:2664-2672. (Pubitemid 30220354)
98. Weeks RA, Aziz-Sultan A, Bushara KO, Tian B, Wessinger CM, Dang N, Rauschecker JP, Hallett M (1999) A PET study of human auditory spatial processing. Neurosci Lett 262:155-158. (Pubitemid 29136793)
99. Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003) Polysensory interactions along lateral temporal regions evoked by audio-visual speech. Cereb Cortex 13:1034-1043. (Pubitemid 37163202)
100. Zangaladze A, Epstein CM, Grafton ST, Sathian K (1999) Involvement of visual cortex in tactile discrimination of orientation. Nature 401:587-590. (Pubitemid 129517072)
101. Zatorre RJ, Bouffard M, Ahad P, Belin P (2002) Where is 'where' in the human auditory cortex? Nat Neurosci 5:905-909.
102. Zeki SM (1978) Functional specialisation in the visual cortex of the rhesus monkey. Nature 274:423-428. (Pubitemid 8396798)
103. Zhang M, Weisser VD, Stilla R, Prather SC, Sathian K (2004) Multisensory cortical processing of object shape and its relation to mental imagery. Cogn Affect Behav Neurosci 4:251-259. (Pubitemid 39167167)
104. Zhang M, Mariola E, Stilla R, Stoesz M, Mao H, Hu X, Sathian K (2005) Tactile discrimination of grating orientation: fMRI activation patterns. Hum Brain Mapp 25:370-377. (Pubitemid 41105592)
105. Zimmer HD (2008) Visual and spatial working memory: from boxes to networks. Neurosci Biobehav Rev 32:1373-1395.
106. Zimmer U, Lewald J, Erb M, Grodd W, Karnath HO (2004) Is there a role of visual cortex in spatial hearing? Eur J Neurosci 20:3148-3156.
107. Zimmer U, Lewald J, Erb M, Karnath HO (2006) Processing of auditory spatial cues in human cortex: an fMRI study. Neuropsychologia 44:454-461. (Pubitemid 41821907)