The contributions of transient and sustained response codes to audiovisual integration

Cerebral Cortex

Volumn 21, Issue 4, 2011, Pages 920-931

  Werner, Sebastian ^a   Noppeney, Uta ^a  

^a MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

Author keywords

amplitude modulation; crossmodal; fMRI; multisensory; sustained; transient

Indexed keywords

OXYGEN;

ADULT; AMPLITUDE MODULATION; ARTICLE; AUDIOVISUAL EQUIPMENT; AUDITORY STIMULATION; BOLD SIGNAL; BRAIN CORTEX; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; MALE; NORMAL HUMAN; PRIORITY JOURNAL; SENSORIMOTOR INTEGRATION; VISUAL STIMULATION;

ADULT; AUDITORY PERCEPTION; BRAIN MAPPING; FEMALE; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; VISUAL PERCEPTION;

EID: 79952858494     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhq161     Document Type: Article

References (87)

1. Amedi A, von Kriegstein K, Van Atteveldt NM, Beauchamp MS, Naumer MJ. 2005. Functional imaging of human crossmodal identification and object recognition. Exp Brain Res. 166:559-571. (Pubitemid 41502099)
2. Barraclough NE, Xiao DK, Baker CI, Oram MW, Perrett DI. 2005. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J Cogn Neurosci. 17:377-391. (Pubitemid 40364527)
3. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A. 2004. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci. 7:1190-1192. (Pubitemid 39426237)
4. Besle J, Fischer C, Bidet-Caulet A, Lecaignard F, Bertrand O, Giard MH. 2008. Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J Neurosci. 28:14301-14310.
5. Boemio A, Fromm S, Braun A, Poeppel D. 2005. Hierarchical and asymmetric temporal sensitivity in human auditory cortices. Nat Neurosci. 8:389-395. (Pubitemid 40300197)
6. Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA. 2007. Neural basis of the ventriloquist illusion. Curr Biol. 17:1697-1703. (Pubitemid 47503808)
7. Brett M, Anton JL, Valabregue R, Poline JB. 2002. Region of interest analysis using an SPM toolbox. Neuroimage. 16(2), abstract 497 (available on CD-ROM): 8th International Conference on Functional Mapping of the Human Brain (OHBM); 2002, June 2-6; Sendai, Japan.
8. Busse L, Roberts KC, Crist RE, Weissman DH, Woldorff MG. 2005. The spread of attention across modalities and space in a multisensory object. Proc Natl Acad Sci USA. 102:18751-18756. (Pubitemid 43011298)
9. Calvert GA. 2001. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex. 11:1110-1123. (Pubitemid 33108238)
10. Calvert GA, Lewis JW. 2004. Hemodynamic studies of audiovisual interactions. In: Calvert GA, Spence C, Stein BE, editors. The handbook of multisensory processes. Cambridge (MA): MIT Press. p. 483-502.
11. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA. 2009. The natural statistics of audiovisual speech. PLoS Comput Biol. 5:e1000436.
12. Dahl CD, Logothetis NK, Kayser C. 2009. Spatial organization of multisensory responses in temporal association cortex. J Neurosci. 29:11924-11932.
13. Dhamala M, Assisi CG, Jirsa VK, Steinberg FL, Kelso JA. 2007. Multisensory integration for timing engages different brain networks. Neuroimage. 34:764-773. (Pubitemid 44827750)
14. Donaldson DI, Petersen SE, Ollinger JM, Buckner RL. 2001. Dissociating state and item components of recognition memory using fMRI. Neuroimage. 13:129-142. (Pubitemid 34188251)
15. Downar J, Crawley AP, Mikulis DJ, Davis KD. 2000. A multimodal cortical network for the detection of changes in the sensory environment. Nat Neurosci. 3:277-283. (Pubitemid 30140791)
16. Driver J, Noesselt T. 2008. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron. 57:11-23.
17. Evans AC, Marrett S, Neelin P, Collins L, Worsley K, Dai W, Milot S, Meyer E, Bub D. 1992. Anatomical mapping of functional activation in stereotactic coordinate space. Neuroimage. 1:43-53.
18. Fairhall SL, Macaluso E. 2009. Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. Eur J Neurosci. 29:1247-1257.
19. Falchier A, Clavagnier S, Barone P, Kennedy H. 2002. Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci. 22:5749-5759. (Pubitemid 35386498)
20. Foxe JJ, Schroeder CE. 2005. The case for feedforward multisensory convergence during early cortical processing. Neuroreport. 16:419-423. (Pubitemid 40477541)
21. Friston K. 2010. The free-energy principle: a unified brain theory? Nat Rev Neurosci. 11:127-138.
22. Friston KJ, Holmes AP, Price CJ, Buchel C, Worsley KJ. 1999. Multisubject fMRI studies and conjunction analyses. Neuroimage. 10:385-396. (Pubitemid 29472824)
23. Friston KJ, Holmes AP, Worsley KJ, Poline JB, Frith CD, Frackowiak RSJ. 1995. Statistical parametric maps in functional imaging: a general linear approach. Hum Brain Mapp. 2:189-210. (Pubitemid 124008664)
24. Friston KJ, Worsley KJ, Frackowiak RSJ, Mazziotta JC, Evans AC. 1994. Assessing the significance of focal activations using their spatial extent. Hum Brain Mapp. 1:214-220.
25. Garrido MI, Kilner JM, Stephan KE, Friston KJ. 2009. The mismatch negativity: a review of underlying mechanisms. Clin Neurophysiol. 120:453-463.
26. Ghazanfar AA, Chandrasekaran C, Logothetis NK. 2008. Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. J Neurosci. 28:4457-4469.
27. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK. 2005. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci. 25:5004-5012. (Pubitemid 40712843)
28. Ghazanfar AA, Schroeder CE. 2006. Is neocortex essentially multisen-sory? Trends Cogn Sci. 10:278-285.
29. Giraud AL, Lorenzi C, Ashburner J, Wable J, Johnsrude I, Frackowiak R, Kleinschmidt A. 2000. Representation of the temporal envelope of sounds in the human brain. J Neurophysiol. 84:1588-1598.
30. Griffiths TD, Buchel C, Frackowiak RS, Patterson RD. 1998. Analysis of temporal structure in sound by the human brain. Nat Neurosci. 1:422-427.
31. Harms MP, Guinan JJ Jr, Sigalovsky IS, Melcher JR. 2005. Short-term sound temporal envelope characteristics determine multisecond time patterns of activity in human auditory cortex as shown by fMRI. J Neurophysiol. 93:210-222. (Pubitemid 40637154)
32. Harms MP, Melcher JR. 2002. Sound repetition rate in the human auditory pathway: representations in the waveshape and amplitude of fMRI activation. J Neurophysiol. 88:1433-1450. (Pubitemid 35025397)
33. Hasson U, Yang E, Vallines I, Heeger DJ, Rubin N. 2008. A hierarchy of temporal receptive windows in human cortex. J Neurosci. 28:2539-2550. (Pubitemid 351348202)
34. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ. 2007. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci. 27:7881-7887. (Pubitemid 47173312)
35. Herdener M, Esposito F, Di SF, Lehmann C, Bach DR, Scheffler K, Seifritz E. 2007. BOLD correlates of edge detection in human auditory cortex. Neuroimage. 36:194-201. (Pubitemid 46617941)
36. Herdener M, Lehmann C, Esposito F, di Salle F, Federspiel A, Bach DR, Scheffler K, Seifritz E. 2009. Brain responses to auditory and visual stimulus offset: shared representations of temporal edges. Hum Brain Mapp. 30:725-733.
37. Josephs O, Turner R, Friston K. 1997. Event-related fMRI. Hum Brain Mapp. 5:243-248. (Pubitemid 27375522)
38. Kayser C, Petkov CI, Lippert M, Logothetis NK. 2005. Mechanisms for allocating auditory attention: an auditory saliency map. Curr Biol. 15:1943-1947. (Pubitemid 41566142)
39. Kayser C, Petkov CI, Logothetis NK. 2008. Visual modulation of neurons in auditory cortex. Cereb Cortex. 18:1560-1574. (Pubitemid 351882035)
40. Kiebel SJ, Daunizeau J, Friston KJ. 2008. A hierarchy of time-scales and the brain. PLoS Comput Biol. 4:e1000209.
41. Lakatos P, Chen CM, O'Connell MN, Mills A, Schroeder CE. 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 53:279-292. (Pubitemid 46074398)
42. Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE. 2002. Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci. 14:420-429. (Pubitemid 34267542)
43. Lehmann C, Herdener M, Esposito F, Hubl D, di Salle F, Scheffler K, Bach DR, Federspiel A, Kretz R, Dierks T, et al. 2006. Differential patterns of multisensory interactions in core and belt areas of human auditory cortex. Neuroimage. 31:294-300. (Pubitemid 43728969)
44. Lewis RK, Noppeney U. Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas. J Neurosci. Forthcoming.
45. Macaluso E, Driver J, Frith CD. 2003. Multimodal spatial representations engaged in human parietal cortex during both saccadic and manual spatial orienting. Curr Biol. 13:990-999. (Pubitemid 36726034)
46. Maier JX, Di Luca M, Noppeney U. Forthcoming. Audiovisual asynchrony detection in human speech. J Exp Psychol Hum Percept Perform.
47. Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP, Clarke S, Meuli RA. 2007. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb Cortex. 17:1672-1679.
48. Meienbrock A, Naumer MJ, Doehrmann O, Singer W, Muckli L. 2007. Retinotopic effects during spatial audio-visual integration. Neuro-psychologia. 45:531-539. (Pubitemid 44894467)
49. Miezin FM, Maccotta L, Ollinger JM, Petersen SE, Buckner RL. 2000. Characterizing the hemodynamic response: effects of presentation rate, sampling procedure, and the possibility of ordering brain activity based on relative timing. Neuroimage. 11:735-759. (Pubitemid 30437663)
50. Miller LM, D'Esposito M. 2005. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci. 25:5884-5893. (Pubitemid 40911419)
51. Mishra J, Martinez A, Sejnowski TJ, Hillyard SA. 2007. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J Neurosci. 27:4120-4131. (Pubitemid 46597162)
52. Molholm S, Ritter W, Javitt DC, Foxe JJ. 2004. Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex. 14:452-465. (Pubitemid 38506976)
53. Musacchia G, Schroeder CE. 2009. Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res. 258:72-79.
54. Neal JW, Pearson RCA, Powell TPS. 1990. The connections of area PG, 7a, with cortex in the parietal, occipital and temporal lobes of the monkey. Brain Res. 532:249-264. (Pubitemid 20384665)
55. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze HJ, Driver J. 2007. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J Neurosci. 27:11431-11441. (Pubitemid 47621322)
56. Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ. 2008. The effect of prior visual information on recognition of speech and sounds. Cereb Cortex. 18:598-609. (Pubitemid 351264229)
57. Noppeney U, Ostwald D, Werner S. 2010. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci. 30:7434-7446.
58. Otten LJ, Henson RN, Rugg MD. 2002. State-related and item-related neural correlates of successful memory encoding. Nat Neurosci. 5:1339-1344. (Pubitemid 35417770)
59. Overath T, Kumar S, von KK, Griffiths TD. 2008. Encoding of spectral correlation over time in auditory cortex. J Neurosci. 28:13268-13273.
60. Perrault TJ, Vaughan W, Stein BE, Wallace MT. 2005. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol. 93:2575-2586. (Pubitemid 40570840)
61. Petkov CI, Kang XJ, Alho K, Bertrand O, Yund EW, Woods DL. 2004. Attentional modulation of human auditory cortex. Nat Neurosci. 7:658-663. (Pubitemid 38691906)
62. Rao RP, Ballard DH. 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci. 2:79-87. (Pubitemid 30488944)
63. Rockland KS, Ojima H. 2003. Multisensory convergence in calcarine visual areas in macaque monkey. Int J Psychophysiol. 50:19-26. (Pubitemid 37162868)
64. Sadaghiani S, Maier JX, Noppeney U. 2009. Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing. J Neurosci. 29:6490-6499.
65. Saito DN, Yoshimura K, Kochiyama T, Okada T, Honda M, Sadato N. 2005. Cross-modal binding and activated attentional networks during audio-visual speech integration: a functional MRI study. Cereb Cortex. 15:1750-1760. (Pubitemid 41447040)
66. Schlack A, Sterbing-D'Angelo SJ, Hartung K, Hoffmann KP, Bremmer F. 2005. Multisensory space representations in the macaque ventral intraparietal area. J Neurosci. 25:4616-4625. (Pubitemid 40629152)
67. Schroeder CE, Foxe JJ. 2002. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res. 14:187-198. (Pubitemid 34626809)
68. Schroeder CE, Smiley J, Fu KG, McGinnis T, O'Connell MN, Hackett TA. 2003. Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. Int J Psychophysiol. 50:5-17. (Pubitemid 37168012)
69. Seifritz E, Esposito F, Hennel F, Mustovic H, Neuhoff JG, Bilecen D, Tedeschi G, Scheffler K, Di SF. 2002. Spatiotemporal pattern of neural processing in the human auditory cortex. Science. 297:1706-1708. (Pubitemid 35000786)
70. Seltzer B, Pandya DN. 1994. Parietal, temporal, and occipital projections to cortex of the superior temporal sulcus in the rhesus-monkey: a retrograde tracer study. J Comp Neurol. 343:445-463. (Pubitemid 24129553)
71. Somers DC, Dale AM, Seiffert AE, Tootell RB. 1999. Functional MRI reveals spatially specific attentional modulation in human primary visual cortex. Proc Natl Acad Sci USA. 96:1663-1668. (Pubitemid 29098508)
72. Stanford TR, Quessy S, Stein BE. 2005. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci. 25:6499-6508. (Pubitemid 40995282)
73. Stein BE, Meredith MA. 1993. The merging of the senses. Cambridge (MA): The MIT Press.
74. Stein BE, Stanford TR. 2008. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci. 9:255-266. (Pubitemid 351430829)
75. Stevenson RA, James TW. 2009. Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage. 44:1210-1223.
76. Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N, Mazoyer B, Joliot M. 2002. Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage. 15:273-289. (Pubitemid 34852226)
77. Uludag K. 2008. Transient and sustained BOLD responses to sustained visual stimulation. Magn Reson Imaging. 26:863-869.
78. Van Atteveldt N, Formisano E, Goebel R, Blomert L. 2004. Integration of letters and speech sounds in the human brain. Neuron. 43:271-282. (Pubitemid 38942966)
79. Van Atteveldt N, Roebroeck A, Goebel R. 2009. Interaction of speech and script in human auditory cortex: insights from neuro-imaging and effective connectivity. Hear Res. 258:152-164.
80. Van Wassenhove V, Grant KW, Poeppel D. 2007. Temporal window of integration in auditory-visual speech perception. Neuropsychologia. 45:598-607. (Pubitemid 44894453)
81. Vatakis A, Spence C. 2006. Audiovisual synchrony perception for music, speech, and object actions. Brain Res. 1111:134-142. (Pubitemid 44331172)
82. Wallace MT, Wilkinson LK, Stein BE. 1996. Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol. 76:1246-1266. (Pubitemid 26284603)
83. Wandell BA, Dumoulin SO, Brewer AA. 2007. Visual field maps in human cortex. Neuron. 56:366-383.
84. Watkins S, Shams L, Tanaka S, Haynes JD, Rees G. 2006. Sound alters activity in human V1 in association with illusory visual perception. Neuroimage. 31:1247-1256. (Pubitemid 43870257)
85. Werner S, Noppeney U. 2009. Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex. Advance Access published November 18, doi: 10.1093/cercor/bhp248.
86. Werner S, Noppeney U. 2010. Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci. 30:2662-2675.
87. Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G. 2003. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex. 13:1034-1043. (Pubitemid 37163202)