Multisensory integration: Current issues from the perspective of the single neuron

Nature Reviews Neuroscience

Volumn 9, Issue 4, 2008, Pages 255-266

  Stein, Barry E ^a   Stanford, Terrence R ^a  

^a WAKE FOREST SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BRAIN CORTEX; EVOKED AUDITORY RESPONSE; EVOKED SOMATOSENSORY RESPONSE; HUMAN; MULTISENSORY INTEGRATION; NERVE CELL; NERVE CELL NETWORK; NEUROPHYSIOLOGY; NONHUMAN; ORIENTING RESPONSE; PRIORITY JOURNAL; REVIEW; SENSORY NERVE CELL; SUPERIOR COLLICULUS; VISUAL ORIENTATION;

AFFERENT PATHWAYS; ANIMALS; BRAIN MAPPING; CEREBRAL CORTEX; HUMANS; MENTAL PROCESSES; NEURONS; SENSATION;

MAMMALIA;

EID: 41149111920     PISSN: 1471003X     EISSN: 14710048     Source Type: Journal    
DOI: 10.1038/nrn2331     Document Type: Review

References (107)

1. Stein, B. E. & Meredith, M. A. The Merging of the Senses (ed. Gazzaniga, M. S.) (The MIT Press, Cambridge, Massachusetts, 1993).
2. Meredith, M. A. & Stein, B. E. Interactions among converging sensory inputs in the superior colliculus. Science 221, 389-391 (1983). This seminal paper demonstrated multisensory integration in single superior colliculus neurons.
3. Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18, 787-792 (2007). This study considers the relatively disproportionate effect of superadditivity in multisensory neurons. Superadditivity in multisensory integration is obtained only when the neuron's response to the component stimuli is weak.
4. Spence, C. & Driver, J. Crossmodal Space and Crossmodal Attention (Oxford Univ. Press, Oxford, 2004).
5. Gillmeister, H. & Eimer, M. Tactile enhancement of auditory detection and perceived loudness. Brain Res. 1160, 58-68 (2007).
6. Calvert, G. A., Spence, C. & Stein, B. E. The Handbook of Multisensory Processes (The MIT Press, Cambridge, Massachusetts, 2004).
7. Bell, A. H., Meredith, M. A., Van Opstal, A. J. & Munoz, D. P. Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements. J. Neurophysiol. 93, 3659-3673 (2005).
8. Diederich, A. & Colonius, H. Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys. 66, 1388-1404 (2004).
9. Frens, M. A., Van Opstal, A. J. & Van der Willigen, R. F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802-816 (1995).
10. Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G. & Fendrich, R. Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J. Exp. Psychol. Hum. Percept. Perform. 20, 131-153 (1994).
11. Nozawa, G., Reuter-Lorenz, P. A. & Hughes, H. C. Parallel and serial processes in the human oculomotor system: bimodal integration and express saccades. Biol. Cybern. 72, 19-34 (1994). This is an early article establishing the relationship between reaction speed and multisensory integration.
12. Rowland, B., Quessy, S., Stanford, T. R. & Stein, B. E. Multisensory integration shortens physiological response latencies. J. Neurosci. 27, 5879-5884 (2007).
13. Burr, D. & Alais, D. Combining visual and auditory information. Prog. Brain Res. 155, 243-258 (2006).
14. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429-433 (2002).
15. Shams, L., Ma, W. J. & Beierholm, U. Sound-induced flash illusion as an optimal percept. Neuroreport 16, 1923-1927 (2005).
16. Kadunce, D. C., Vaughan, J. W., Wallace, M. T. & Stein, B. E. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp. Brain Res. 139, 303-310 (2001).
17. Meredith, M. A. & Stein, B. E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J. Neurophysiol. 75, 1843-1857 (1996).
18. Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350-354 (1986).
19. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G. & Stein, B. E. Mechanisms of within- and cross-modality suppression in the superior colliculus. J. Neurophysiol. 78, 2834-2847 (1997).
20. Hartline, P. H., Vimal, R. L., King, A. J., Kurylo, D. D. & Northmore, D. P. Effects of eye position on auditory localization and neural representation of space in superior colliculus of cats. Exp. Brain Res. 104, 402-408 (1995).
21. Jay, M. F. & Sparks, D. L. Auditory receptive fields in primate superior colliculus shift with changes in eye position. Nature 309, 345-347 (1984). This article is a seminal study of sensory coordinate frames; it is the first to systematically examine the eye-position-dependence of auditory receptive fields in the superior colliculus.
22. Peck, C. K., Baro, J. A. & Warder, S. M. Effects of eye position on saccadic eye movements and on the neuronal responses to auditory and visual stimuli in cat superior colliculus. Exp. Brain Res. 103, 227-242 (1995).
23. Groh, J. M. & Sparks, D. L. Saccades to somatosensory targets. III. Eye-position-dependent somatosensory activity in primate superior colliculus. J. Neurophysiol. 75, 439-453 (1996).
24. Meredith, M. A., Nemitz, J. W. & Stein, B. E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215-3229 (1987).
25. Recanzone, G. H. Auditory influences on visual temporal rate perception. J. Neurophysiol. 89, 1078-1093 (2003).
26. Meredith, M. A. & Stein, B. E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640-662 (1986).
27. Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499-6508 (2005).
28. Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575-2586 (2005).
29. Wallace, M. T., Wilkinson, L. K. & Stein, B. E. Representation and integration of multiple sensory inputs in primate superior colliculus. J. Neurophysiol. 76, 1246-1266 (1996).
30. Stein, B. E., Meredith, M. A., Huneycutt, W. S. & McDade, L. Behavioral indices of multisensory integration: orientation to visual cues is affected by visual stimuli. J. Cogn. Neurosci. 1, 12-24 (1989).
31. Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1-10 (1996).
32. Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240-1255 (2002).
33. Jiang, W., Jiang, H. & Stein, B. E. Neonatal cortical ablation disrupts multisensory development in superior colliculus. J. Neurophysiol. 95, 1380-1396 (2006).
34. Wallace, M. T. & Stein, B. E. Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71, 429-432 (1994).
35. Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193-3205 (2007).
36. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506-522 (2001). The critical nature of influences from two cortical areas on multisensory integration in superior colliculus neurons is demonstrated here.
37. Stein, B. E., Spencer, R. F. & Edwards, S. B. Corticotectal and corticothalamic efferent projections of SIV somatosensory cortex in cat. J. Neurophysiol. 50, 896-909 (1983).
38. Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797-1809 (1993).
39. Rowland, B., Stanford, T. R. & Stein, B. E. A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception 36, 1431-1443 (2007).
40. Anastasio, T. J. & Patton, P. E. A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system. J. Neurosci. 23, 6713-6727 (2003).
41. Wallace, M. T., Meredith, M. A. & Stein, B. E. Integration of multiple sensory modalities in cat cortex. Exp. Brain Res. 91, 484-488 (1992).
42. Stein, B. E. & Wallace, M. T. Comparisons of cross-modality integration in midbrain and cortex. Prog. Brain Res. 112, 289-299 (1996).
43. Stricanne, B., Andersen, R. A. & Mazzoni, P. Eye-centered, head-centered, and intermediate coding of remembered sound locations in area LIP. J. Neurophysiol. 76, 2071-2076 (1996). This was one of the first studies to demonstrate that the auditory receptive fields of LIP neurons can be coded in an eye-centred reference frame.
44. Batista, A. P., Buneo, C. A., Snyder, L. H. & Andersen, R. A. Reach plans in eye-centered coordinates. Science 285, 257-260 (1999).
45. Cohen, Y. E. & Andersen, R. A. Reaches to sounds encoded in an eye-centered reference frame. Neuron 27, 647-652 (2000).
46. Cohen, Y. E. & Andersen, R. A. A common reference frame for movement plans in the posterior parietal cortex. Nature Rev. Neurosci. 3, 553-562 (2002).
47. Pouget, A., Ducom, J. C., Torri, J. & Bavelier, D. Multisensory spatial representations in eye-centered coordinates for reaching. Cognition 83, 1-11 (2002).
48. Pouget, A., Deneve, S. & Duhamel, J. R. A computational perspective on the neural basis of multisensory spatial representations. Nature Rev. Neurosci. 3, 741-747 (2002). In this article, the issue of partial reference-frame shifts is considered in detail from a computational perspective.
49. Avillac, M., Deneve, S., Olivier, E., Pouget, A. & Duhamel, J. R. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neurosci. 8, 941-949 (2005).
50. Mullette-Gillman, O. A., Cohen, Y. E. & Groh, J. M. Eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus. J. Neurophysiol. 94, 2331-2352 (2005).
51. Schlack, A., Sterbing-D'Angelo, S. J., Hartung, K., Hoffmann, K.-P. & Bremmer, F. Multisensory space representations in the macaque ventral intraparietal area. J. Neurosci. 25, 4616-4625 (2005).
52. Snyder, L. H. Frame-up. Focus on "eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus". J. Neurophysiol. 94, 2259-2260 (2005).
53. Avillac, M., Ben Hamed, S. & Duhamel, J. R. Multisensory integration in the ventral intraparietal area of the macaque monkey. J. Neurosci. 27, 1922-1932 (2007).
54. Laurienti, P. J. et al. Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum. Brain Mapp. 19, 213-223 (2003).
55. Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W. & Perrett, D. I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 17, 377-391 (2005).
56. Sugihara, T., Diltz, M. D., Averbeck, B. B. & Romanski, L. M. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J. Neurosci. 26, 11138-11147 (2006). This paper is one of the first to examine multisensory integration in single neurons of the prefrontal cortex. It suggests a role for this region in the integration of the visual and auditory stimuli that constitute coherent communication signals.
57. Romanski, L. M. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cereb. Cortex 17, i61-i69 (2007).
58. Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278-285 (2006).
59. Ghazanfar, A. A., Maier, J. X., Hoffman, K. L. & Logothetis, N. K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25, 5004-5012 (2005).
60. Calvert, G. A., Campbell, R. & Brammer, M. J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10, 649-657 (2000).
61. Calvert, G. A., Hansen, P. C., Iversen, S. D. & Brammer, M. J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14, 427-438 (2001). A rationale is provided here for using superadditivity of the BOLD signal as a definitive criterion for detecting multisensory integration in functional-imaging studies.
62. Logothetis, N. K. & Pfeuffer, J. On the nature of the BOLD fMRI contrast mechanism. Magn. Reson. Imaging 22, 1517-1531 (2004).
63. Laurienti, P. J., Perrault, T. J., Stanford, T. R., Wallace, M. T. & Stein, B. E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166, 289-297 (2005).
64. Beauchamp, M. S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3, 93-113 (2005).
65. Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809-823 (2004).
66. Stevenson, R. A., Geoghegan, M. L. & James, T. W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp. Brain Res. 179, 85-95 (2007). Using the stringent superadditivity criterion for BOLD activation, this study is the first to demonstrate that the superior temporal sulcus integrates information about multisensory objects.
67. Foxe, J. J. & Schroeder, C. E. The case for feed-forward multisensory convergence during early cortical processing. Neuroreport 16, 419-423 (2005). The issue dealt with here is how multisensory convergence at the level of the sensory cortex can be accomplished as a consequence of feedforward projections in the CNS.
68. Busse, L., Roberts, K. C., Crist, R. E., Weissman, D. H. & Woldorff, M. G. The spread of attention across modalities and space in a multisensory object. Proc. Natl Acad. Sci. USA 102, 18751-18756 (2005).
69. Talsma, D., Doty, T. J. & Woldorff, M. G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679-690 (2007).
70. Giard, M. H. & Peronnet, F. Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 11, 473-490 (1999). This early event-related-potential study in human subjects demonstrated that multisensory integration can take place very early in the processing of auditory and visual information.
71. Macaluso, E. & Driver, J. Multisensory spatial interactions: a window onto functional integration in the human brain. Trends Neurosci. 28, 264-271 (2005).
72. Bizley, J. K., Nodal, F. R., Bajo, V. M., Nelken, I. & King, A. J. Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb. Cortex 17, 2172-2189 (2007).
73. Cappe, C. & Barone, P. Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. Eur. J. Neurosci. 22, 2886-2902 (2005).
74. Rockland, K. S. & Ojima, H. Multisensory convergence in calcarine visual areas in macaque monkey. Int. J. Psychophysiol. 50, 19-26 (2003).
75. Falchier, A., Clavagnier, S., Barone, P. & Kennedy, H. Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22, 5749-5759 (2002). This paper provided one of the first anatomical demonstrations of non-visual inputs to the primary visual cortex.
76. Meredith, M. A., Keniston, L. R., Dehner, L. R. & Clemo, H. R. Crossmodal projections from somatosensory area SIV to the auditory field of the anterior ectosylvian sulcus (FAES) in cat: further evidence for subthreshold forms of multisensory processing. Exp. Brain Res. 172, 472-484 (2006).
77. Allman, B. L. & Meredith, M. A. Multisensory processing in "unimodal" neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. J. Neurophysiol. 98, 545-549 (2007).
78. Wallace, M. T., Ramachandran, R. & Stein, B. E. A revised view of sensory cortical parcellation. Proc. Natl Acad. Sci. USA 101, 2167-2172 (2004).
79. Brosch, M., Selezneva, E. & Scheich, H. Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J. Neurosci. 25, 6797-6806 (2005).
80. Fishman, M. C. & Michael, P. Integration of auditory information in the cat's visual cortex. Vision Res. 13, 1415-1419 (1973).
81. Morrell, F. Visual system's view of acoustic space. Nature 238, 44-46 (1972).
82. Schroeder, C. E. et al. Somatosensory input to auditory association cortex in the macaque monkey. J. Neurophysiol. 85, 1322-1327 (2001).
83. Penfield, W. & Rasmussen, T. The Cerebral Cortex of Man: A Clinical Study of Localization of Function (Macmillan, New York, 1950).
84. Brindley, G. S. & Lewin, W. S. The sensations produced by electrical stimulation of the visual cortex. J. Physiol. 196, 479-493 (1968).
85. Dobelle, W. H., Stensaas, S. S., Mladejovsky, M. G. & Smith, J. B. A prosthesis for the deaf based on cortical stimulation. Ann. Otol. Rhinol. Laryngol. 82, 445-463 (1973).
86. Penfield, W. Some mechanisms of consciousness discovered during electrical stimulation of the brain. Proc. Natl Acad. Sci. USA 44, 51-66 (1958).
87. Lakatos, P., Chen, C. M., O'Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279-292 (2007).
88. McGurk, H. & MacDonald, J. Hearing lips and seeing voices. Nature 264, 746-748 (1976).
89. Sumby, W. H. & Pollack, I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26, 212-215 (1954).
90. Sams, M. et al. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci. Lett. 127, 141-145 (1991).
91. Howard, I. P. & Templeton, W. B. Human Spatial Orientation (Wiley, London, 1966).
92. Jousmaki, V. & Hari, R. Parchment-skin illusion: sound-biased touch. Curr. Biol. 8, R190 (1998).
93. Shams, L., Kamitani, Y. & Shimojo, S. Visual illusion induced by sound. Brain Res. Cogn. Brain Res. 14, 147-152 (2002).
94. Graybiel, A. Oculogravic illusion. AMA Arch. Ophthalmol. 48, 605-615 (1952).
95. Wallace, M. T., Carriere, B. N., Perrault, T. J. Jr, Vaughan, J. W. & Stein, B. E. The development of cortical multisensory integration. J. Neurosci. 26, 11844-11849 (2006).
96. Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429-2444 (1997).
97. Stein, B. E., Labos, E. & Kruger, L. Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J. Neurophysiol. 36, 667-679 (1973).
98. Wallace, M. T. & Stein, B. E. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21, 8886-8894 (2001).
99. Stein, B. E., Wallace, M. W., Stanford, T. R. & Jiang, W. Cortex governs multisensory integration in the midbrain. Neuroscientist 8, 306-314 (2002).
100. Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578-3582 (2000).
101. Wallace, M. T., Perrault, T. J. Jr, Hairston, W. D. & Stein, B. E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24, 9580-9584 (2004).
102. Wallace, M. T. & Stein, B. E. Early experience determines how the senses will interact. J. Neurophysiol. 97, 921-926 (2007). This was the first paper to show that the gradual postnatal development of multisensory integration in superior colliculus neurons allows early experience to form its operational principles.
103. Rowland, B., Jiang, W. & Stein, B. E. Long-term plasticity in multisensory integration. Soc. Neurosci. Abstr. 31, 505.8 (2005).
104. Neil, P. A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D. J. & Shimojo, S. Development of multisensory spatial integration and perception in humans. Dev. Sci. 9, 454-464 (2006).
105. Putzar, L., Goerendt, I., Lange, K., Rosler, F. & Roder, B. Early visual deprivation impairs multisensory interactions in humans. Nature Neurosci. 10, 1243-1245 (2007).
106. Stein, B. E. The development of a dialogue between cortex and midbrain to integrate multisensory information. Exp. Brain Res. 166, 305-315 (2005).
107. Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11-23 (2008).