Multisensory integration: psychophysics, neurophysiology, and computation

Current Opinion in Neurobiology

Volumn 19, Issue 4, 2009, Pages 452-458

  Angelaki, Dora E ^a   Gu, Yong ^a   DeAngelis, Gregory C ^b  

^a WASHINGTON UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b UNIVERSITY OF ROCHESTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASSOCIATION; BAYESIAN LEARNING; BEHAVIOR; HUMAN; MATHEMATICAL MODEL; MAXIMUM LIKELIHOOD METHOD; MOVEMENT PERCEPTION; NEUROPHYSIOLOGY; NONHUMAN; ORIENTING RESPONSE; PRIORITY JOURNAL; PSYCHOPHYSICS; REVIEW; SENSORIMOTOR INTEGRATION; STIMULUS RESPONSE; VESTIBULAR NERVE; VISUAL NERVOUS SYSTEM;

ANIMALS; BRAIN; MODELS, NEUROLOGICAL; NERVE NET; NEURAL NETWORKS (COMPUTER); NEURONS; SENSATION; SENSORY THRESHOLDS; SYNAPTIC TRANSMISSION;

EID: 69449087069     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2009.06.008     Document Type: Review

References (50)

1. Meredith M.A., and Stein B.E. Interactions among converging sensory inputs in the superior colliculus. Science 221 (1983) 389-391
2. Stein B.E., and Stanford T.R. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9 (2008) 255-266. This review summarizes properties of single neurons in the superior colliculus and cortical areas in response to bimodal stimuli. This is a good review of the traditional neurophysiological approaches to multisensory integration.
3. Perrault Jr. T.J., Vaughan J.W., Stein B.E., and Wallace M.T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90 (2003) 4022-4026
4. Stanford T.R., Quessy S., and Stein B.E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25 (2005) 6499-6508
5. Calvert G.A., Hansen P.C., Iversen S.D., and Brammer M.J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14 (2001) 427-438
6. Driver J., and Noesselt T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57 (2008) 11-23
7. Stanford T.R., and Stein B.E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18 (2007) 787-792
8. Laurienti P.J., Perrault T.J., Stanford T.R., Wallace M.T., and Stein B.E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166 (2005) 289-297
9. Beauchamp M.S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3 (2005) 93-113
10. Ernst M.O., and Banks M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415 (2002) 429-433
11. Battaglia P.W., Jacobs R.A., and Aslin R.N. Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis 20 (2003) 1391-1397
12. Alais D., and Burr D. The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14 (2004) 257-262
13. Mamassian P., Landy M.S., and Maloney L.T. Bayesian modelling of visual perception. In: Rao P.N., et al. (Ed). Probabilistic Models of the Brain (2002) 13-36
14. Kersten D., Mamassian P., and Yuille A. Object perception as Bayesian inference. Annu Rev Psychol 55 (2004) 271-304
15. Knill D.C., and Pouget A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci 27 (2004) 712-719
16. Ma W.J., Beck J.M., Latham P.E., and Pouget A. Bayesian inference with probabilistic population codes. Nat Neurosci 9 (2006) 1432-1438. This theoretical study proposes that the commonly observed Poisson-like variability in neuronal activity facilitates representation of probability distributions. For optimal cue integration, the product of sensory likelihoods can be simply accomplished by summing the neural population responses elicited by each cue. This theory predicts that multisensory neurons should simply sum their unimodal inputs, thus yielding a simple neural implementation of Bayesian inference.
17. Anastasio T.J., Patton P.E., and Belkacem-Boussaid K. Using Bayes' rule to model multisensory enhancement in the superior colliculus. Neural Comput 12 (2000) 1165-1187
18. Clark J.J., and Yuille A.L. Data Fusion for Sensory Information Processing Systems (1990), Kluwer Academic Publishers, Norwell, MA
19. Knill D.C., and Saunders J.A. Do humans optimally integrate stereo and texture information for judgments of surface slant?. Vision Res 43 (2003) 2539-2558
20. Hillis J.M., Watt S.J., Landy M.S., and Banks M.S. Slant from texture and disparity cues: optimal cue combination. J Vis 4 (2004) 967-992
21. van Beers R.J., Sittig A.C., and Gon J.J. Integration of proprioceptive and visual position-information: An experimentally supported model. J Neurophysiol 81 (1999) 1355-1364
22. Gu Y., Angelaki D.E., and Deangelis G.C. Neural correlates of multisensory cue integration in macaque MSTd. Nat Neurosci 11 (2008) 1201-1210. This is the first study to examine multisensory integration in neurons while trained animals performed a behavioral task that requires perceptual cue integration. This study shows that nonhuman primates, like humans, can combine cues near-optimally, and it identifies a population of neurons in area MSTd that may form the neural substrate for perceptual integration of visual and vestibular cues to heading.
23. Fetsch CR, Angelaki DE, DeAngelis GC: Dynamic cue re-weighting in rhesus monkeys performing a visual-vestibular heading discrimination task. Soc. Neurosci. Abstr. 715.26 2007.
24. Rowland B., Stanford T., and Stein B. A Bayesian model unifies multisensory spatial localization with the physiological properties of the superior colliculus. Exp Brain Res 180 (2007) 153-161
25. Bremmer F., Kubischik M., Pekel M., Lappe M., and Hoffmann K.P. Linear vestibular self-motion signals in monkey medial superior temporal area. Ann N Y Acad Sci 871 (1999) 272-281
26. Duffy C.J. MST neurons respond to optic flow and translational movement. J Neurophysiol 80 (1998) 1816-1827
27. Gu Y., Watkins P.V., Angelaki D.E., and DeAngelis G.C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J Neurosci 26 (2006) 73-85
28. Page W.K., and Duffy C.J. Heading representation in MST: sensory interactions and population encoding. J Neurophysiol 89 (2003) 1994-2013
29. Takahashi K., Gu Y., May P.J., Newlands S.D., DeAngelis G.C., and Angelaki D.E. Multimodal coding of three-dimensional rotation and translation in area MSTd: comparison of visual and vestibular selectivity. J Neurosci 27 (2007) 9742-9756
30. Schlack A., Hoffmann K.P., and Bremmer F. Interaction of linear vestibular and visual stimulation in the macaque ventral intraparietal area (VIP). Eur J Neurosci 16 (2002) 1877-1886
31. Green D.M., and Swets J.A. Signal Detection Theory and Psychophysics (1966), Wiley, New York
32. Britten K.H., Newsome W.T., Shadlen M.N., Celebrini S., and Movshon J.A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis Neurosci 13 (1996) 87-100
33. Krug K. A common neuronal code for perceptual processes in visual cortex? Comparing choice and attentional correlates in V5/MT. Philos Trans R Soc Lond B Biol Sci 359 (2004) 929-941
34. Parker A.J., and Newsome W.T. Sense and the single neuron: probing the physiology of perception. Annu Rev Neurosci 21 (1998) 227-277
35. Morgan M.L., Deangelis G.C., and Angelaki D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59 (2008) 662-673. This is the first single-unit study to directly measure the mathematical rule by which multisensory neurons combine their inputs. It shows that MSTd neurons perform a linear weighted summation of visual and vestibular inputs related to heading. This study also shows that the weights that single neurons apply to their inputs can change with cue reliability. This might contribute to perceptual reweighting of cues according to reliability, and these findings provide important constraints for physiologically plausible models of multisensory integration.
36. Heuer H.W., and Britten K.H. Linear responses to stochastic motion signals in area MST. J Neurophysiol 98 (2007) 1115-1124
37. Heeger D.J. Normalization of cell responses in cat striate cortex. Vis Neurosci 9 (1992) 181-197
38. Priebe N.J., Mechler F., Carandini M., and Ferster D. The contribution of spike threshold to the dichotomy of cortical simple and complex cells. Nat Neurosci 7 (2004) 1113-1122
39. Holmes N.P., and Spence C. Multisensory integration: space, time and superadditivity. Curr Biol 15 (2005) R762-764
40. Frens M.A., and Van Opstal A.J. Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Res Bull 46 (1998) 211-224
41. Populin L.C., and Yin T.C. Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci 22 (2002) 2826-2834
42. Bizley J.K., Nodal F.R., Bajo V.M., Nelken I., and King A.J. Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb Cortex 17 (2007) 2172-2189
43. Avillac M., Ben Hamed S., and Duhamel J.R. Multisensory integration in the ventral intraparietal area of the macaque monkey. J Neurosci 27 (2007) 1922-1932
44. Ghazanfar A.A., Maier J.X., Hoffman K.L., and Logothetis N.K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25 (2005) 5004-5012
45. Kayser C., Petkov C.I., and Logothetis N.K. Visual modulation of neurons in auditory cortex. Cereb Cortex 18 (2008) 1560-1574
46. Lakatos P., Chen C.M., O'Connell M.N., Mills A., and Schroeder C.E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53 (2007) 279-292
47. Sugihara T., Diltz M.D., Averbeck B.B., and Romanski L.M. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J Neurosci 26 (2006) 11138-11147
48. Rowland B.A., Quessy S., Stanford T.R., and Stein B.E. Multisensory integration shortens physiological response latencies. J Neurosci 27 (2007) 5879-5884. Superadditivity is commonly observed in responses of superior colliculus (SC) neurons to combinations of visual and auditory cues. This study examines the time course of these responses and shows that superadditivity is prevalent in the early portion of neural responses. This leads to reduced multisensory response latencies, which may account for reduced reaction times in some behavioral tasks.
49. Bernstein I.H., Clark M.H., and Edelstein B.A. Effects of an auditory signal on visual reaction time. J Exp Psychol 80 (1969) 567-569
50. Diederich A., and Colonius H. Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66 (2004) 1388-1404