Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons

Nature Reviews Neuroscience

Volumn 14, Issue 6, 2013, Pages 429-442

  Fetsch, Christopher R ^a   Deangelis, Gregory C ^b   Angelaki, Dora E ^c  

^a New York Presbyterian Morgan Stanley Children's Hospital   (United States)

^b UNIVERSITY OF ROCHESTER   (United States)

^c BAYLOR COLLEGE OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BRAIN FUNCTION; MATHEMATICAL MODEL; NERVE CELL; NEUROPHYSIOLOGY; PERCEPTION; PRIORITY JOURNAL; PSYCHOPHYSICS; PSYCHOPHYSIOLOGY; REVIEW; SENSORIMOTOR INTEGRATION; SENSORY NERVE CELL; THEORY; VESTIBULAR STIMULATION; VISION;

ANIMALS; BRAIN; CUES; HUMANS; MODELS, NEUROLOGICAL; PERCEPTION; PSYCHOLOGICAL THEORY; SENSORY RECEPTOR CELLS; VISION, OCULAR;

EID: 84878240611     PISSN: 1471003X     EISSN: 14710048     Source Type: Journal    
DOI: 10.1038/nrn3503     Document Type: Review

References (119)

1. Faisal, A. A., Selen, L. P. & Wolpert, D. M. Noise in the nervous system. Nature Rev. Neurosci. 9, 292-303 (2008).
2. Gepshtein, S. Two psychologies of perception and the prospect of their synthesis. Philos. Psychol. 23, 217-281 (2010).
3. Knill, D. C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712-719 (2004).
4. Einstein, A. Über das Relativitätsprinzip und die aus demselben gezogenen Folgerungen. Jahrbuch Radioaktivität Elektronik 4, 411-462 (1907).
5. Angelaki, D. E., Shaikh, A. G., Green, A. M. & Dickman, J. D. Neurons compute internal models of the physical laws of motion. Nature 430, 560-564 (2004).
6. Merfeld, D., Zupan, L. & Peterka, R. Humans use internal models to estimate gravity and linear acceleration. Nature 398, 615-618 (1999).
7. Landy, M. S., Banks, M. S. & Knill, D. C. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 5-29 (Oxford Univ. Press, 2011).
8. Angelaki, D. E., Gu, Y. & DeAngelis, G. C. Multisensory integration: psychophysics, neurophysiology, and computation. Curr. Opin. Neurobiol. 19, 452-458 (2009).
9. Kajikawa, Y., Falchier, A., Musacchia, G., Lakatos, P. & Schroeder, C. E. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).
10. Lakatos, P., Chen, C. M., O'Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279-292 (2007).
11. Kayser, C., Petkov, C. I., Remedios, R. & Logothetis, N. K. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).
12. Kayser, C. & Logothetis, N. K. Do early sensory cortices integrate cross-modal information? Brain Struct. Funct. 212, 121-132 (2007).
13. Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278-285 (2006).
14. Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11-23 (2008).
15. Geisler, W. S. Contributions of ideal observer theory to vision research. Vision Res. 51, 771-781 (2011).
16. Jacobs, R. A. Optimal integration of texture and motion cues to depth. Vision Res. 39, 3621-3629 (1999).
17. Landy, M. S., Maloney, L. T., Johnston, E. B. & Young, M. Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35, 389-412 (1995).
18. Maloney, L. T. & Landy, M. S. in Proceedings of the SPIE (ed. Pearlman, W. A.) 1154-1163 (SPIE, 1989).
19. Clark, J. J. & Yuille, A. L. Data Fusion For Sensory Information Processing Systems (Kluwer, 1990).
20. Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257-262 (2004).
21. Battaglia, P. W., Jacobs, R. A. & Aslin, R. N. Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391-1397 (2003).
22. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429-433 (2002).
23. Sober, S. J. & Sabes, P. N. Flexible strategies for sensory integration during motor planning. Nature Neurosci. 8, 490-497 (2005).
24. van Beers, R. J., Sittig, A. C. & Gon, J. J. Integration of proprioceptive and visual position-information: an experimentally supported model. J. Neurophysiol. 81, 1355-1364 (1999).
25. Körding, K. P. & Wolpert, D. M. Bayesian decision theory in sensorimotor control. Trends Cogn. Sci. 10, 319-326 (2006).
26. Gu, Y., Angelaki, D. E. & DeAngelis, G. C. Neural correlates of multisensory cue integration in macaque MSTd. Nature Neurosci. 11, 1201-1210 (2008).
27. de Winkel, K. N., Weesie, J., Werkhoven, P. J. & Groen, E. L. Integration of visual and inertial cues in perceived heading of self-motion. J. Vis. 10, 1 (2010).
28. Edwards, M., O'Mahony, S., Ibbotson, M. R. & Kohlhagen, S. Vestibular stimulation affects optic-flow sensitivity. Perception 39, 1303-1310 (2010).
29. Butler, J. S., Smith, S. T., Campos, J. L. & Bulthoff, H. H. Bayesian integration of visual and vestibular signals for heading. J. Vis. 10, 23 (2010).
30. Fetsch, C. R., Turner, A. H., DeAngelis, G. C. & Angelaki, D. E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601-15612 (2009).
31. Cochran, W. G. Problems arising in the analysis of a series of similar experiments. Appl. Stat. 4, 102-118 (1937).
32. Yuille, A. L. & Bülthoff, H. H. in Perception as Bayesian Inference (eds Knill, D. C. & Richards, W.) 123-162 (Cambridge Univ. Press, 1996).
33. Young, M. J., Landy, M. S. & Maloney, L. T. A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Res. 33, 2685-2696 (1993).
34. Knill, D. C. & Saunders, J. A. Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539-2558 (2003).
35. Hillis, J. M., Watt, S. J., Landy, M. S. & Banks, M. S. Slant from texture and disparity cues: optimal cue combination. J. Vis. 4, 967-992 (2004).
36. Clemens, I. A., De Vrijer, M., Selen, L. P., Van Gisbergen, J. A. & Medendorp, W. P. Multisensory processing in spatial orientation: an inverse probabilistic approach. J. Neurosci. 31, 5365-5377 (2011).
37. Ma, W., Beck, J. M. & Pouget, A. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 393-405 (Oxford Univ. Press, 2011).
38. Oruc, I., Maloney, L. T. & Landy, M. S. Weighted linear cue combination with possibly correlated error. Vision Res. 43, 2451-2468 (2003).
39. Rosas, P., Wagemans, J., Ernst, M. O. & Wichmann, F. A. Texture and haptic cues in slant discrimination: reliability-based cue weighting without statistically optimal cue combination. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 22, 801-809 (2005).
40. Rosas, P., Wichmann, F. A. & Wagemans, J. Texture and object motion in slant discrimination: failure of reliability-based weighting of cues may be evidence for strong fusion. J. Vis. 7, 3 (2007).
41. Körding, K. P. et al. Causal inference in multisensory perception. PLoS ONE 2, e943 (2007).
42. Knill, D. C. Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J. Vis. 7, 5 (2007).
43. Ernst, M. O. & Di Luca, M. in Sensory Cue Integration (eds Trommershäuser, J., Landy, M. S. & Kording, K. P.) 224-250 (Oxford Univ. Press, 2011).
44. Zaidel, A., Turner, A. H. & Angelaki, D. E. Multisensory calibration is independent of cue reliability. J. Neurosci. 31, 13949-13962 (2011).
45. Raposo, D., Sheppard, J. P., Schrater, P. R. & Churchland, A. K. Multisensory decision-making in rats and humans. J. Neurosci. 32, 3726-3735 (2012).
46. Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).
47. Meredith, M. & Stein, B. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640-662 (1986).
48. Sparks, D. L. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol. Rev. 66, 118-171 (1986).
49. Stein, B. E. Development of the superior colliculus. Annu. Rev. Neurosci. 7, 95-125 (1984).
50. Wurtz, R. H. & Albano, J. E. Visual-motor function of the primate superior colliculus. Annu. Rev. Neurosci. 3, 189-226 (1980).
51. Stein, B. E. & Stanford, T. R. Multisensory integration: current issues from the perspective of the single neuron. Nature Rev. Neurosci. 9, 255-266 (2008).
52. James, T. W. & Stevenson, R. A. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).
53. Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809-823 (2004).
54. Sarko, D. K. et al. in The Neural Bases of Multisensory Processes (eds Murray, M. M. & Wallace, M. T.) (CRC Press, 2012).
55. Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499-6508 (2005).
56. Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240-1255 (2002).
57. Stein, B. E., Huneycutt, W. S. & Meredith, M. A. Neurons and behavior: the same rules of multisensory integration apply. Brain Res. 448, 355-358 (1988).
58. Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1-10 (1996).
59. Frens, M. A., Van Opstal, A. J. & Van der Willigen, R. F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802-816 (1995).
60. Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797-1809 (1993).
61. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Tw o cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506-522 (2001).
62. Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429-2444 (1997).
63. Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578-3582 (2000).
64. Wallace, M. T. & Stein, B. E. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21, 8886-8894 (2001).
65. Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J. Neurophysiol. 90, 4022-4026 (2003).
66. Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193-3205 (2007).
67. Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575-2586 (2005).
68. Cuppini, C., Ursino, M., Magosso, E., Rowland, B. A. & Stein, B. E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).
69. Patton, P. E. & Anastasio, T. J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783-810 (2003).
70. Alvarado, J. C., Rowland, B. A., Stanford, T. R. & Stein, B. E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13-23 (2008).
71. Rowland, B. A., Stein, B. E. & Stanford, T. R. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 333-344 (Oxford Univ. Press, 2011).
72. Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966).
73. Kim, B. & Basso, M. A. Saccade target selection in the superior colliculus: a signal detection theory approach. J. Neurosci. 28, 2991-3007 (2008).
74. Kim, B. & Basso, M. A. A probabilistic strategy for understanding action selection. J. Neurosci. 30, 2340-2355 (2010).
75. Ma, W. J., Beck, J. M., Latham, P. E. & Pouget, A. Bayesian inference with probabilistic population codes. Nature Neurosci. 9, 1432-1438 (2006).
76. Beck, J. M. et al. Probabilistic population codes for Bayesian decision making. Neuron 60, 1142-1152 (2008).
77. Jazayeri, M. & Movshon, J. A. Optimal representation of sensory information by neural populations. Nature Neurosci. 9, 690-696 (2006).
78. Ma, W. J. & Pouget, A. Linking neurons to behavior in multisensory perception: a computational review. Brain Res. 1242, 4-12 (2008).
79. Populin, L. C. & Yin, T. C. Bimodal interactions in the superior colliculus of the behaving cat. J. Neurosci. 22, 2826-2834 (2002).
80. Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18, 787-792 (2007).
81. Perrault, T. J. Jr, Stein, B. E. & Rowland, B. A. Non-stationarity in multisensory neurons in the superior colliculus. Front. Psychol. 2, 144 (2011).
82. Gibson, J. J. The Perception of the Visual World (Houghton-Mifflin, 1950).
83. Warren, W. H. Jr, Morris, M. W. & Kalish, M. Perception of translational heading from optical flow. J. Exp. Psychol. Hum. Percept. Perform. 14, 646-660 (1988).
84. Benson, A. J., Spencer, M. B. & Stott, J. R. Thresholds for the detection of the direction of whole-body, linear movement in the horizontal plane. Aviat. Space Environ. Med. 57, 1088-1096 (1986).
85. Fernandez, C. & Goldberg, J. M. Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. II. Directional selectivity and force-response relations. J. Neurophysiol. 39, 985-995 (1976).
86. Fernandez, C. & Goldberg, J. M. Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. I. Response to static tilts and to long-duration centrifugal force. J. Neurophysiol. 39, 970-984 (1976).
87. Guedry, F. E. in Handbook of Sensory Physiology, The Vestibular System (ed. Kornhuber, H. H.) (Springer, 1974).
88. Chen, A., DeAngelis, G. C. & Angelaki, D. E. Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex. J. Neurosci. 31, 12036-12052 (2011).
89. Gu, Y., Watkins, P. V., Angelaki, D. E. & DeAngelis, G. C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73-85 (2006).
90. Chen, A., DeAngelis, G. C. & Angelaki, D. E. Convergence of vestibular and visual self-motion signals in an area of the posterior sylvian fissure. J. Neurosci. 31, 11617-11627 (2011).
91. Duffy, C. J. MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816-1827 (1998).
92. Britten, K. H. & van Wezel, R. J. Electrical microstimulation of cortical area MST biases heading perception in monkeys. Nature Neurosci. 1, 59-63 (1998).
93. Gu, Y., DeAngelis, G. C. & Angelaki, D. E. Causal links between dorsal medial superior temporal area neurons and multisensory heading perception. J. Neurosci. 32, 2299-2313 (2012).
94. Morgan, M. L., DeAngelis, G. C. & Angelaki, D. E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662-673 (2008).
95. Holmes, N. P. & Spence, C. Multisensory integration: space, time and superadditivity. Curr. Biol. 15, R762-R764 (2005).
96. Parker, A. J. & Newsome, W. T. Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227-277 (1998).
97. Bradley, A., Skottun, B. C., Ohzawa, I., Sclar, G. & Freeman, R. D. Visual orientation and spatial frequency discrimination: a comparison of single neurons and behavior. J. Neurophysiol. 57, 755-772 (1987).
98. Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966).
99. Britten, K. H., Shadlen, M. N., Newsome, W. T. & Movshon, J. A. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745-4765 (1992).
100. Fetsch, C. R., Pouget, A., DeAngelis, G. C. & Angelaki, D. E. Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neurosci. 15, 146-154 (2011).
101. Paradiso, M. A. A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biol. Cybern. 58, 35-49 (1988).
102. Seung, H. S. & Sompolinsky, H. Simple models for reading neuronal population codes. Proc. Natl Acad. Sci. USA 90, 10749-10753 (1993).
103. Carandini, M. & Heeger, D. J. Normalization as a canonical neural computation. Nature Rev. Neurosci. 13, 51-62 (2011).
104. Louie, K., Grattan, L. E. & Glimcher, P. W. Reward value-based gain control: divisive normalization in parietal cortex. J. Neurosci. 31, 10627-10639 (2011).
105. Ohshiro, T., Angelaki, D. E. & DeAngelis, G. C. A normalization model of multisensory integration. Nature Neurosci. 14, 775-782 (2011).
106. Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350-354 (1986).
107. Carandini, M., Heeger, D. & Movshon, J. Linearity and normalization in simple cells of the macaque primary visual cortex. J. Neurosci. 17, 8621-8644 (1997).
108. Carandini, M. From circuits to behavior: a bridge too far? Nature Neurosci. 15, 507-509 (2012).
109. Beauchamp, M. S., Pasalar, S. & Ro, T. Neural substrates of reliability-weighted visual-tactile multisensory integration. Front. Syst. Neurosci. 4, 25 (2010).
110. Helbig, H. B. et al. The neural mechanisms of reliability weighted integration of shape information from vision and touch. Neuroimage 60, 1063-1072 (2012).
111. Takahashi, K. et al. Multimodal coding of three-dimensional rotation and translation in area MSTd: comparison of visual and vestibular selectivity. J. Neurosci. 27, 9742-9756 (2007).
112. Gu, Y., DeAngelis, G. C. & Angelaki, D. E. A functional link between area MSTd and heading perception based on vestibular signals. Nature Neurosci. 10, 1038-1047 (2007).
113. Chen, A., DeAngelis, G. C. & Angelaki, D. E. A comparison of vestibular spatiotemporal tuning in macaque parietoinsular vestibular cortex, ventral intraparietal area, and medial superior temporal area. J. Neurosci. 31, 3082-3094 (2011).
114. Shams, L., Kamitani, Y. & Shimojo, S. Illusions. What you see is what you hear. Nature 408, 788 (2000).
115. Shimojo, S. & Shams, L. Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11, 505-509 (2001).
116. Sekuler, R., Sekuler, A. B. & Lau, R. Sound alters visual motion perception. Nature 385, 308 (1997).
117. Avillac, M., Denève, S., Olivier, E., Pouget, A. & Duhamel, J.-R. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neurosci. 8, 941-949 (2005).
118. Wallace, M. T. et al. Unifying multisensory signals across time and space. Exp. Brain Res. 158, 252-258 (2004).
119. Fetsch, C. R., DeAngelis, G. C. & Angelaki, D. E. Visual-vestibular cue integration for heading perception: applications of optimal cue integration theory. Eur. J. Neurosci. 31, 1721-1729 (2010).