Brain Mechanisms for Synthesizing Information from Different Sensory Modalities

Blackwell Handbook of Sensation and Perception

Volumn , Issue , 2008, Pages 709-736

  Stein, Barry E ^a   Wallace, Mark T ^a   Stanford, Terrence R ^a  

^a WAKE FOREST SCHOOL OF MEDICINE   (United States)

Author keywords

Brain mechanisms; Cinematography; Information; Sensory modalities; Sensory systems

Indexed keywords

EID: 84889267654     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9780470753477.ch22     Document Type: Chapter

References (127)

1. Ackerman, D. (1990). A. natural history of the senses. New York: Vintage Books/Random House.
2. Andreassi, J. L., & Greco, J. R. (1975). Effects of bisensory stimulation on reaction time and the evoked cortical potential. Physiology and Psychology, 3,189-194.
3. Asanuma, C., Ohkawa, R., Stanfield, B. B., & Cowan, W. M. (1988). Observations on the development of certain ascending inputs to the thalamus in rats, I. Postnatal development. Developments in Brain Research, 41, 159-170.
4. Benedetti, F. (1991). The postnatal emergence of a functional somatosensory representation in the superior colliculus of the mouse, Developments in Brain Research, 60, 51-57.
5. Bernstein, I. H., Clark, M. H., & Edelstein, B. A. (1969). Effects of an auditory signal on visual reaction time, Journal of Experimental Psychology, 80, 567-569.
6. Biguer, B., Donaldson, I. M. L., Hein, A., & Jeannerod, M. (1988). Neck muscle vibration modifies the representation of visual motion and direction in man, Brain, 111, 1405-1424.
7. Bower, T. G. R. (1977). A. primer of infant development. San Francisco: WH Freeman.
8. Bruce, C., Desimone, R., & Gross, C. G. (1981). Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque, Journal of Neurophysiology, 46, 369-384.
9. Bullock, T. H. (1984). Physiology of the tectum mesencephali in elasmobranchs. In H. Vanegas (Ed.), Comparative neurology of the optic tectum (pp 47-68). New York: Plenum Press.
10. Casagrande, V. A., Harting. J. K., Hall, W. C., & Diamond, I. T. (1972). Superior colliculus of the tree shrew: A structural and functional subdivision into superficial and deep layers, Science, 177, 444-447.
11. Chalupa, L. M., & Rhoades, R. W. (1977). Responses of visual, somatosensory, and auditory neurones in the golden hamster's superior colliculus, Journal of Physiology (London), 207, 595-626.
12. Clarey, J. C., & Irvine, D. R. F. (1986). Auditory response properties of neurons in the anterior ectosylvian sulcus of the cat, Brain Research, 386, 12-19.
13. Clark, B., & Graybiel, A. (1966). Contributing factors in the perception of the oculogravic illusion, American Journal of Psychology, 79, 377-388.
14. Clemo, H. R., Meredith, M. A., Wallace, M. T., & Stein, B. E. (1991). Is the cortex of cat anterior ectosylvian sulcus a polysensory area? Society of Neuroscience Abstracts, 17, 1585.
15. Clemo, H. R., & Stein, B. E. (1982). Somatosensory cortex: A "new" somatotopic representation, Brain Research, 235, 162-168.
16. Cytowic, R. E. (1989). Synesthesia:. A union of the senses. New York: Springer-Verlag.
17. Drager, U. C., & Hubel, D. H. (1975). Responses to visual stimulation and relationship between visual, auditory and somatosensory inputs in mouse superior colliculus, Journal of Neurophysiology, 38, 690-713.
18. Duhamel, J.-R., Colby, C. L., & Goldberg, M. E. (1991). Congruent representations of visual and somatosensory space in single neurons of monkey ventral intraparietal cortex (Area VIP). In J. Paillard (Ed.), Brain and space (pp. 223-236). New York: Oxford University Press.
19. Edwards, S. B., Ginsburgh, C. L., Henkel, C. K., & Stein, B. E. (1979). Sources of subcortical projections to the superior colliculus in the cat, Journal of Comparative Neurology, 184, 309-330.
20. Finlay, B. L., Schneps, S. E., Wilson, K. G., & Schneider, G. E. (1978). Topography of visual and somatosensory projections to the superior colliculus of the golden hamster, Brain Research, 142, 223-235.
21. Fogassi, L., Gallese, V., Fadiga, L., Luppino, G., Matelli, M., & Rizzolatti, G. (1996). Coding of peripersonal space in inferior premotor cortex (area F4), Journal of Neurophysiology, 76, 141-157.
22. Frens, M. A., & Van Opstal, A. J. (1998). Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus, Brain Research Bulletin, 46, 211-224.
23. Frens, M. A., Van Opstal, A. J., & Van der Willigen, R. F. (1995). Spatial and temporal factors determine audio-visual interactions in human saccadic eye movements, Perception & Psychophysics, 57, 802-816.
24. Frost, D. O. (1984). Axonal growth and target selection during development: Retinal projections to the ventrobasal complex and other "nonvisual" structures in neonatal Syrian hamsters, Journal of Comparative Neurology, 230, 576-592.
25. Gibson, J. J. (1966). The. senses considered as perceptual systems. Boston: Houghton Mifflin.
26. Gielen, S. C. A. M., Schmidt, R. A., & van den Heuvel, P. J. M. (1983). On the nature of intersensory facilitation of reaction time, Perception & Psychophysics, 34, 161-168.
27. Goldring, J. E., Dorris, M. C., Corneil, B. D., Ballantyne, P. A., & Munoz, D. P. (1996). Combined eye-head gaze shifts to visual and auditory targets in humans, Experimental Brain Research, 111, 68-78.
28. Gottlieb, G., Tomlinson, W. R., & Radell, P. L. (1989). Developmental intersensory interference: Premature visual experience suppresses auditory learning in ducklings, Infant Behavior and Development, 12, 1-12.
29. Graybiel, A., & Niven, J. I. (1951). The effect of a change in direction of resultant force on sound localization: The audiogravic illusion, Journal of Experimental Psychology, 42, 227-230.
30. Graziano, M. S., Hu, X. T., & Gross, C. G. (1997). Visuospatial properties of ventral premotor cortex, Journal of Neurophysiology, 77, 2268-2292.
31. Graziano, M. S., Yap, G. S., & Gross, C. G. (1994). Coding of visual space by premotor neurons, Science, 266, 1054-1057.
32. Groh, J. M., & Sparks, D. L. (1996). Saccades to somatosensory targets, III. Eye-position-dependent somatosensory activity in primate superior colliculus. Journal of Neurophysiology, 75, 439-453.
33. Hartline, P. H., Kass, L., & Loop, M. S. (1978). Merging of modalities in the optic tectum: Infrared and visual integration in rattlesnakes, Science, 199, 1225-1229.
34. Hartline, P. H., Pandey Vimal, R. L., King, A. J., Kurylo, D. D., & Northmore, D. P. M. (1995). Effects of eye position on auditory localization and neural representation of space in superior colliculus of cats, Experimental Brain Research, 104, 402-408.
35. Held, R. (1955). Shifts in binaural localization after prolonged exposures to atypical combinations of stimuli, American Journal of Psychology, 68, 526-548.
36. Helmholtz, H. von (1968). The origin of the correct interpretation of our sensory impressions. In R. M. Warren & R. P. Warren (Eds.), Helmholtz on perception: Its physiology and development (pp 247-266). New York: Wiley.
37. Hershenson, M. (1962). Reaction time as a measure of intersensory facilitation, Journal of Experimental Psychology, 63, 289-293.
38. Highstein, S. M., & Baker, R. (1985). Action of the efferent vestibular system on primary afferents in the toadfish, Opsanus tau, Journal of Neurophysiology, 54, 370-384.
39. Hillyard, S. A., Mangun, G. R., Woldorff, M. G., & Luck, S. J. (1995). Neural mechanisms mediating selective attention. In M. Gazzaniga (Ed.), The cognitive neurosciences (pp. 665-681). Cambridge, MA: MIT Press.
40. Howard, I. P., & Templeton, W. B. (1966). Human. spatial orientation. London: Wiley.
41. Huerta, M. F., & Harting, J. K. (1984). The mammalian superior colliculus: Studies of its morphology and connections. In H. Vanegas (Ed.), Comparative neurology of the optic tectum (pp. 687-773). New York: Plenum Press.
42. Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G., & Fendrich, R. (1994). Visual-auditory interactions in sensorimotor processing: Saccades versus manual responses, Journal of Experimental Psychology: Human Perception & Performance, 20, 131-153.
43. Innocenti, G. M., & Clarke, S. (1984). Bilateral transitory projection to visual areas from auditory cortex in kittens, Developments in Brain Research, 14, 143-148.
44. Jay, M. F., & Sparks, D. L. (1987a). Sensorimotor integration in the primate superior colliculus, I. Motor convergence. Journal of Neurophysiology, 57, 22-34.
45. Jay, M. F., & Sparks, D. L. (1987b). Sensorimotor integration in the primate superior colliculus, II. Coordinates of auditory signals. Journal of Neurophysiology, 57, 35-55.
46. Jiang, H., Lepore, F., Ptito, M. & Guillemot, J. P. (1994a). Sensory modality distribution in the anterior ectosylvian cortex (AEC) of cats, Experimental Brain Research, 97, 404-414.
47. Jiang, H., Lepore, F., Ptito, M. & Guillemot, J. P. (1994b). Sensory interactions in the anterior ectosylvian cortex of cats, Experimental Brain Research, 101, 385-396.
48. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within-modality and cross-modality suppression in the superior colliculus, Journal of Neurophysiology, 78, 2834-2847.
49. King, A. J., & Carlile, S. (1993). Changes induced in the representation of auditory space in the superior colliculus by rearing ferrets with binocular lid suture, Experimental Brain Research, 94, 444-455.
50. King, A. J., & Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus, Experimental Brain Research, 60, 492-500.
51. Knudsen, E. I. (1982). Auditory and visual maps of space in the optic tectum of the owl, Journal of Neuroscience, 2, 1177-1194.
52. Knudsen, E. I., & Brainard, M. S. (1991). Visual instruction of the neural map of auditory space in the developing optic tectum, Science, 253, 85-87.
53. Lackner, J. R. (1974a). Changes in auditory localization during body tilt, Acta Oto-Laryngologica (Stockh), 77, 19-28.
54. Lackner, J. R. (1974b). Influence of visual rearrangement and visual motion on sound localization, Neuropsychologia, 12, 291-293.
55. Larson, M., & Stein, B. E. (1984). The use of tactile and olfactory cues in neonatal orientation and localization of the nipple, Developmental Psychobiology, 17, 423-436.
56. Lee, C., Chung, S., Kim, J., & Park, J. (1991). Auditory facilitation of visually guided saccades, Society of Neuroscience Abstracts, 17, 862.
57. Lickliter, R. (1990). Premature visual stimulation accelerates intersensory functioning in bobwhite quail neonates, Developmental Psychobiology, 23, 15-27.
58. Marks, L. E. (1975). On colored-hearing synesthesia: Cross-modal translations of sensory dimensions, Psychological Bulletin, 82, 303-331.
59. Marks, L. E. (1978). The. unity of the senses: Interrelations among the modalities. New York: Academic Press.
60. Massaro, D. W., & Stork, D. G. (1998). Speech recognition and sensory integration, American Science, 86, 236-244.
61. Maurer, D., & Maurer, C. (1988). The. world of the newborn. New York: Basic Books.
62. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices, Nature, 264, 746-748.
63. McHaffie, J. G., Kao, C.-Q., & Stein, B. E. (1989). Nociceptive neurons in rat superior colliculus: Response properties, topography and functional implications, Journal of Neurophysiology, 62, 510-525.
64. Meltzoff, A. N., & Moore, M. K. (1977). Imitation of facial and manual gestures by human neonates, Science, 198, 75-78.
65. Meltzoff, A. N., & Moore, M. K. (1983a). The origins of imitation in infancy: Paradigm, phenomena, and theories. In L. P. Lipsitt (Ed.), Advances in infancy research (Vol. 2, pp. 265-301). Norwood, NJ: Ablex.
66. Meltzoff, A. N., & Moore, M. K. (1983b). Newborn infants imitate adult facial gestures, Child Development, 54, 702-709.
67. Meredith, M. A., Nemitz, J. W., & Stein, B. E. (1987). Determinants of multisensory integration in superior colliculus neurons, I. Temporal factors. Journal of Neuroscience, 10, 3215-3229.
68. Meredith, M. A., & Stein, B. E. (1986). Visual, auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration, Journal of Neurophysiology, 56, 640-662.
69. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus, Journal of Neurophysiology, 75, 1843-1857.
70. Meredith, M. A., Wallace, M. T., & Stein, B. E. (1992). Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: Multisensory properties of the tectoreticulo-spinal projection, Experimental Brain Research, 88, 181-186.
71. Morrell, L. K. (1968a). Cross-modality effects upon choice reaction time, Psychonomic Science, 11, 129-130.
72. Morrell, L. K. (1968b). Temporal characteristics of sensory interaction in choice reaction times, Journal of Experimental Psychology, 77, 14-18.
73. Morrell, L. K. (1972). Visual system's view of acoustic space, Nature, 238, 44-46.
74. Mucke, L., Norita, M., Benedek, G., & Creutzfeldt, O. (1982). Physiologic and anatomic investigation of a visual cortical area situated in the ventral bank of the anterior ectosylvian sulcus of the cat, Experimental Brain Research, 46, 1-11.
75. Neville, H. J. (1990). Intermodal competition and compensation in development, Evidence from studies of the visual system in congenitally deaf adults. Annals of the New York Academy of Science, 608, 71-87.
76. Olson, C. R., & Graybiel, A. M. (1987). Ectosylvian visual area of the cat: Location, retinotopic organization, and connections, Journal of Comparative Neurology, 261, 277-294.
77. Paulesu, E., Harrison, J., Baron-Cohen, S., Watson, J. D., Goldstein, L., Heather, J., Frackowiak, R. S., & Frith, C. D. (1995). The physiology of coloured hearing, A PET activation study of colour-word synaesthesia. Brain, 118, 661-676.
78. Peck, C. K., Baro, J. A., & Warder, S. M. (1995). Effects of eye position on saccadic eye movements and on the neural responses to auditory and visual stimuli in cat superior colliculus, Experimental Brain Research, 103, 227-242.
79. Perrott, D. R., Saberi, K., Brown, K., & Strybel, T. Z (1990). Auditory psychomotor coordination and visual search performance. Perception & Psychophysics, 48, 214-226.
80. Piaget, J. (1952). The. origins of intelligence in children. New York: International Universities Press.
81. Pick, H. L., Jr., Warren, D. H., & Hay, J. C. (1969). Sensory conflict in judgements of spatial direction, Perception & Psychophysics, 6, 203-205.
82. Posner, M. I., Nissen, M. J., & Klein, R. M. (1976). Visual dominance: An information-processing account of its origins and significance, Psychological Reviews, 83, 157-171.
83. Ramachandran, R., Wallace, M. T., Clemo, H. R., & Stein, B. E. (1993). Multisensory convergence and integration in rat cortex, Society of Neuroscience Abstracts, 19, 1447.
84. Rauschecker, J. P. (1995). Compensatory plasticity and sensory substitution in the cerebral cortex, Trends in Neuroscience, 18, 36-43.
85. Rizzolatti, G., Scandolara, C., Matelli, M., & Gentilucci, M. (1981a). Afferent properties of periarcuate neurons in macaque monkeys, I. Somatosensory responses. Behavioral Brain Research, 2, 125-146.
86. Rizzolatti, G., Scandolara, C., Matelli, M., & Gentilucci, M. (1981b). Afferent properties of periarcuate neurons in macaque monkeys, II. Visual responses. Behavioral Brain Research, 2, 147-163.
87. Roll, R., Velay, J. L., Roll, J. P. (1991). Eye and neck proprioceptive messages contribute to the spatial coding of retinal input in visually oriented activities, Experimental Brain Research, 85, 423-431.
88. Ryan, T. A. (1940). Interrelations of the sensory systems in perception, Psychological Bulletin, 37, 659-698.
89. Schneider, G. E. (1969). Two visual systems: Brain mechanisms for localization and discrimination are dissociated by tectal and cortical lesions, Science, 163, 895-902.
90. Schnupp, J. W. H., King, A. J., Smith, A. L., & Thompson, I. D. (1995). NMDA-receptor antagonists disrupt the formation of the auditory space map in the mammalian superior colliculus, Journal of Neuroscience, 15, 1516-1531.
91. Shelton, B. R., & Searle, C. L. (1980). The influence of vision on the absolute identification of sound-source position, Perception & Psychophyics, 28, 589-596.
92. Sparks, D. L., & Groh, J. M. (1994). The superior colliculus: A window for viewing issues in integrative neuroscience. In M. Gazzaniga (Ed.), The cognitive neurosciences (pp. 565-584). Cambridge, MA: MIT Press..
93. Sparks, D. L., & Nelson, J. S. (1987). Sensory and motor maps in the mammalian superior colliculus, Trends in Neuroscience, 10, 312-317.
94. Sprague, J. M., & Meikle, T. H., Jr. (1965). The role of the superior colliculus in visually guided behavior, Experimental Neurology, 11, 115-146.
95. Stein, B. E., & Dixon, J. P. (1979). Properties of superior colliculus neurons in the golden hamster, Journal of Comparative Neurology, 183, 269-284.
96. Stein, B. E., & Gaither, N. (1981). Sensory representation in reptilian optic tectum: Some comparisons with mammals, Journal of Comparative Neurology, 202, 69-87.
97. Stein, B. E., & Gallagher, H. (1981). Maturation of cortical control over superior colliculus cells in cat, Brain Research, 223, 429-435.
98. Stein, B. E., Labos, E., & Kruger, L. (1973). Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation, Journal of Neurophysiology, 36, 667-679.
99. Stein, B. E., London, N., Wilkinson, L. K., & Price, D. D. (1996). Enhancement of perceived visual intensity by auditory stimuli: A psychophysical analysis, Journal of Cognitive Neuroscience, 8, 497-506.
100. Stein, B. E., & Meredith, M. A. (1991). Functional organization of the superior colliculus. In A. G. Leventhal (Ed.), The neural basis of visual function (pp. 85-110). Hampshire, UK: Macmillan.
101. Stein, B. E., & Meredith, M. A. (1993). The. merging of the senses. Cambridge, MA: MIT Press.
102. Stein, B. E., Meredith, M. A., Huneycutt, W. S., McDade, L. (1989). Behavioral indices of multisensory integration: Orientation to visual cues is affected by auditory stimuli, Journal of Cognitive Neuroscience, 1, 12-24.
103. Stein, B. E., Meredith, M. A., & Wallace, M. T. (1993). The visually responsive neuron and beyond: Multisensory integration in cat and monkey, Progress in Brain Research, 95, 79-90.
104. Stein, B. E., Spencer, R. F., & Edwards, S. B. (1983). Corticotectal and corticothalamic efferent projections of SIV somatosensory cortex in cat, Journal of Neurophysiology, 50, 896-909.
105. Stein, B. E., & Wallace, M. T. (1996). Comparisons of cross-modality integration in midbrain and cortex, Progress in Brain Research, 112, 289-299.
106. Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise, Journal of the Acoustical Society of America, 26, 212-215.
107. Sur, M., Garraghty, P. E., & Roe, A. W. (1988). Experimentally induced visual projections into auditory thalamus and cortex, Science, 242, 1437-1441.
108. Thurlow, W. R., & Rosenthal, T. M. (1976). Further study of existence regions for the "ventriloquism effect." Journal of the American Audiological Society, 1, 280-286.
109. Tiao, Y.-C., & Blakemore, C. (1976). Functional organization in the superior colliculus of the golden hamster, Journal of Comparative Neurology, 168, 483-504.
110. Turkewitz, G., & Mellon, R. C. (1989). Dynamic organization of intersensory function, Canadian Journal of Psychology, 43, 286-307.
111. von Hornbostel, E. M. (1938). The unity of the senses. In W. D. Ellis (Ed.), A sourcebook of Gestalt psychology (pp 211-216). New York: Harcourt Brace.
112. Wade, N. J., & Day, R. H. (1968). Apparent head position as a basis for a visual aftereffect of prolonged head tilt, Perception & Psychophysics, 3, 324-326.
113. Wallace, M. T., McHaffie, J. G., & Stein, B. E. (1995). Sensory response properties in the superior colliculus (SC) of the newborn rhesus monkey, Society of Neuroscience Abstracts, 21, 655.
114. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1992). Integration of multiple sensory modalities in cat cortex, Experimental Brain Research, 91, 484-488.
115. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1993). Converging influences from visual, auditory and somatosensory cortices onto output neurons of the superior colliculus, Journal of Neurophysiology, 69, 1797-1809.
116. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1998). Multisensory integration in the superior colliculus of the alert cat, Journal of Neurophysiology, 80, 1006-1010.
117. Wallace, M. T., & Stein, B. E. (1994). Cross-modal synthesis in the midbrain depends on input from association cortex, Journal of Neurophysiology, 71, 429-432.
118. Wallace, M. T., & Stein, B. E. (1996). Sensory organization of the superior colliculus in cat and monkey, Progress in Brain Research, 112, 301-311.
119. Wallace, M. T., & Stein, B. E. (1997). Development of multisensory neurons and multisensory integration in cat superior colliculus, Journal of Neuroscience, 17, 2429-2444.
120. Wallace, M. T., Wilkinson, L. K., & Stein, B. E. (1996). Representation and integration of multiple sensory inputs in primate superior colliculus, Journal of Neurophysiology, 76, 1246-1266.
121. Warren, D. H., Welch, R. B., & McCarth, T. J. (1981). The role of visual-auditory "compellingness" in the ventriloquism effect: Implications for transitivity among the spatial senses, Perception & Psychophyics, 30, 557-564.
122. Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy, Psychological Bulletin, 88, 638-667.
123. Welch, R. B., & Warren, D. H. (1986). Intersensory interactions. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, Volume I: Sensory processes and perception (pp.1-36). New York: Wiley.
124. Wilkinson, L. K., Meredith, M. A., & Stein, B. E. (1996). The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior, Experimental Brain Research, 112, 1-10.
125. Withington-Wray, D. J., Binns, K. E., & Keating, M. J. (1990). The maturation of the superior collicular map of auditory space in the guinea pig is disrupted by developmental visual deprivation, European Journal of Neuroscience, 2, 682-692.
126. Withington-Wray, D. J., Binns, K. E., Dhanjal, S. S., Brickley, S. G., & Keating, M. J. (1990). The maturation of the collicular map of auditory space in the guinea pig is disrupted by developmental auditory deprivation, European Journal of Neuroscience, 2, 693-703.
127. Zahn, J. R., Abel, L. A., & Dell'Osso, L. F. (1978). Audio-ocular response characteristics, Sensory Process, 2, 32-37.