Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study

Journal of Cognitive Neuroscience

Volumn 11, Issue 5, 1999, Pages 473-490

  Giard, M H ^a   Peronnet, F ^a  

^a INSERM   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; AUDITORY STIMULATION; BEHAVIOR; ELECTROPHYSIOLOGY; EVENT RELATED POTENTIAL; HUMAN; HUMAN EXPERIMENT; NORMAL HUMAN; PRIORITY JOURNAL; RECOGNITION; VISUAL MEMORY; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ADAPTATION, PHYSIOLOGICAL; ADULT; AUDITORY CORTEX; AUDITORY PERCEPTION; BRAIN MAPPING; COGNITION; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; EVOKED POTENTIALS, AUDITORY; EVOKED POTENTIALS, VISUAL; FEMALE; FRONTAL LOBE; HUMANS; MALE; PATTERN RECOGNITION, VISUAL; PHOTIC STIMULATION; REACTION TIME; VISUAL CORTEX; VISUAL PERCEPTION;

EID: 0032709727     PISSN: 0898929X     EISSN: None     Source Type: Journal    
DOI: 10.1162/089892999563544     Document Type: Article

References (86)

1. Andreassi, J. L., & Greco, J. R. (1975). Effects of bisensory stimulation on reaction time and the evoked cortical potential. Physiological Psychology, 3, 189-194.
2. Anllo-Vento, L., & Hillyard, S. A. (1996). Selective attention to the color and direction of moving stimuli: Electrophysiological correlates of hierarchical feature selection. Perception and Psychophysics, 58, 191-206.
3. Barth, D. S., Goldberg, N., Brett, B., & Di, S. (1995). The spatiotemporal organization of auditory, visual and auditory-visual evoked potentials in rat cortex. Brain Research, 678, 177-190.
4. Benevento, L. A., Fallon, J., Davis, B. J., & Rezak, M. (1977). Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology, 57, 849-872.
5. Bernstein, I. H. (1970). Can we see and hear at the same time? Acta Physiologica Academiae Scientarum Hungaricae, 33, 21-35.
6. Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Perception and Psychophysics, 29, 578-584.
7. Busch, C., Wilson, G., Orr, C., & Papanicolaou, A. C. (1989). Crossmodal interactions of auditory stimulus presentation on the visual evoked magnetic reponse. In S. J. Williamson (Ed.), Advances in biomagnetism (pp. 221-224). New York: Plenum Press
8. Clark, V. P., Fan, S., & Hillyard, S. A. (1995). Identification of early visual evoked potential generators by retinotopic and topographic analyses. Human Brain Mapping, 2, 170-187.
9. Clark, V. P., & Hillyard, S. A. (1996). Spatial selective attention affects early extrastriate but not striate components of the visual evoked potential. Journal of Cognitive Neuroscience, 8, 387-402.
10. Costin, D., Neville, H. J., Meredith, M. A., & Stein, B. E. (1991). Rules of multisensory integration and attention: ERP and behavioral evidence in humans. Society for Neuroscience Abstracts, 17, 656.
11. Davis, H., Osterhammel, P. A., Wier, C. C., & Gjerdingen, D. B. (1972). Slow vertex potentials: Interactions among auditory, tactile, electric and visual stimuli. Electroencephalography and Clinical Neurophysiology, 33, 537-545.
12. Duhamel, J. R., Colby, C. L., & Goldberg, M. E. (1991). Congruent representations of visual and somatosensory space in single neurons of monkey ventral intraparietal cortex (Area VIP). In J. Paillard (Ed.), Brain and space (pp. 223-236). New York: Oxford University Press.
13. Echallier, J. F., Perrin, F., & Pernier, J. (1992). Computer-assisted placement of electrodes on the human head. Electroencephalography and Clinical Neurophysiology, 82, 160-163.
14. Ehrenstein, W. H., & Reinhardt-Rutland, A. H. (1996). A cross-modal aftereffect: Auditory displacement following adaptation to visual motion. Perceptual and Motor Skills, 82, 23-26.
15. Fisher, B. D., & Pylyshyn, Z. W. (1994). The cognitive architecture of bimodal event perception-A commentary and addendum to Radeau (1994). Cahiers de Psychologie Cognitive, 13, 92-96.
16. Fishman, M. C., & Michael, C. R. (1973). Integration of auditory information in the cat visual cortex. Vision Research, 13, 1415-1419.
17. Fournier, L. R., & Eriksen, C. W. (1990). Coactivation in the perception of redundant targets. Journal of Experimental Psychology: Human Perception and Performance, 16, 538-550.
18. Frens, M. A., Vanopstal, A. J., & Vanderwilligen, R. F. (1995). Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Perception and Psychophysics, 57, 802-816.
19. Gaffan, D., & Harrison, S. (1991). Auditory-visual associations, hemispheric specialization and temporo-frontal interaction in the rhesus monkey. Brain, 114, 2133-2144.
20. Gordon, B. G. (1973). Receptive fields in the deep layers of the cat superior colliculus. Journal of Neurophysiology, 36, 157-178.
21. Harter, M. R., & Aine, C. J. (1984). Brain mechanisms of visual selective attention. In R. Parasuraman & R. Davies (Eds.), Varieties of attention (pp. 293-321). London: Academic Press.
22. Heinze, H. J., Mangun, G. R., Burchert, W., Hinrichs, H., Scholz, M., Munte, T. F., Gos, A., Scherg, M., Johannes, S., Hundeshagen, H., Gazzaniga, M. S., & Hillyard, S. A. (1994). Combined spatial and temporal imaging of brain activity during visual selective attention in humans. Nature, 372, 543-546.
23. Held, R. (1955). Shifts in binaural localization after prolonged exposures to atypical combinations of stimuli. American Journal of Psychology, 68, 526-548.
24. Hershenson, M. (1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63, 289-293.
25. Hikosaka, K., Iwai, E., Saito, H., & Tanaka, K. (1988). Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of Neurophysiology, 60, 1615-1637.
26. Hubbard, T. L. (1996). Synesthesia-like mappings of lightness, pitch, and melodic interval. American Journal of Psychology, 109, 219-238.
27. Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G., & Fendrich, R. (1994). Visual-auditory interactions in sensorimotor processing-Saccades versus manual responses. Journal of Experimental Psychology: Human Perception and Performance, 20, 131-153.
28. Ito, S. I. (1982). Prefrontal unit activity of macaque monkeys during auditory and visual reaction time tasks. Brain Research, 247, 39-47.
29. Jay, M. F., & Sparks, D. L. (1984). Auditory receptive fields in primate superior colliculus shift with changes in eye position. Nature, 309, 345-347.
30. Jones, E. G., & Powell, T. P. S. (1970). An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain, 93, 793-820.
31. Kimura, A., & Tamai, Y. (1992). Sensory response of cortical neurons in the anterior ectosylvian sulcus, including the area evoking eye movement. Brain Research, 575, 181-186.
32. King, A. J., & Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurons in the guinea-pig superior colliculus. Experimental Brain Research, 60, 492-500.
33. Logothetis, N. K., & Sheinberg, D. L. (1996). Visual object recognition. Annual Review of Neuroscience, 19, 577-621.
34. Loveless, N., Levänen, S., Jousmäki, V., Sams, M., & Hari, R. (1996). Temporal integration in auditory sensory memory: Neuromagnetic evidence. Evoked Potential, 100, 220-228.
35. Luck, S. J., Heinze, H. J., Mangun, G. R., & Hillyard, S. A. (1990). Visual event-related potentials index focused attention within bilateral stimulus array. II. Functional dissociation of P1 and N1 components. Electroencephalography and Clinical Neurophysiology, 75, 528-542.
36. Mangun, G. R. (1995). Neural mechanisms of visual selective attention. Psychophysiology, 32, 4-18.
37. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
38. Meredith, M. A., & Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science, 221, 389-391.
39. Meredith, M. A., & Stein, B. E. (1986a). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365, 350-354.
40. Meredith, M. A., & Stein, B. E. (1986b), Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56, 640-662.
41. Miller, J. O. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247-279.
42. Miller, J. O. (1986). Time course of coactivation in bimodal divided attention. Perception and Psychophysics, 40, 331-343.
43. Mistlin, A. J., & Perrett, D. I. (1990). Visual and somatosensory processing in the macaque temporal cortex: The role of "expectation." Experimental Brain Research, 82, 437-450.
44. Morrell, F. (1972). Visual system's view of acoustic space. Nature, 238, 44-46.
45. O'Hare, J. J. (1991). Perceptual integration. Journal of the Washington Academy of Sciences, 81, 44-59.
46. O'Leary, A., & Rhodes, G. (1984). Cross-modal effects on visual and auditory object perception. Perception and Psychophysics, 35, 565-569.
47. Oram, M. W., & Perrett, D. I. (1996). Integration of form and motion in the anterior superior temporal polysensory area (STPa) of the macaque monkey. Journal of Neurophysiology, 76, 109-129.
48. Paulesu, E., Harrison, J., Baron Cohen, S., Watson, J. D. G., Goldstein, L., Heather, J., Frackowiak, R. S. J., & Frith, C. D. (1995). The physiology of colored hearing. A PET activation study of color-word synaesthesia. Brain. 118, 661-676.
49. Peck, C. K. (1987). Auditory interactions in cat's superior colliculus: Their role in the control of gaze. Brain Research, 420, 162-166.
50. Perrin, F., Pernier, J., Bertrand, O., & Echallier, J. F. (1989). Spherical splines for scalp potential and current density mapping. Electroencephalography and Clinical Neurophysiology, 72, 184-187.
51. Pick, H. L., Jr., Warren, D. H., & Hay, J. C. (1969). Sensory conflict in judgements of spatial direction. Perception and Psychophysics, 6, 203-205.
52. Posner, M. I., Nissen, M. J., & Klein, R. M. (1976). Visual dominance: An information-processing account of its origins and significance. Psychological Review, 83, 157-171.
53. Roda, J. M., & Reinoso-Suarez, F. (1983). Topographic organization of the thalamic projections to the cortex of the anterior ectosylvian sulcus in the cat. Experimental Brain Research, 49, 131-139.
54. Rugg, M. D., Doyle, M. C., & Wells, T. (1995). Word and non-word repetition within-and across-modality; An event-related potential study. Journal of Cognitive Neuroscience, 7, 209-227.
55. Sams, M., Aulanko, R., Hamalainen, H., Hari, R., Lounasmaa, O. V., Lu, S. T., & Simola, J. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127, 141-145.
56. Sams, M., & Imada, T. (1997). Integration of auditory and visual information in the human brain: Neuromagnetic evidence. Society for Neuroscience Abstracts, 23, 1305.
57. Scherg, M., & von Cramon, D. (1986). Evoked dipole source potentials of the human auditory cortex. Etectroencephalography and Clinical Neurophysiology, 65, 344-360.
58. Schmidt, R. A., Giele, S. C. A. M., & van den Heuvel, P. J. M. (1984). The locus of intersensory facilitation of reaction time. Acta Psychologica, 57, 145-164.
59. Schroeder, C. E., Mehta, A. D., Ulbert, I., Steinschneider, M., & Vaughan, H. G. (1995). Visual responses in auditory cortex and their modulation by attention. Society for Neuroscience Abstracts, 21, 694.
60. Schröger, E., & Widmann, A. (1997). Crossmodal effects in the processing of task-irrelevant location changes. Psychophysiology, 34, Suppl. 1, S80.
61. Schröger, E., & Widman, A. (1998) Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology, 35, 755-759.
62. Sekuler, R., Sekuler, A. B., & Lau, R. (1997). Sounds alter visual motion perception. Nature, 385, 308.
63. Shelton, B. R., & Searle, C. L. (1980). The influence of vision on the absolute identification of sound-source position. Perception and Psychophysics, 28, 589-596.
64. Singer, W., & Gray, C. M. (1995). Visual feature integration and the temporal correlation hypothesis. Annual Review of Neuroscience, 18, 555-586.
65. Spence, C. J., & Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting. Perception and Psychophysics, 59, 1-22.
66. Stein, B. E., Huneycutt, W. S., & Meredith, M. A. (1988). Neurons and behavior: The same rules of multisensory integration apply. Brain Research, 448, 355-358.
67. Stein, B. E., London, N., Wilkinson, L. K., & Price, D. D. (1996). Enhancement of perceived visual intensity by auditory stimuli: A psychophysical analysis. Journal of Cognitive Neuroscience, 8, 497-506.
68. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press.
69. Stein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade, L. (1989). Behavioral indices of multisensory integration: Orientation to visual cues is affected by auditory stimuli. Journal of Cognitive Neuroscience, 1, 12-24.
70. Stein, B. E., Meredith, M. A., & Wallace, M. T. (1994). Development and neural basis of multisensory integration. In D. J. Lewkowicz & R. Lickliter (Eds.), The development of intersensory perception: Comparative perspectives (pp. 81-105). Hillsdale, NJ: Erlbaum
71. Stein, B. E., & Wallace, M. T. (1996). Comparison of cross-modality integration in midbrain and cortex. In M. Norita, T. Bando, & B. Stein (Eds.), Progress in brain research (vol. 112, pp. 289-299). Elsevier Science.
72. Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381, 520-522.
73. Toldi, J., Feher, O., & Feuer, L. (1984). Dynamic interactions of evoked potentials in a polysensory cortex of the cat. Neuroscience, 13, 945-952.
74. Treisman, A. (1996). The binding problem. Current Opinion in Neurobiology, 6, 171-178.
75. Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale, & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549-586). Cambridge, MA: MIT Press.
76. Vaadia, E., Benson, D. A., Heinz, R. D., & Golstein, M. H. (1986). Unit study of monkey frontal cortex: Active localization of auditory and of visual stimuli. Journal of Neurophysiology, 56, 934-952.
77. Vaughan, H. G., & Ritter, W. (1970). The sources of auditory evoked responses recorded from the human scalp. Electroencephalography and Clinical Neurophysiology, 28, 360-367.
78. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1992). Integration of multiple sensory modalities in cat cortex. Experimental Brain Research, 91, 484-488.
79. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1993). Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. Journal of Neurophysiology, 69, 1797-1809.
80. Wallace, M. T., Wilkinson, L. K., & Stein, B. E. (1996). Representation and integration of multiple sensory inputs in primate superior colliculus. Journal of Neurophysiology, 76, 1246-1266.
81. Walsh, V. (1996). Perception: The seeing ear. Current Biology, 6, 389-391.
82. Walter, W. G. (1964). The convergence and interaction of visual, auditory and tactile responses in human non-specific cortex. Annals of the New York Academy of Science, 122, 320-361.
83. Warren, D. H., McCarthy, T. J., & Welch, R. B. (1983). Discrepancy and nondiscrepancy methods of assessing visual-auditory interaction. Perception and Psychophysics, 33, 413-419.
84. Watanabe, J., & Iwai, E. (1991). Neuronal activity in visual, auditory and polysensory areas in the monkey temporal cortex during a visual fixation task. Brain Research Bulletin, 26, 583-592.
85. Welch, R. B., & Warren, D. H. (1986). Intersensory interactions. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, volume I: Sensory processes and perception (pp. 25-1-25-36). New York: Wiley.
86. Wolpaw, J. R., & Penry, J. K. (1975). A temporal component of the auditory evoked response. Electroencephalography and Clinical Neurophysiology, 39, 609-620.