Chapter 14 Combining visual and auditory information

Progress in Brain Research

Volumn 155 B, Issue , 2006, Pages 243-258

  Burr, David ^a,b   Alais, David ^c  

^a UNIVERSITY OF FLORENCE   (Italy)

^b CNR   (Italy)

^c UNIVERSITY OF SYDNEY   (Australia)

Author keywords

attention; audition; crossmodal integration; flash lag effect; ventriloquist effect; vision

Indexed keywords

ATTENTION; AUDITORY STIMULATION; BAYES THEOREM; HEARING; LATENT PERIOD; MATHEMATICAL ANALYSIS; PRIORITY JOURNAL; REVIEW; VISION; VISUAL STIMULATION; ANIMAL; HUMAN; MENTAL FUNCTION; METHODOLOGY; PHOTOSTIMULATION; PHYSIOLOGY; PSYCHOPHYSICS;

ACOUSTIC STIMULATION; ANIMALS; AUDITORY PERCEPTION; HUMANS; MENTAL PROCESSES; PHOTIC STIMULATION; PSYCHOPHYSICS; VISUAL PERCEPTION;

EID: 33749261415     PISSN: 00796123     EISSN: None     Source Type: Book Series    
DOI: 10.1016/S0079-6123(06)55014-9     Document Type: Review

References (83)

1. Alais D., and Burr D. The "flash-lag" effect occurs in audition and cross-modally. Curr. Biol. 13 (2003) 59-63
2. Alais D., and Burr D. No direction-specific bimodal facilitation for audiovisual motion detection. Brain Res. Cogn. Brain Res. 19 2 (2004) 185-194
3. Alais D., and Burr D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14 3 (2004) 257-262
4. Alais D., and Carlile S. Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proc. Natl. Acad. Sci. USA 102 6 (2005) 2244-2247
5. Alais D., and Weston E. Temporal ventriloquism: perceptual shifts in temporal position and improved audiovisual precision predicted by maximum likelihood estimation. J. Vis. 6 (2006) 171
6. Arrighi R., Alais D., and Burr D. Neural latencies do not explain the auditory and audio-visual flash-lag effect. Vision Res. 45 (2005) 917-2925
7. Bald L., Berrien F.K., Price J.B., and Sprague R.O. Errors in perceiving the temporal order of auditory and visual stimuli. J. Appl. Psychol. 26 (1942) 382-388
8. Baldo M.V., and Klein S.A. Extrapolation or attention shift?. Nature 378 6557 (1995) 565-566
9. Battaglia P.W., Jacobs R.A., and Aslin R.N. Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image. Sci. Vis. 20 7 (2003) 1391-1397
10. Berger T.D., Martelli M., and Pelli D.G. Flicker flutter: is an illusory event as good as the real thing?. J. Vis. 3 6 (2003) 406-412
11. Bonnel A.M., and Hafter E.R. Divided attention between simultaneous auditory and visual signals. Percept. Psychophys. 60 2 (1998) 179-190
12. Boynton G.M., Demb J.B., Glover G.H., and Heeger D.J. Neuronal basis of contrast discrimination. Vision Res. 39 2 (1999) 257-269
13. Brefczynski J.A., and DeYoe E.A. A physiological correlate of the 'spotlight' of visual attention. Nat. Neurosci. 2 4 (1999) 370-374
14. Bremmer F., Schlack A., Shah N.J., Zafiris O., Kubischik M., Hoffmann K., Zilles K., and Fink G.R. Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys. Neuron 29 1 (2001) 287-296
15. Britten K.H., Newsome W.T., Shadlen M.N., Celebrini S., and Movshon J.A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 13 1 (1996) 87-100
16. Britten K.H., Shadlen M.N., Newsome W.T., and Movshon J.A. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12 12 (1992) 4745-4765
17. Bronkhorst A.W., and Houtgast T. Auditory distance perception in rooms. Nature 397 6719 (1999) 517-520
18. Burr D.C., Morrone M.C., and Banks M.S. The ventriloquist effect in time is consistent with optimal combination across senses. J. Vis. 6 (2006) 387
19. Bushara K.O., Grafman J., and Hallett M. Neural correlates of auditory-visual stimulus onset asynchrony detection. J. Neurosci. 21 1 (2001) 300-304
20. Caclin A., Soto-Faraco S., Kingstone A., and Spence C. Tactile "capture" of audition. Percept. Psychophys. 64 4 (2002) 616-630
21. Chambers C.D., Stokes M.G., and Mattingley J.B. Modality-specific control of strategic spatial attention in parietal cortex. Neuron 44 6 (2004) 925-930
22. Clarke J.J., and Yuille A.L. Data Fusion for Sensory Information Processing (1990), Kluwer Academic, Boston
23. Colby C.L., Duhamel J.R., and Goldberg M.E. Ventral intraparietal area of the macaque: anatomic location and visual response properties. J. Neurophysiol. 69 3 (1993) 902-914
24. Connor S. Dumbstruck: A Cultural History of Ventriloquism (2000), Oxford University Press, Oxford
25. Corbetta M., and Shulman G.L. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3 3 (2002) 201-215
26. Corey D.P., and Hudspeth A.J. Response latency of vertebrate hair cells. Biophys. J. 26 3 (1979) 499-506
27. Desimone R., and Duncan J. Neural mechanisms of selective visual attention. Ann. Rev. Neurosci. 18 (1995) 193-222
28. Driver J., and Spence C. Spatial synergies between auditory and visual attention. In: Umiltà C., and Moscovitch M. (Eds). Attention and Performance: Conscious and Nonconscious Information Processing Vol. 15 (1994), MIT Press, Cambridge, MA 311-331
29. Driver J., and Spence C. Crossmodal spatial attention: evidence from human performance. In: Spence C., and Driver J. (Eds). Crossmodal Space and Crossmodal Attention (2004), Oxford University Press, Oxford
30. Duncan J., Martens S., and Ward R. Restricted attentional capacity within but not between sensory modalities. Nature 387 6635 (1997) 808-810
31. Eagleman D.M., and Sejnowski T.J. Motion integration and postdiction in visual awareness. Science 287 5460 (2000) 2036-2038
32. Efron, B. and Tibshirani, R.J., 1993. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, Vol. 57. Chapman & Hall, New York.
33. Ernst M.O., and Banks M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415 6870 (2002) 429-433
34. Ernst M.O., and Bülthoff H.H. Merging the senses into a robust percept. Trends Cogn. Sci. 8 4 (2004) 162-169
35. Ferlazzo F., Couyoumdjian M., Padovani T., and Belardinelli M.O. Head-centred meridian effect on auditory spatial attention orienting. Q. J. Exp. Psychol. A 55 3 (2002) 937-963
36. Gandhi S.P., Heeger D.J., and Boynton G.M. Spatial attention affects brain activity in human primary visual cortex. Proc. Natl. Acad. Sci. USA 96 6 (1999) 3314-3319
37. Ghahramani Z., Wolpert D.M., and Jordan M.I. Computational models of sensorimotor integration. In: Morasso P.G., and Sanguineti V. (Eds). Self-Organization, Computational Maps and Motor Control (1997), Elsevier Science, Amsterdam 117-147
38. Grady C.L., Van Meter J.W., Maisog J.M., Pietrini P., Krasuski J., and Rauschecker J.P. Attention-related modulation of activity in primary and secondary auditory cortex. Neuroreport 8 11 (1997) 2511-2516
39. Graziano M.S. A system of multimodal areas in the primate brain. Neuron 29 1 (2001) 4-6
40. Hamlin A.J. On the least observable interval between stimuli addressed to disparate senses and to different organs of the same sense. Am. J. Psychol. 6 (1895) 564-575
41. Hillis J.M., Ernst M.O., Banks M.S., and Landy M.S. Combining sensory information: mandatory fusion within, but not between, senses. Science 298 5598 (2002) 1627-1630
42. Jacobs R.A. Optimal integration of texture and motion cues to depth. Vision Res. 39 21 (1999) 3621-3629
43. Jacobs R.A., and Fine I. Experience-dependent integration of texture and motion cues to depth. Vision Res. 39 24 (1999) 4062-4075
44. Jancke L., Mirzazade S., and Shah N.J. Attention modulates activity in the primary and the secondary auditory cortex: a functional magnetic resonance imaging study in human subjects. Neurosci. Lett. 266 2 (1999) 125-128
45. Kanwisher N., and Wojciulik E. Visual attention: insights from brain imaging. Nat. Rev. Neurosci. 1 2 (2000) 91-100
46. King A.J., and Palmer A.R. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain. Res. 60 3 (1985) 492-500
47. Kopinska A., and Harris L.R. Simultaneity constancy. Perception 33 9 (2004) 1049-1060
48. Krekelberg B., and Lappe M. A model of the perceived relative positions of moving objects based upon a slow averaging process. Vision Res. 40 2 (2000) 201-215
49. Krekelberg B., and Lappe M. Neuronal latencies and the position of moving objects. Trends Neurosci. 24 6 (2001) 335-339
50. Lamb T.D., and Pugh Jr. E.N. A quantitative account of the activation steps involved in phototransduction in amphibian photoreceptors. J. Physiol. 449 (1992) 719-758
51. Lennie P. The physiological basis of variations in visual latency. Vision Res. 21 6 (1981) 815-824
52. Lewald J., and Guski R. Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neurosci. Lett. 357 2 (2004) 119-122
53. Luck S.J., Chelazzi L., Hillyard S.A., and Desimone R. Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of macaque visual cortex. J. Neurophysiol. 77 1 (1997) 24-42
54. Macaluso E., Frith C.D., and Driver J. Supramodal effects of covert spatial orienting triggered by visual or tactile events. J. Cogn. Neurosci. 14 3 (2002) 389-401
55. Mackay D.M. Perceptual stability of a stroboscopically lit visual field containing self-luminous objects. Nature 181 4607 (1958) 507-508
56. Mateeff S., Hohnsbein J., and Noack T. Dynamic visual capture: apparent auditory motion induced by a moving visual target. Perception 14 6 (1985) 721-727
57. Meyer G.F., and Wuerger S.M. Cross-modal integration of auditory and visual motion signals. Neuroreport 12 11 (2001) 2557-2560
58. Mills A. On the minimum audible angle. J. Acoust. Soc. Am. 30 (1958) 237-246
59. Morein-Zamir S., Soto-Faraco S., and Kingstone A. Auditory capture of vision: examining temporal ventriloquism. Brain Res. Cogn. Brain. Res. 17 1 (2003) 154-163
60. Moutoussis K., and Zeki S. A direct demonstration of perceptual asynchrony in vision. Proc. R. Soc. Lond. B Biol. Sci. 264 1380 (1997) 393-399
61. Nijhawan R. Motion extrapolation in catching. Nature 370 6487 (1994) 256-257
62. Pashler H.E. The Psychology of Attention (1998), MIT Press, Cambridge, MA
63. Perrott D., and Saberi K. Minimum audible angle thresholds for sources varying in both elevation and azimuth. J. Acoust. Soc. Am. 87 (1990) 1728-1731
64. Pick H.L., Warren D.H., and Hay J.C. Sensory conflict in judgements of spatial direction. Percept. Psychophys. 6 (1969) 203-205
65. Purushothaman G., Patel S.S., Bedell H.E., and Ogmen H. Moving ahead through differential visual latency. Nature 396 6710 (1998) 424
66. Recanzone G.H., Schreiner C.E., and Merzenich M.M. Plasticity in the frequency representation of primary auditory cortex following discrimination training in adult owl monkeys. J. Neurosci. 13 1 (1993) 87-103
67. Shams L., Kamitani Y., and Shimojo S. Illusions. What you see is what you hear. Nature 408 6814 (2000) 788
68. Shams L., Kamitani Y., and Shimojo S. Visual illusion induced by sound. Brain Res. Cogn. Brain Res. 14 1 (2002) 147-152
69. Shams L., Kamitani Y., Thompson S., and Shimojo S. Sound alters visual evoked potentials in humans. Neuroreport 12 17 (2001) 3849-3852
70. Spence C., and Driver J. Audiovisual links in endogenous covert spatial attention. J. Exp. Psychol. Hum. Percept. Perform. 22 4 (1996) 1005-1030
71. Spence C., Ranson J., and Driver J. Cross-modal selective attention: on the difficulty of ignoring sounds at the locus of visual attention. Percept. Psychophys. 62 2 (2000) 410-424
72. Standage G.P., and Benevento L.A. The organization of connections between the pulvinar and visual area MT in the macaque monkey. Brain. Res. 262 2 (1983) 288-294
73. Stein B.E. Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Exp. Brain. Res. 123 1-2 (1998) 124-135
74. Sugita Y., and Suzuki Y. Audiovisual perception: implicit estimation of sound-arrival time. Nature 421 6926 (2003) 911
75. Titchener E.B. Lectures on the Elementary Psychology of Feeling and Attention (1908), MacMillan, New York
76. Triesch J., Ballard D.H., and Jacobs R.A. Fast temporal dynamics of visual cue integration. Perception 31 4 (2002) 421-434
77. Triesman A.M., and Davies A. Divided attention to ear and eye. In: Kornblum S. (Ed). Attention and Performance Vol. 4 (1973), Academic Press, New York
78. Warren D.H., Welch R.B., and McCarthy T.J. The role of visual-auditory "compellingness" in the ventriloquism effect: implications for transitivity among the spatial senses. Percept. Psychophys. 30 6 (1981) 557-564
79. Wickens, C.D., 1980. The structure of attentional resources. In: Attention and Performance, Vol. VIII. Erbaum, Hillsdale, NJ.
80. Woodruff P.W., Benson R.R., Bandettini P.A., Kwong K.K., Howard R.J., Talavage T., Belliveau J., and Rosen B.R. Modulation of auditory and visual cortex by selective attention is modality-dependent. Neuroreport 7 12 (1996) 1909-1913
81. Wuerger S.M., Hofbauer M., and Meyer G.F. The integration of auditory and visual motion signals at threshold. Percept. Psychophys. 65 8 (2003) 1188-1196
82. Zahorik P. Direct-to-reverberant energy ratio sensitivity. J. Acoust. Soc. Am. 112 5 Pt 1 (2002) 2110-2117
83. Zenger-Landolt B., and Heeger D.J. Response suppression in v1 agrees with psychophysics of surround masking. J. Neurosci. 23 17 (2003) 6884-6893