Multisensory integration, perception and ecological validity

Trends in Cognitive Sciences

Volumn 7, Issue 10, 2003, Pages 460-467

  De Gelder, Beatrice ^a,b   Bertelson, Paul ^a,c  

^a TILBURG UNIVERSITY   (Netherlands)

^b MASSACHUSETTS GENERAL HOSPITAL   (United States)

^c UNIVERSITÉ LIBRE DE BRUXELLES   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ECOLOGY; ENVIRONMENTAL IMPACT; INTEGRATION; MODAL ANALYSIS; SENSOR DATA FUSION;

SENSORY MODALITIES;

COGNITIVE SYSTEMS;

AUDITORY STIMULATION; BEHAVIOR; BRAIN DISEASE; BRAIN INJURY; COGNITION; CONFLICT; CONSCIOUSNESS; ENVIRONMENTAL FACTOR; HEARING; HUMAN; IDENTITY; LIGHT; ODOR; PERCEPTION; PREDICTION; REVIEW; SENSE ORGAN; SENSORY STIMULATION; SIGNAL TRANSDUCTION; SOUND; TASK PERFORMANCE; TOUCH; VISION; VISUAL INFORMATION;

EID: 0141644327     PISSN: 13646613     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.tics.2003.08.014     Document Type: Review

References (60)

1. Bertelson, P. and de Gelder, B. The psychology of multimodal perception. In Crossmodal Space and Crossmodal Attention (Spence, C. and Driver, J., eds), Oxford University Press (in press).
2. Radeau M. Auditory-visual spatial interaction and modularity. Curr. Psychol. Cogn. 13:1994;3-51.
3. Bertelson P. Starting from the ventriloquist: the perception of multimodal events. Sabourin M., et al. Advances in Psychological Science. 1998;419-439 Erlbaum.
4. Bertelson P. Ventriloquism: A case of crossmodal perceptual grouping. Aschersleben G., et al. Cognitive Contributions to the Perception of Spatial and Temporal Events. 1999;347-362 Elsevier.
5. Welch R.B., Warren D.H. Immediate perceptual response to intersensory discrepancy. Psychol. Bull. 88:1980;638-667.
6. Bermant R.I., Welch R.B. Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept. Mot. Skills. 42:1976;487-493.
7. Bertelson P., Radeau M. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29:1981;578-584.
8. Jack C.E., Thurlow W.R. Effects of degree of visual association and angle of displacement on the 'ventriloquism' effect. Percept. Mot. Skills. 37:1973;967-979.
9. Choe C.S., et al. The 'ventriloquist effect': visual dominance or response bias? Percept. Psychophys. 18:1975;55-60.
10. Thomas G.J. Experimental study of the influence of vision on sound localization. J. Exp. Psychol. 28:1940;163-177.
11. Radeau M., Bertelson P. Auditory-visual interaction and the timing of inputs: Thomas (1941) revisited. Psychol. Res. 49:1987;17-22.
12. Bertelson P., Aschersleben G. Automatic visual bias of perceived auditory location. Psychon. B. Rev. 5:1998;482-489.
13. Hay J.C., et al. Visual capture produced by prism spectacles. Psychonom. Sci. 2:1965;215-216.
14. Pavani F., et al. Visual capture of touch: Out-of-the-body experiences with rubber gloves. Psychol. Sci. 11:2000;353-359.
15. Caclin A., et al. Tactile 'capture' of audition. Percept. Psychophys. 64:2002;616-630.
16. Jousmaki V., Hari R. Parchment-skin illusion: sound-biased touch. Curr. Biol. 8:1998;R190.
17. Guest S., et al. Audiotactile interactions in roughness perception. Exp. Brain Res. 146:2002;161-171.
18. Soto-Faraco S., et al. The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. Cogn. Brain Res. 14:2002;139-146.
19. Vroomen J., de Gelder B. Visual motion influences the contingent auditory motion aftereffect. Psychol. Sci. 14:2003;357-361.
20. Fendrich R., Corballis P.M. The temporal cross-capture of audition and vision. Percept. Psychophys. 63:2001;719-725.
21. Bertelson, P. and Aschersleben, G. Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from temporal order judgements. Int. J. Psychophysiol. (in press).
22. Morein-Zamir S., et al. Auditory capture of vision: examining temporal ventriloquism. Cogn. Brain Res. 17:2003;154-163.
23. Ernst M.O., Banks M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature. 415:2002;429-433.
24. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature. 264:1976;746-748.
25. Massaro D.W. Speechreading: illusion or window into pattern recognition. Trends Cogn. Sci. 3:1999;310-317.
26. Bertelson P., et al. Auditory-visual interaction in voice localization and in bimodal speech recognition: the effects of desynchronization. Benoit C., Campbell R. Proceedings of Rhodes Workshop on Audio-Visual Speech Processing: Cognitive and Computational Approaches. 1997;97-100 European Cooperation in Scientific and Technical Research.
27. Munhall K.G., et al. Temporal constraints on the McGurk effect. Percept. Psychophys. 58:1996;351-362.
28. Saldaña H.M., Rosenblum L.D. Visual influences on auditory pluck and bow judgments. Percept. Psychophys. 54:1993;406-416.
29. Sekuler R., et al. Sound alters visual motion perception. Nature. 385:1997;308.
30. Shimojo S., Shams L. Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11:2001;505-509.
31. de Gelder B., et al. Hearing smiles and seeing cries: the bimodal perception of emotions. Bull. Psychonomic Soc. 30:1995;196.
32. Massaro D.W., Egan P.B. Perceiving affect from the voice and the face. Psychonomic Bull. Rev. 3:1996;215-221.
33. de Gelder B., Vroomen J. The perception of emotion by ear and by eye. Cogn. Emot. 14:2000;289-311.
34. Held R. Plasticity in sensory-motor systems. Sci. Am. 213:1965;84-94.
35. Harris C.S. Perceptual adaptation to inverted, reversed, and displaced vision. Psychol. Rev. 72:1965;419-444.
36. Welch R.B. Perceptual Modification: Adapting to Altered Sensory Environments. 1978;Academic Press.
37. Radeau M., Bertelson P. The after-effects of ventriloquism. Q. J. Exp. Psychol. 26:1974;63-71.
38. Radeau M., Bertelson P. The effect of a textured visual field on modality dominance in a ventriloquism situation. Percept. Psychophys. 20:1976;227-235.
39. Recanzone G. Rapidly induced auditory plasticity: the ventriloquism aftereffect. Proc. Natl. Acad. Sci. U. S. A. 95:1998;869-875.
40. Bedford F. Constraints on learning new mappings between perceptual dimensions. J. Exp. Psychol.: Human Percept. Perform. 15:1989;232-248.
41. Frissen I., et al. The aftereffects of ventriloquism: are they sound-frequency specific? Acta Psychol. (Amst.). 113:2003;315-327.
42. Vroomen J., et al. The ventriloquist effect does not depend on the direction of automatic visual attention. Percept. Psychophys. 63:2001;651-659.
43. Driver J. Enhancement of selective listening by illusory mislocation of speech sounds due to lip reading. Nature. 381:1996;66-68.
44. Vroomen J., de Gelder B. Sound enhances visual perception: cross-modal effects of auditory organization on vision. J. Exp. Psychol. Hum. Percept. Perform. 26:2000;1583-1590.
45. Spence C., Driver J. Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism. Neuroreport. 11:2000;2057-2061.
46. Vroomen J., et al. Directing spatial attention towards the illusory location of a ventriloquized sound. Acta Psychol. (Amst.). 108:2001;21-33.
47. Dong C., et al. A contingent aftereffect in the auditory system. Nat. Neurosci. 2:1999;863-865.
48. McCullough C. Color adaptation of edge detectors in the human visual system. Science. 149:1965;1113-1116.
49. Bertelson P., et al. Ventriloquism in patients with unilateral visual neglect. Neuropsychologia. 38:2000;1634-1642.
50. de Gelder B., et al. Covert processing of faces in prosopagnosia is restricted to facial expressions: evidence from cross-modal bias. Brain Cogn. 44:2000;425-444.
51. Weiskrantz L. Consciousness Lost and Found: A Neuropsychological Exploration. 1997;Oxford University Press.
52. de Gelder B., et al. Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proc. Natl. Acad. Sci. U. S. A. 99:2002;4121-4126.
53. Fuster J.M., et al. Cross-modal and cross-temporal association in neurons of frontal cortex. Nature. 405:2000;347-351.
54. Calvert G.A., et al. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport. 10:1999;2619-2623.
55. Dolan R.J., et al. Crossmodal binding of fear in voice and face. Proc. Natl. Acad. Sci. U. S. A. 98:2001;10006-10010.
56. Sams M., Hari R. Magnetoencephalography in the study of human auditory information processing. Ann. N. Y. Acad. Sci. 620:1991;102-117.
57. Calvert G.A., et al. Activation of auditory cortex during silent lipreading. Science. 276:1997;593-596.
58. Stein B.E., Meredith M.A. The Merging of the Senses. 1993;MIT Press.
59. Calvert G.A., et al. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10:2000;649-657.
60. de Gelder, B. et al. Multisensory perception of emotion, its time course and its neural basis. In Handbook of Multisensory Perception (Stein, B et al., eds), MIT Press (in press).