Perception of intersensory synchrony: A tutorial review

Attention, Perception, and Psychophysics

Volumn 72, Issue 4, 2010, Pages 871-884

  Vroomen, Jean ^a   Keetels, Mirjam ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTATION; BRAIN; BRAIN MAPPING; CHRONOBIOLOGY; DECISION MAKING; DIFFERENTIAL THRESHOLD; HUMAN; METHODOLOGY; PERCEPTION; PHYSIOLOGY; REVIEW; SENSATION; STATISTICS; STIMULATION; TIME;

ADAPTATION, BIOLOGICAL; BRAIN; BRAIN MAPPING; CHRONOBIOLOGY DISCIPLINE; DIFFERENTIAL THRESHOLD; HUMANS; JUDGMENT; PERCEPTION; PHYSICAL STIMULATION; SENSATION; TIME FACTORS;

EID: 77955287335     PISSN: 19433921     EISSN: None     Source Type: Journal    
DOI: 10.3758/APP.72.4.871     Document Type: Review

References (130)

1. Alais, D., & Carlile, S. (2005). Synchronizing to real events: Subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proceedings of the National Academy of Sciences, 102, 2244-2247.
2. Allik, J., & Kreegipuu, K. (1997). Multiple visual latency. Perception, 26 (ECVP Abstract Suppl.).
3. Arnold, D. H., Johnston, A., & Nishida, S. (2005). Timing sight and sound. Vision Research, 45, 1275-1284.
4. Asakawa, K., Tanaka, A., & Imai, H. (2009, July). Temporal recalibration in audio-visual speech integration using a simultaneity judgment task and the McGurk identification task. Paper presented at the 31st Annual Meeting of the Cognitive Science Society, Amsterdam.
5. Bald, L., Berrien, F. K., Price, J. B., & Sprague, R. O. (1942). Errors in perceiving the temporal order of auditory and visual stimuli. Journal of Applied Psychology, 26, 382-388.
6. Bedford, F. L. (1989). Constraints on learning new mappings between perceptual dimensions. Journal of Experimental Psychology: Human Perception & Performance, 15, 232-248.
7. Benjamins, J. S., van der Smagt, M. J., & Verstraten, F. A. J. (2008). Matching auditory and visual signals: Is sensory modality just another feature? Perception, 37, 848-858.
8. Bertelson, P. (1999). Ventriloquism: A case of crossmodal perceptual grouping. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 347-362). Amsterdam: Elsevier, North-Holland.
9. Bertelson, P., & Aschersleben, G. (1998). Automatic visual bias of perceived auditory location. Psychonomic Bulletin & Review, 5, 482-489.
10. Bertelson, P., & Aschersleben, G. (2003). Temporal ventriloquism: Crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. International Journal of Psychophysiology, 50, 147-155.
11. Blauert, J. (1997). Spatial hearing: The psychophysics of human sound localization (J. S. Allen, Trans.) (Rev. ed.). Cambridge, MA: MIT Press.
12. Calvert, G. [A.], Spence, C., & Stein, B. E. (Eds.) (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press.
13. Conrey, B., & Pisoni, D. B. (2006). Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. Journal of the Acoustical Society of America, 119, 4065-4073.
14. Di Luca, M., Machulla, T.-K., & Ernst, M. O. (2007, July). Perceived timing across modalities. Paper presented at the International Intersensory Research Symposium 2007: Perception and Action, Sydney, Australia.
15. Di Luca, M., Machulla, T.-K., & Ernst, M. O. (2009). Recalibration of multisensory simultaneity: Cross-modal transfer coincides with a change in perceptual latency, Journal of Vision, 9, 1-16.
16. Dinnerstein, A. J., & Zlotogura, P. (1968). Intermodal perception of temporal order and motor skills: Effects of age. Perceptual & Motor Skills, 26, 987-1000.
17. Dixon, N. F., & Spitz, L. (1980). The detection of auditory visual desynchrony. Perception, 9, 719-721.
18. Engel, G. R., & Dougherty, W. G. (1971). Visual-auditory distance constancy. Nature, 234, 308. doi:10.1038/234308a0
19. Enns, J. T. (2004). Object substitution and its relation to other forms of visual masking. Vision Research, 44, 1321-1331.
20. Enns, J. T., & Di Lollo, V. (1997). Object substitution: A new form of masking in unattended visual locations. Psychological Science, 8, 135-139.
21. Enns, J. T., & Di Lollo, V. (2000). What's new in visual masking? Trends in Cognitive Sciences, 4, 345-352.
22. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429-433.
23. Ernst, M. O., Banks, M. S., & Bülthoff, H. H. (2000). Touch can change visual slant perception. Nature Neuroscience, 3, 69-73.
24. Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162-169.
25. Fain, G. L. (2003). Sensory transduction. Sunderland, MA: Sinauer.
26. Fendrich, R., & Corballis, P. M. (2001). The temporal cross-capture of audition and vision. Perception & Psychophysics, 63, 719-725.
27. Finger, R., & Davis, A. W. (2001). Measuring video quality in videoconferencing systems (Tech. Rep. No. SN187-D). Los Gatos, CA: Pixel Instrument Corporation.
28. Freeman, E., & Driver, J. (2008). Direction of visual apparent motion driven solely by timing of a static sound. Current Biology, 18, 1262-1266.
29. Frey, R. D. (1990). Selective attention, event perception and the criterion of acceptability principle: Evidence supporting and rejecting the doctrine of prior entry. Human Movement Science, 9, 481-530.
30. Fujisaki, W., & Nishida, S. (2005). Temporal frequency characteristics of synchrony-asynchrony discrimination of audio-visual signals. Experimental Brain Research, 166, 455-464.
31. Fujisaki, W., & Nishida, S. (2009). Audio-tactile superiority over visuo-tactile and audio-visual combinations in the temporal resolution of synchrony perception. Experimental Brain Research, 198, 245-259.
32. Fujisaki, W., Shimojo, S., Kashino, M., & Nishida, S. (2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7, 773-778.
33. Getzmann, S. (2007). The effect of brief auditory stimuli on visual apparent motion. Perception, 36, 1089-1103.
34. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin.
35. Hanson, J. V. M., Heron, J., & Whitaker, D. (2008). Recalibration of perceived time across sensory modalities. Experimental Brain Research, 185, 347-352.
36. Harrar, V., & Harris, L. R. (2005). Simultaneity constancy: Detecting events with touch and vision. Experimental Brain Research, 166, 465-473.
37. Harrar, V., & Harris, L. R. (2008). The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Experimental Brain Research, 186, 517-524.
38. Helmholtz, H. von (1867). Handbuch der physiologischen Optik. Leipzig: Voss.
39. Heron, J., Whitaker, D., McGraw, P. V., & Horoshenkov, K. V. (2007). Adaptation minimizes distance-related audiovisual delays. Journal of Vision, 7, 1-8.
40. Hirsh, I. J., & Fraisse, P. (1964). Simultanéité et succession de stimuli hétérogènes. L' Année Psychologique, 64, 1-19.
41. Hirsh, I. J., & Sherrick, C. E., Jr. (1961). Perceived order in different sense modalities. Journal of Experimental Psychology, 62, 423-432.
42. Jaśkowski, P. (1999). Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 265-282). Amsterdam: Elsevier.
43. Jaśkowski, P., Jaroszyk, F., & Hojan-Jezierska, D. (1990). Temporalorder judgments and reaction time for stimuli of different modalities. Psychological Research, 52, 35-38.
44. Jaśkowski, P., & Verleger, R. (2000). Attentional bias toward lowintensity stimuli: An explanation for the intensity dissociation between reaction time and temporal order judgment? Consciousness & Cognition, 9, 435-456.
45. Jones, J. A., & Jarick, M. (2006). Multisensory integration of speech signals: The relationship between space and time. Experimental Brain Research, 174, 588-594.
46. Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008). Visual modulation of neurons in auditory cortex. Cerebral Cortex, 18, 1560-1574.
47. Keetels, M., & Vroomen, J. (2005). The role of spatial disparity and hemifields in audio-visual temporal order judgments. Experimental Brain Research, 167, 635-640.
48. Keetels, M., & Vroomen, J. (2007). No effect of auditory-visual spatial disparity on temporal recalibration. Experimental Brain Research, 182, 559-565.
49. Keetels, M., & Vroomen, J. (2008a). Tactile-visual temporal ventriloquism: No effect of spatial disparity. Perception & Psychophysics, 70, 765-771. doi:10.3758/PP.70.5.765
50. Keetels, M., & Vroomen, J. (2008b). Temporal recalibration to tactile-visual asynchronous stimuli. Neuroscience Letters, 430, 130-134.
51. King, A. J., & Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Experimental Brain Research, 60, 492-500.
52. Kopinska, A., & Harris, L. R. (2004). Simultaneity constancy. Perception, 33, 1049-1060.
53. Lewald, J., & Guski, R. (2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cognitive Brain Research, 16, 468-478.
54. Lewald, J., & Guski, R. (2004). Auditory-visual temporal integration as a function of distance: No compensation for sound-transmission time in human perception. Neuroscience Letters, 357, 119-122.
55. Lewkowicz, D. J. (1996). Perception of auditory-visual temporal synchrony in human infants. Journal of Experimental Psychology: Human Perception & Performance, 22, 1094-1106.
56. Macefield, G., Gandevia, S. C., & Burke, D. (1989). Conduction velocities of muscle and cutaneous afferents in the upper and lower limbs of human subjects. Brain, 112, 1519-1532.
57. MacKay, D. M. (1958). Perceptual stability of a stroboscopically lit visual field containing self-luminous objects. Nature, 181, 507-508. doi:10.1038/181507a0
58. Mattes, S., & Ulrich, R. (1998). Directed attention prolongs the perceived duration of a brief stimulus. Perception & Psychophysics, 60, 1305-1317.
59. McGrath, M., & Summerfield, Q. (1985). Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America, 77, 678-685.
60. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
61. Mitrani, L., Shekerdjiiski, S., & Yakimoff, N. (1986). Mechanisms and asymmetries in visual perception of simultaneity and temporal order. Biological Cybernetics, 54, 159-165.
62. Mollon, J. D., & Perkins, A. J. (1996). Errors of judgment at Greenwich in 1796. Nature, 380, 101-102.
63. Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154-163.
64. Mortlock, A. N., Machin, D., McConnell, S., & Sheppard, P. (1997). Virtual conferencing. BT Technology Journal, 15, 120-129.
65. Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58, 351-362.
66. Navarra, J., Hartcher-O'Brien, J., Piazza, E., & Spence, C. (2009). Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proceedings of the National Academy of Sciences, 106, 9169-9173.
67. Navarra, J., Soto-Faraco, S., & Spence, C. (2007). Adaptation to audiotactile asynchrony. Neuroscience Letters, 413, 72-76.
68. Navarra, J., Vatakis, A., Zampini, M., Soto-Faraco, S., Humphreys, W., & Spence, C. (2005). Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Research, 25, 499-507.
69. Neumann, O., & Niepel, M. (2004). Timing of "perception" and perception of "time." In C. Kaernbach, E. Schröger, & H. Müller (Eds.), Psychophysics beyond sensation: Laws and invariants of human cognition (pp. 245-269). Mahwah, NJ: Erlbaum.
70. Nijhawan, R. (1994). Motion extrapolation in catching. Nature, 370, 256-257.
71. Nijhawan, R. (1997). Visual decomposition of colour through motion extrapolation. Nature, 386, 66-69.
72. Nijhawan, R. (2002). Neural delays, visual motion and the flash-lag effect. Trends in Cognitive Sciences, 6, 387-393.
73. Petrini, K., Russell, M., & Pollick, F. (2009). When knowing can replace seeing in audiovisual integration of actions. Cognition, 110, 432-439.
74. Pöppel, E. (1985). Grenzen des Bewußtseins. Stuttgart: Deutsche VerlagsAnstalt. [Translated as Mindworks: Time and conscious experience. New York: Harcourt Brace Jovanovich, 1988.]
75. Pöppel, E., Schill, K., & von Steinbüchel, N. (1990). Sensory integration within temporally neutral systems states: A hypothesis. Naturwissenschaften, 77, 89-91. doi:10.1007/BF01131783
76. Radeau, M. (1994). Auditory-visual spatial interaction and modularity. Cahiers de Psychologie Cognitive, 13, 3-51.
77. Rihs, S. (1995). The influence of audio on perceived picture quality and subjective audio-visual delay tolerance. In R. Hamberg & H. de Ridder (Eds.), Advanced methods for the evaluation of television picture quality: Proceedings of the MOSAIC workshop (pp. 133-137). Eindhoven: Institute for Perception Research.
78. Roufs, J. A. J. (1963). Perception lag as a function of stimulus luminance. Vision Research, 3, 81-91.
79. Rutschmann, J., & Link, R. (1964). Perception of temporal order of stimuli differing in sense mode and simple reaction time. Perceptual & Motor Skills, 18, 345-352.
80. Sanford, A. J. (1971). Effects of changes in the intensity of white noise on simultaneity judgments and simple reaction time. Quarterly Journal of Experimental Psychology, 23, 296-303.
81. Scheier, C. R., Nijhawan, R., & Shimojo, S. (1999). Sound alters visual temporal resolution. Investigative Ophthalmology & Visual Science, 40, 4169.
82. Schneider, K. A., & Bavelier, D. (2003). Components of visual prior entry. Cognitive Psychology, 47, 333-366.
83. Sekuler, R., Sekuler, A. B., & Lau, R. (1997). Sound alters visual motion perception. Nature, 385, 308. doi:10.1038/385308a0
84. Shams, L., Kamitani, Y., & Shimojo, S. (2002). Visual illusion induced by sound. Cognitive Brain Research, 14, 147-152.
85. Shimojo, S., Scheier, C., Nijhawan, R., Shams, L., Kamitani, Y., & Watanabe, K. (2001). Beyond perceptual modality: Auditory effects on visual perception. Acoustical Science & Technology, 22, 61-67.
86. Shipley, T. (1964). Auditory flutter-driving of visual flicker. Science, 145, 1328-1330.
87. Shore, D. I., & Spence, C. (2005). Prior entry. In L. Itti, G. Rees, & J. K. Tsotsos (Eds.), Neurobiology of attention (pp. 89-95). Amsterdam: Elsevier, North-Holland.
88. Shore, D. I., Spence, C., & Klein, R. M. (2001). Visual prior entry. Psychological Science, 12, 205-212.
89. Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12, 7-10.
90. Smith, W. F. (1933). The relative quickness of visual and auditory perception. Journal of Experimental Psychology, 16, 239-257.
91. Soto-Faraco, S., & Alsius, A. (2007). Conscious access to the unisensory components of a cross-modal illusion. NeuroReport, 18, 347-350.
92. Soto-Faraco, S., & Alsius, A. (2009). Deconstructing the McGurk- MacDonald illusion. Journal of Experimental Psychology: Human Perception & Performance, 35, 580-587.
93. Spence, C. (2007). Audiovisual multisensory integration. Acoustic Science & Technology, 28, 61-70.
94. Spence, C., Baddeley, R., Zampini, M., James, R., & Shore, D. I. (2003). Multisensory temporal order judgments: When two locations are better than one. Perception & Psychophysics, 65, 318-328.
95. Spence, C., Shore, D. I., & Klein, R. M. (2001). Multisensory prior entry. Journal of Experimental Psychology: General, 130, 799-832.
96. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press, Bradford Books.
97. Stekelenburg, J. J., & Vroomen, J. (2005). An event-related potential investigation of the time-course of temporal ventriloquism. NeuroReport, 16, 641-644.
98. Stekelenburg, J. J., & Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience, 19, 1964-1973.
99. Stelmach, L. B., & Herdman, C. M. (1991). Directed attention and perception of temporal order. Journal of Experimental Psychology: Human Perception & Performance, 17, 539-550.
100. Stetson, C., Cui, X., Montague, P. R., & Eagleman, D. M. (2006). Motor-sensory recalibration leads to an illusory reversal of action and sensation. Neuron, 51, 651-659.
101. Stone, J. V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., et al. (2001). When is now? Perception of simultaneity. Proceedings of the Royal Society B, 268, 31-38.
102. Sugita, Y., & Suzuki, Y. (2003). Implicit estimation of sound-arrival time. Nature, 421, 911.
103. Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.
104. Summerfield, A. Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 3-51). London: Erlbaum.
105. Takahashi, K., Saiki, J., & Watanabe, K. (2008). Realignment of temporal simultaneity between vision and touch. NeuroReport, 19, 319-322.
106. Tanaka, A., Sakamoto, S., Tsumura, K., & Suzuki, Y. (2009). Visual speech improves the intelligibility of time-expanded auditory speech. NeuroReport, 20, 473-477.
107. Teatini, G., FernÈ, M., Verzella, F., & Berruecos, P., Jr. (1976). Perception of temporal order: Visual and auditory stimuli. Giornale Italiano di Psicologia, 3, 157-164.
108. van der Burg, E., Cass, J., Olivers, C. N. L., Theeuwes, J., & Alais, D. (2009). Efficient visual search from nonspatial auditory cues requires more than temporal synchrony. In E. van der Burg (Ed.), Temporal multisensory processing and its effects on attention (pp. 63-84). Amsterdam: Vrije Universiteit.
109. van Eijk, R. L. J. (2008). Audio-visual synchrony perception. Unpublished doctoral dissertation, Technische Universiteit Eindhoven, Eindhoven.
110. van Eijk, R. L. J., Kohlrausch, A., Juola, J. F., & van de Par, S. (2008). Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Perception & Psychophysics, 70, 955-968.
111. van Eijk, R. L. J., Kohlrausch, A., Juola, J. F., & van de Par, S. (2009). Temporal interval discrimination thresholds depend on perceived synchrony for audio-visual stimulus pairs. Journal of Experimental Psychology: Human Perception & Performance, 35, 1254-1263.
112. van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neuropsychologia, 45, 598-607.
113. Vatakis, A., Ghazanfar, A. A., & Spence, C. (2008). Facilitation of multisensory integration by the "unity effect" reveals that speech is special. Journal of Vision, 8, 1-11.
114. Vatakis, A., Navarra, J., Soto-Faraco, S., & Spence, C. (2007). Temporal recalibration during asynchronous audiovisual speech perception. Experimental Brain Research, 181, 173-181.
115. Vatakis, A., & Spence, C. (2006). Audiovisual synchrony perception for music, speech, and object actions. Brain Research, 1111, 134-142.
116. Vatakis, A., & Spence, C. (2007). Crossmodal binding: Evaluating the "unity assumption" using audiovisual speech stimuli. Perception & Psychophysics, 69, 744-756.
117. Vatakis, A., & Spence, C. (2008). Evaluating the influence of the "unity assumption" on the temporal perception of realistic audiovisual stimuli. Acta Psychologica, 127, 12-23.
118. Vroomen, J., & de Gelder, B. (2004a). Perceptual effects of crossmodal stimulation: Ventriloquism and the freezing phenomenon. In G. [A.] Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 141-150). Cambridge, MA: MIT Press.
119. Vroomen, J., & de Gelder, B. (2004b). Temporal ventriloquism: Sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception & Performance, 30, 513-518.
120. Vroomen, J., & Keetels, M. (2009). Sounds change four-dot masking. Acta Psychologica, 130, 58-63.
121. Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research, 22, 32-35.
122. Vroomen, J., & Stekelenburg, J. J. (2010). Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. Journal of Cognitive Neuroscience, 22, 1583-1596. doi:10.1162/jocn.2009.21308
123. Welch, R. B. (1999). Meaning, attention, and the "unity assumption" in the intersensory bias of spatial and temporal perceptions. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 371-387). Amsterdam: Elsevier.
124. Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638-667.
125. Yamamoto, S., Miyazaki, M., Iwano, T., & Kitazawa, S. (2008, July). Bayesian calibration of simultaneity in audiovisual temporal order judgment. Poster presented at the 9th International Multisensory Research Forum, Hamburg.
126. Zampini, M., Brown, T., Shore, D. I., Maravita, A., Röder, B., & Spence, C. (2005). Audiotactile temporal order judgments. Acta Psychologica, 118, 277-291.
127. Zampini, M., Guest, S., Shore, D. I., & Spence, C. (2005). Audio-visual simultaneity judgments. Perception & Psychophysics, 67, 531-544.
128. Zampini, M., Shore, D. I., & Spence, C. (2003a). Audiovisual temporal order judgments. Experimental Brain Research, 152, 198-210.
129. Zampini, M., Shore, D. I., & Spence, C. (2003b). Multisensory temporal order judgments: The role of hemispheric redundancy. International Journal of Psychophysiology, 50, 165-180.
130. Zampini, M., Shore, D. I., & Spence, C. (2005). Audiovisual prior entry. Neuroscience Letters, 381, 217-222.