The optimal time window of visual-auditory integration: A reaction time analysis

Frontiers in Integrative Neuroscience

Volumn , Issue MAY 2010, 2010, Pages

  Colonius, Hans ^a   Diederich, Adele ^b  

^a UNIVERSITY OF OLDENBURG   (Germany)

^b JACOBS UNIVERSITY BREMEN   (Germany)

Author keywords

Bayesian estimation; Multisensory integration; Optimal decision rule; Saccadic reaction time; Time window of integration; Twin model

Indexed keywords

ARTICLE; AUDITORY STIMULATION; AUDITORY SYSTEM; DECISION MAKING; NONHUMAN; OSCILLATION; REACTION TIME; REDUNDANCY ANALYSIS; SENSORY STIMULATION; VISUAL STIMULATION; VISUAL SYSTEM;

EID: 81855172153     PISSN: 16625145     EISSN: None     Source Type: Journal    
DOI: 10.3389/fnint.2010.00011     Document Type: Article

References (74)

1. Alais, D., and Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257-262.
2. Anastasio, T. J., Patton, P. E., and Belkacem-Boussaid, K. (2000). Using Bayes' rule to model multisensory enhancement in the superior colliculus. Neural Comput. 12, 1165-1187.
3. Battaglia, P. W., Jacobs, R. A., and Aslin, R. N. (2003). Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391-1397.
4. Beierholm, U., Koerding, K., Shams, L., and Ma, W. J. (2008). Comparing Bayesian models for multisensory cue combination without mandatory integration. Adv. Neural Inf. Process. Syst. 20, 81-88.
5. Bell, A. H., Meredith, A., Van Opstal, A. J., and Munoz, D. P. (2005). Crossmodal integration in the primate superior col-liculus underlying the preparation and initiation of saccadic eye movements. J. Neurophysiol. 93, 3659-3673.
6. Bell, A. H., Meredith, A., Van Opstal, A. J., and Munoz, D. P. (2006). Stimulus intensity modifes saccadic reaction time and visual response latency in the superior colliculus. Exp. Brain Res. 174, 53-59.
7. Burr, D., Silva, O., Cicchini, G. M., Banks, M., and Morrone, M. C. (2009). Temporal mechanisms of multimodal binding. Proc. R. Soc. Lond., B, Biol. Sci. 276, 1761-1769.
8. Chandrasekaran, C., and Ghazanfar, A. A. (2009). Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus. J. Neurophysiol. 101, 773-788.
9. Colonius, H., and Arndt, P. (2001). A two-stage model for visual-auditory interaction in saccadic latencies. Percept. Psychophys. 63, 126-147.
10. Colonius, H., and Diederich, A. (2004a). Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J. Cogn. Neurosci. 16, 1000-1009
11. Colonius, H., and Diederich, A. (2004b). Why aren't all deep superior colliculus neurons multisensory? A Bayes' Ratio Analysis. Cogn. Affect. Behav. Neurosci. 4, 344-353.
12. Colonius, H., Diederich, A., and Steenken, R. (2009). Time-window-of-integration (TWIN) model for saccadic reaction time: effect of auditory masker level on visual-auditory spatial interaction in elevation. Brain Topogr. 21, 177-184.
13. Conrey, B., and Pisoni, D. B. (2006). Auditory-visual speech perception and synchrony detection for speech and nonspeech signals J. Acoust. Soc. Am. 119, 4065-4073.
14. Corneil, B. D., and Munoz, D. P. (1996). The infuence of auditory and visual distractors on human orienting gaze shifts. J. Neurosci. 16, 8193-8207.
15. Corneil, B. D., Van Wanrooij, M., Munoz, D. P., and Van Opstal, A. J. (2002). Auditory-visual interactions subserving goal-directed saccades in a complex scene. J. Neurophysiol. 88, 438-454.
16. Diederich, A., and Colonius, H. (1987). Intersensory facilitation in the motor component? A reaction time analysis. Psychol. Res. 49, 23-29.
17. Diederich, A., and Colonius, H. (2004). "Modeling the time course of mul-tisensory interaction in manual and saccadic responses," in Handbook of Multisensory Processes, eds G. Calvert, C. Spence and B. E. Stein (Cambridge, MA: MIT Press), 395-408.
18. Diederich, A., and Colonius, H. (2007a). Why two "distractors" are better then one: modeling the effect on non-target auditory and tactile stimuli on visual saccadic reaction time. Exp. Brain Res. 179, 43-54.
19. Diederich, A., and Colonius, H. (2007b). Modeling spatial effects in visual-tactile saccadic reaction time. Percept. Psychophys. 69, 56-67.
20. Diederich, A., and Colonius, H. (2008a). Crossmodal interaction in saccadic reaction time: separating multisensory from warning effects in the time window of integration model. Exp. Brain Res. 186, 1-22.
21. Diederich, A., and Colonius, H. (2008b). When a high-intensity "distractor" is better then a low-intensity one: modeling the effect of an auditory or tactile nontarget stimulus on visual saccadic reaction time. Brain Res. 1242, 219-230.
22. Diederich, A., Colonius, H., and Schomburg, A. (2008). Assessing age-related multisensory enhancement with the time-window-of-integra-tion model. Neuropsychologia 46, 2556-2562.
23. Di Luca, M., Machulla, T. K., and Ernst, M. O. (2009). Recalibration of mul-tisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J. Vision 9, 1-16.
24. Dixon, N. F., and Spitz, L. (1980). The detection of auditory visual desyn-chrony. Perception 9, 719-721.
25. Egan, J. P. (1975). Signal Detection Theory and ROC Analysis. New York: Academic Press.
26. Ernst, M. O. (2005). "A Bayesian view on multimodal cue integration," in Human Body Perception From the Inside Out, eds G. Knoblich, I. Thornton, M. Grosjean and M. Shiffrar (New York, NY: Oxford University Press), 105-131.
27. Ernst, M. O., and Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429-433.
28. Frens, M. A., Van Opstal, A. J., and Van der Willigen, R. F. (1995). Spatial and temporal factors determine auditory- visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802-816.
29. Green, D. M., and Swets, J. A. (1974). Signal Detection Theory and Psychophysics. Oxford: Robert E. Krieger.
30. Hairston, W. D., Wallace, M. T., Vaughan, J. W., Stein, B. E., Norris, J. L., and Schirillo, J. A. (2003). Visual localization ability influences cross-modal bias. J. Cogn. Neurosci. 15, 20-29.
31. Hillis, J. M., Ernst, M. O., Banks, M. S., and Landy, M. S. (2002). Combining sensory information: mandatory fusion within, but not between, senses. Science 298, 1627-1630.
32. Keetels, M., and Vroomen, J. (2007). No effect of auditory-visual spatial disparity on temporal recalibration. Exp. Brain Res. 182, 559-65.
33. King, A. J., and Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60, 492-500.
34. Koerding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., and Shams, L. (2007). Causal inference in multisensory perception. PLoS ONE 2, e943. doi:10.1371/journal. pone.0000943.
35. Kotz, S., Kozubowski, T. J., and Podgórski, K. (2001). The Laplace Distribution and Generalizations. A Revisit with Applications to Communications, Economics, Engineering, and Finance. Boston: Birkhäuser.
36. Lakatos, P., Chen, C.M., O'Connell, M. N., Mills, A., and Schroeder, C. E. (2007). Neuronal oscillations and multisen-sory interaction in primary auditory cortex. Neuron 53, 279-292.
37. Lakatos, P., O'Connell, M. N., Barczak, A., Mills, A., Javitt, D. C., and Schroeder, C. E. (2009). The leading sense: supramodal control of neurophysi-ological context by attention. Neuron 64, 419-430.
38. Lakatos, P., Shah, A. S., Knuth, K. H., Ulbert, I., Karmos, G., and Schroeder, C. E. (2005). An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J. Neurophysiol. 94, 1904-1911.
39. Lewald, J., Ehrenstein, W. H., and Guski, R. (2001). Spatio-temporal constraints for auditory-visual integration. Behav. Brain Res. 121, 69-79.
40. Lewald, J., and Guski, R. (2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Brain Res. Cogn. Brain Res. 16, 468-478.
41. Lewald, J., and Guski, R. (2004). Auditory- visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neurosci. Lett. 357, 119-122.
42. Ma, W. J., and Pouget, A. (2008). Linking neurons to behavior in multisensory perception: a computational review. Brain Res. 1242, 4-12.
43. Maloney, L. T., and Mamassian, P. (2009). Bayesian decision theory as a model of human visual perception: testing Bayesian transfer. Vis. Neurosci. 26, 147-155.
44. Meredith, M. A. (2002). On the neuronal basis for multisensory convergence: a brief overview. Brain Res. Cogn. Brain Res. 14, 31-40.
45. Meredith, M. A., Nemitz, J. W., and Stein, B. E. (1987). Determinants of mul-tisensory integration in superior col-liculus neurons. I. Temporal factors. J. Neurosci. 7, 3215-3229.
46. Miller, J., and Schwarz, W. (2006). Dissociations between reaction times and temporal order judgments: a diffusion model approach. J. Exp. Psychol. Hum. Percept. Perform. 32, 394-412.
47. Morein-Zamir, S., Soto-Faraco, S., and Kingstone, A. (2003). Auditory capture of vision: examining temporal ventriloquism. Brain Res. Cogn. Brain Res. 17, 154-163.
48. Musacchia, G., and Schroeder, C. E. (2009). Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear. Res. 258, 72-79.
49. Navarra, J., Vatakis, A., Zampini, M., Soto-Faraco, S., Humphreys, W., and Spence, C. (2005). Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Brain Res. Cogn. Brain Res. 25, 499-507
50. Neumann, O., and Niepel, M. (2004). "Timing of perception and perception of time," in Psychophysics Beyond Sensation: Laws and Invariants of Human Cognition, eds C. Kaernbach, E. Schroeger and H. Mueller, (Mahwah, NJ: Erlbaum), 245-269.
51. Powers A. R. III, Hillock, A. R., and Wallace, M. T. (2009). Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29, 12265-12274.
52. Recanzone, G. H. (2003). Auditory infuences on visual temporal rate perception. J. Neurophysiol. 89, 1078-1093.
53. Roach, N. W., Heron, J., and McGraw, P. V. (2006). Resolving multisensory confict: a strategy for balancing the costs and benefits of audio-visual integration. Proc. R. Soc. Lond., B, Biol. Sci. 273, 2159-2168.
54. Romei, V., Murray, M. M., Merabet, L. B., and Thut, G. (2007). Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J. Neurosci. 7, 11465-11472.
55. Roseboom, W., Nishida, S., and Arnold, D. H. (2009). The sliding window of audio-visual simultaneity. J. Vision 9, 1-8.
56. Rowland, B. A., and Stein, B. E. (2007). Multisensory integration produces an initial response enhancement. Front. Integr. Neurosci. 1, 1-8.
57. Rowland, B. A., and Stein, B. E. (2008). Temporal profiles of response enhancement in multisensory integration. Front. Neurosci. 2, 218-224.
58. Rowland, B. A., Quessy, S., Stanford, T. R., and Stein, B. E. (2007). Multisensory integration shortens physiological response latencies. J. Neurosci. 27, 5879-5884.
59. Royal, D. W., Carriere, B. N., and Wallace, M. T. (2009). Spatiotemporal architecture of cortical receptive felds and its impact on multisensory interactions. Exp. Brain Res. 198, 127-136.
60. Sato, Y., Toyoizumi, T., and Aihara, K. (2007). Bayesian inference explains perception of unity and ventriloquism aftereffect: identifcation of common sources of audiovisual stimuli. Neural. Comput. 19, 3335-3355.
61. Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S., and Puce, A. (2008). Neuronal oscillations and visual amplifcation of speech. Trends Cogn. Sci. 12, 106-113.
62. Senkowski, D., Schneider, T. R., Foxe, J. J., and Engel, A. K. (2008). Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci. 31, 401-409.
63. Shams, L., Ma, W. J., and Beierholm, U. (2005). Sound-induced fash illusion as an optimal percept. Neuroreport 16, 1923-1927.
64. Spence, C., and Squire, S. (2003). Multisensory integration: maintaining the perception of synchrony. Curr. Biol. 13, R519-R521.
65. Stein, B. E., and Meredith, M. A. (1993). The Merging of the Senses. Cambridge MA: The MIT Press.
66. Sternberg, S., and Knoll, R. L. (1973). "The perception of temporal order: fundamental issues and a general model," in Attention and Performance, Vol. IV, ed. S. Kornblum (New York: Academic Press), 629-685.
67. Tappe, T., Niepel, M., and Neumann, O. (1994). A dissociation between reaction time to sinusoidal gratings and temporal-order judgment. Perception 23, 335-347.
68. van Attefeldt, N. M., Formisano, E., Blomert, L., and Goebel, R. (2007). The effect of temporal asynchrony on the multisen-sory integration of letters and speech sounds. Cereb. Cortex 17, 962-974.
69. Van Wanrooij, M. M., Bell, A. H., Munoz, D. P., and Van Opstal, A. J. (2009). The effect of spatial-temporal audiovisual disparities on saccades in a complex scene. Exp. Brain Res. 198, 425-437.
70. van Wassenhove, V., Grant, K. W., and Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45, 598-607.
71. Vroomen, J., Keetels, M., de Gelder, B., and Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Brain Res. Cogn. Brain Res. 22, 32-35.
72. Wallace, M. T., Roberson, G. E., Hairston, W. D., Stein, B. E., Vaughan, J. W., and Schirillo, J. A. (2004). Unifying multi-sensory signals across time and space. Exp. Brain Res. 158, 252-258.
73. Whitchurch, E. A., and Takahashi, T. T. (2006). Combined auditory and visual stimuli facilitate head saccades in the barn owl (Tyto alba). J. Neurophysiol. 96, 730-745.
74. Zampini, M., Shore D. I., and Spence, C. (2003). Audiovisual temporal order judgments. Exp. Brain Res. 152, 198-210.