Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance

Experimental Brain Research

Volumn 219, Issue 1, 2012, Pages 121-137

  Stevenson, Ryan A ^a   Fister, Juliane Krueger ^a,b   Barnett, Zachary P ^a   Nidiffer, Aaron R ^a   Wallace, Mark T ^a,b,c  

^a VANDERBILT UNIVERSITY MEDICAL CENTER   (United States)

^b VANDERBILT UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c VANDERBILT UNIVERSITY   (United States)

Author keywords

Audiovisual; Inverse effectiveness; Multisensory; Race model; Response time

Indexed keywords

ADULT; ARTICLE; AUDITORY NERVOUS SYSTEM; AUDITORY STIMULATION; FEMALE; HUMAN; HUMAN EXPERIMENT; MALE; NORMAL HUMAN; PRIORITY JOURNAL; RESPONSE TIME; SENSORY INTEGRATION; SENSORY STIMULATION; UNDERGRADUATE STUDENT; VISUAL NERVOUS SYSTEM; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ADOLESCENT; AUDITORY PERCEPTION; FEMALE; HUMANS; JUDGMENT; MALE; PHOTIC STIMULATION; PSYCHOPHYSICS; REACTION TIME; SIGNAL DETECTION, PSYCHOLOGICAL; SPACE PERCEPTION; TIME FACTORS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84860348145     PISSN: 00144819     EISSN: 14321106     Source Type: Journal    
DOI: 10.1007/s00221-012-3072-1     Document Type: Article

References (68)

1. Blauert J (1997) Spatial hearing: the psychophysics of human sound localization. MIT Press, Cambridge
2. BN Carriere DW Royal MT Wallace 2008 Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions J Neurophysiol 99 5 2357 2368 10.1152/jn.01386.2007 18287544 10.1152/jn.01386.2007
3. F Colavita 1974 Human sensory dominance Atten Percept Psychophys 16 2 409 412 10.3758/bf03203962
4. H Colonius A Diederich 2006 The race model inequality: interpreting a geometric measure of the amount of violation Psychol Rev 113 1 148 154 10.1037/0033-295X.113.1.148 16478305 10.1037/0033-295X.113.1.148 (Pubitemid 43283714)
5. Conrey BL, Pisoni DB (2004) Detection of auditory-visual asynchrony in speech and nonspeech signals. Research on Spoken Language Processing, vol 26. Indiana University, Bloomington
6. B Conrey DB Pisoni 2006 Auditory-visual speech perception and synchrony detection for speech and nonspeech signals J Acoust Soc Am 119 6 4065 4073 16838548 10.1121/1.2195091 (Pubitemid 43886378)
7. DP Corey AJ Hudspeth 1979 Response latency of vertebrate hair cells Biophys J 26 3 499 506 10.1016/S0006-3495(79)85267-4 318064 10.1016/S0006- 3495(79)85267-4 1:STN:280:DyaL3M3ls1yqsg%3D%3D (Pubitemid 9205704)
8. A Diederich H Colonius 2004 Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time Percept Psychophys 66 8 1388 1404 15813202 10.3758/BF03195006 (Pubitemid 40778227)
9. A Diederich H Colonius 2008 When a high-intensity "distractor" is better then a low-intensity one: modeling the effect of an auditory or tactile nontarget stimulus on visual saccadic reaction time Brain Res 1242 219 230 10.1016/j.brainres.2008.05.081 18573240 10.1016/j.brainres.2008.05.081 1:CAS:528:DC%2BD1cXhtlyhsb7P
10. NF Dixon L Spitz 1980 The detection of auditory visual desynchrony Perception 9 6 719 721 7220244 10.1068/p090719 1:STN:280:DyaL3M7nsVGltQ%3D%3D (Pubitemid 11114821)
11. JH Foss-Feig LD Kwakye CJ Cascio CP Burnette H Kadivar WL Stone MT Wallace 2010 An extended multisensory temporal binding window in autism spectrum disorders Exp Brain Res 203 2 381 389 10.1007/s00221-010-2240-4 20390256 10.1007/s00221-010-2240-4
12. WD Hairston PJ Laurienti G Mishra JH Burdette MT Wallace 2003 Multisensory enhancement of localization under conditions of induced myopia Exp Brain Res 152 3 404 408 14504674 10.1007/s00221-003-1646-7 (Pubitemid 37204605)
13. D Hecht M Reiner A Karni 2008 Multisensory enhancement: gains in choice and in simple response times Exp Brain Res 189 2 133 143 10.1007/s00221-008- 1410-0 18478210 10.1007/s00221-008-1410-0
14. M Hershenson 1962 Reaction time as a measure of intersensory facilitation J Exp Psychol 63 289 293 13906889 10.1037/h0039516 1:STN:280: DyaF38%2FkvVaksA%3D%3D
15. AR Hillock AR Powers MT Wallace 2011 Binding of sights and sounds: age-related changes in multisensory temporal processing Neuropsychologia 49 3 461 467 10.1016/j.neuropsychologia.2010.11.041 21134385 10.1016/j. neuropsychologia.2010.11.041
16. IJ Hirsh CE Sherrick Jr 1961 Perceived order in different sense modalities J Exp Psychol 62 423 432 13907740 10.1037/h0045283 1:STN:280:DyaF38%2FkvFWhtw%3D%3D
17. M Keetels J Vroomen 2005 The role of spatial disparity and hemifields in audio-visual temporal order judgments Exp Brain Res 167 4 635 640 10.1007/s00221-005-0067-1 16175363 10.1007/s00221-005-0067-1
18. S Kim TW James 2010 Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience Hum Brain Mapp 31 5 678 693 10.1002/hbm.20897 19830683 10.1002/hbm.20897 1:CAS:528: DC%2BC3cXktFKlur8%3D
19. Kim S, Stevenson RA, James TW (in press) Visuo-haptic neuronal convergence demonstrated with an inversely effective pattern of BOLD activation. J Cognit Neurosci. doi: 10.1162/jocn-a-00176
20. AJ King AR Palmer 1985 Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus Exp Brain Res 60 3 492 500 4076371 10.1007/BF00236934 1:STN:280:DyaL28%2Fot1Wrsw%3D%3D (Pubitemid 16191793)
21. J Krueger DW Royal MC Fister MT Wallace 2009 Spatial receptive field organization of multisensory neurons and its impact on multisensory interactions Hear Res 258 1-2 47 54 10.1016/j.heares.2009.08.003 19698773 10.1016/j.heares.2009.08.003
22. TD Lamb EN Pugh Jr 1992 A quantitative account of the activation steps involved in phototransduction in amphibian photoreceptors J Physiol 449 719 758 1326052 1:CAS:528:DyaK38XhslSlurs%3D
23. P Lennie 1981 The physiological basis of variations in visual latency Vision Res 21 6 815 824 7314459 10.1016/0042-6989(81)90180-2 1:STN:280:DyaL38%2Fos1WksA%3D%3D (Pubitemid 11066973)
24. J Lewald R Guski 2003 Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli Brain Res Cogn Brain Res 16 3 468 478 12706226 10.1016/S0926-6410(03)00074-0 (Pubitemid 36426932)
25. J Lewald R Guski 2004 Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception Neurosci Lett 357 2 119 122 10.1016/j.neulet.2003.12.045 15036589 10.1016/j.neulet.2003.12.045 1:CAS:528:DC%2BD2cXhsVKhtrY%3D (Pubitemid 38230945)
26. CT Lovelace BE Stein MT Wallace 2003 An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection Brain Res Cogn Brain Res 17 2 447 453 12880914 10.1016/S0926-6410(03)00160-5 (Pubitemid 36860069)
27. E Macaluso N George R Dolan C Spence J Driver 2004 Spatial and temporal factors during processing of audiovisual speech: a PET study NeuroImage 21 2 725 732 14980575 10.1016/j.neuroimage.2003.09.049 1:STN:280: DC%2BD2c%2FptFOitg%3D%3D (Pubitemid 38229935)
28. Matin L (1986) Visual localization and eye movements. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance, vol 1. Wiley-Interscience, New York, pp 20.21-20.45
29. H McGurk J MacDonald 1976 Hearing lips and seeing voices Nature 264 5588 746 748 1012311 10.1038/264746a0 1:STN:280:DyaE2s7gtVamuw%3D%3D
30. MA Meredith BE Stein 1986 Spatial factors determine the activity of multisensory neurons in cat superior colliculus Brain Res 365 2 350 354 3947999 10.1016/0006-8993(86)91648-3 1:STN:280:DyaL287jt1Ogsw%3D%3D (Pubitemid 16120784)
31. MA Meredith BE Stein 1986 Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration J Neurophysiol 56 3 640 662 3537225 1:STN:280:DyaL2s%2FlvF2rtw%3D%3D (Pubitemid 16031523)
32. MA Meredith JW Nemitz BE Stein 1987 Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors J Neurosci 7 10 3215 3229 3668625 1:STN:280:DyaL1c%2FjsFWltg%3D%3D
33. MA Meredith MT Wallace BE Stein 1992 Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection Exp Brain Res 88 1 181 186 1541354 10.1007/BF02259139 1:STN:280:DyaK387nsFWjtw%3D%3D
34. J Miller 1982 Divided attention: evidence for coactivation with redundant signals Cogn Psychol 14 2 247 279 7083803 10.1016/0010-0285(82)90010-X 1:STN:280:DyaL383gvFGqtA%3D%3D
35. LM Miller M D'Esposito 2005 Perceptual fusion and stimulus coincidence in the cross-modal integration of speech J Neurosci 25 25 5884 5893 15976077 10.1523/JNEUROSCI.0896-05.2005 1:CAS:528:DC%2BD2MXlslGisLk%3D (Pubitemid 40911419)
36. WT Nelson LJ Hettinger JA Cunningham BJ Brickman MW Haas RL McKinley 1998 Effects of localized auditory information on visual target detection performance using a helmet-mounted display Hum Factors 40 3 452 460 9849103 10.1518/001872098779591304 1:STN:280:DyaK1M%2FmsF2ltw%3D%3D (Pubitemid 28551223)
37. B Nordlund 1962 Physical factors in angular localization Acta Otolaryngol 54 75 93 14480310 10.3109/00016486209126924 1:STN:280:DyaF38%2FotFyiug%3D%3D
38. E Pöppel K Schill N von Steinbüchel 1990 Sensory integration within temporally neutral systems states: a hypothesis Naturwissenschaften 77 89 91 2314478 10.1007/BF01131783 (Pubitemid 20127671)
39. AR Powers III AR Hillock MT Wallace 2009 Perceptual training narrows the temporal window of multisensory binding J Neurosci 29 39 12265 12274 10.1523/JNEUROSCI.3501-09.2009 19793985 10.1523/JNEUROSCI.3501-09.2009 1:CAS:528:DC%2BD1MXht12qu73F
40. DH Raab 1962 Statistical facilitation of simple reaction times Trans NY Acad Sci 24 574 590 1:STN:280:DyaF38%2FptFSrtQ%3D%3D
41. NW Roach J Heron D Whitaker PV McGraw 2011 Asynchrony adaption reveals neural population code for audio-visual timing Proc R Soc 278 9 10.1098/rspb.2010.1267
42. LA Ross D Saint-Amour VM Leavitt DC Javitt JJ Foxe 2007 Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments Cereb Cortex 17 5 1147 1153 10.1093/cercor/bhl024 16785256 10.1093/cercor/bhl024 (Pubitemid 46598855)
43. LA Ross D Saint-Amour VM Leavitt S Molholm DC Javitt JJ Foxe 2007 Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions Schizophr Res 97 1-3 173 183 10.1016/j.schres.2007.08.008 17928202 10.1016/j.schres.2007.08.008 (Pubitemid 350088394)
44. DW Royal BN Carriere MT Wallace 2009 Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions Exp Brain Res 198 2-3 127 136 10.1007/s00221-009-1772-y 19308362 10.1007/s00221-009-1772-y
45. S Schall C Quigley S Onat P Konig 2009 Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli Exp Brain Res 198 2-3 137 151 10.1007/s00221-009-1867-5 19526359 10.1007/s00221-009-1867-5
46. AR Seitz Sr JE Nanez SR Holloway T Watanabe 2006 Perceptual learning of motion leads to faster flicker perception PLoS ONE 1 1 e28 10.1371/journal.pone. 0000028 17183655 10.1371/journal.pone.0000028
47. D Senkowski D Talsma M Grigutsch CS Herrmann MG Woldorff 2007 Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations Neuropsychologia 45 3 561 571 16542688 10.1016/j.neuropsychologia.2006.01.013 (Pubitemid 44894455)
48. L Shams Y Kamitani S Shimojo 2000 Illusions. What you see is what you hear Nature 408 6814 788 11130706 10.1038/35048669 1:CAS:528: DC%2BD3MXhtlWqtA%3D%3D (Pubitemid 32016079)
49. BE Stein MT Wallace 1996 Comparisons of cross-modality integration in midbrain and cortex Prog Brain Res 112 289 299 8979836 10.1016/S0079-6123(08) 63336-1 1:STN:280:DyaK2s7jsVClsw%3D%3D (Pubitemid 26400858)
50. BE Stein WS Huneycutt MA Meredith 1988 Neurons and behavior: the same rules of multisensory integration apply Brain Res 448 2 355 358 3378157 10.1016/0006-8993(88)91276-0 1:STN:280:DyaL1c3ktFeltA%3D%3D
51. RA Stevenson TW James 2009 Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition NeuroImage 44 3 1210 1223 10.1016/j.neuroimage.2008.09.034 18973818 10.1016/j.neuroimage.2008.09.034
52. RA Stevenson ML Geoghegan TW James 2007 Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects Exp Brain Res 179 1 85 95 17109108 10.1007/s00221-006-0770-6 (Pubitemid 46659222)
53. RA Stevenson S Kim TW James 2009 An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI Exp Brain Res 198 2-3 183 194 10.1007/s00221-009-1783-8 19352638 10.1007/s00221-009-1783-8
54. RA Stevenson NA Altieri S Kim DB Pisoni TW James 2010 Neural processing of asynchronous audiovisual speech perception NeuroImage 49 4 3308 3318 10.1016/j.neuroimage.2009.12.001 20004723 10.1016/j.neuroimage.2009.12.001
55. RA Stevenson RM VanDerKlok DB Pisoni TW James 2011 Discrete neural substrates underlie complementary audiovisual speech integration processes NeuroImage 55 3 1339 1345 10.1016/j.neuroimage.2010.12.063 21195198 10.1016/j.neuroimage.2010.12.063
56. Y Sugita Y Suzuki 2003 Audiovisual perception: implicit estimation of sound-arrival time Nature 421 6926 911 10.1038/421911a 12606990 10.1038/421911a 1:CAS:528:DC%2BD3sXhsVKgt7Y%3D (Pubitemid 36288014)
57. D Talsma D Senkowski MG Woldorff 2009 Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli Exp Brain Res 198 2-3 313 328 10.1007/s00221-009-1858-6 19495733 10.1007/s00221-009-1858-6
58. NM van Atteveldt E Formisano L Blomert R Goebel 2007 The effect of temporal asynchrony on the multisensory integration of letters and speech sounds Cereb Cortex 17 4 962 974 10.1093/cercor/bhl007 16751298 10.1093/cercor/bhl007 (Pubitemid 46434637)
59. V van Wassenhove KW Grant D Poeppel 2007 Temporal window of integration in auditory-visual speech perception Neuropsychologia 45 3 598 607 10.1016/j.neuropsychologia.2006.01.001 16530232 10.1016/j.neuropsychologia.2006. 01.001 (Pubitemid 44894453)
60. A Vatakis C Spence 2006 Audiovisual synchrony perception for music, speech, and object actions Brain Res 1111 1 134 142 10.1016/j.brainres.2006.05. 078 16876772 10.1016/j.brainres.2006.05.078 1:CAS:528:DC%2BD28XpsVWqsrg%3D (Pubitemid 44331172)
61. J Vroomen M Keetels 2010 Perception of intersensory synchrony: a tutorial review Atten Percept Psychophys 72 4 871 884 10.3758/APP.72.4.871 20436185 10.3758/APP.72.4.871
62. Wallace MH, Murray MM (eds) (2011) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London
63. MT Wallace MA Meredith BE Stein 1992 Integration of multiple sensory modalities in cat cortex Exp Brain Res 91 3 484 488 1483520 10.1007/BF00227844 1:STN:280:DyaK3s7jtFSjsQ%3D%3D
64. MT Wallace GE Roberson WD Hairston BE Stein JW Vaughan JA Schirillo 2004 Unifying multisensory signals across time and space Exp Brain Res 158 2 252 258 10.1007/s00221-004-1899-9 15112119 10.1007/s00221-004-1899-9 1:STN:280:DC%2BD2cvgsVGgtg%3D%3D (Pubitemid 39281121)
65. LK Wilkinson MA Meredith BE Stein 1996 The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior Exp Brain Res 112 1 1 10 8951401 10.1007/BF00227172 1:STN:280:DyaK2s7gvFWhsw%3D%3D (Pubitemid 26388581)
66. M Zampini DI Shore C Spence 2003 Audiovisual temporal order judgments Exp Brain Res 152 2 198 210 10.1007/s00221-003-1536-z 12879178 10.1007/s00221-003- 1536-z (Pubitemid 37153641)
67. M Zampini S Guest DI Shore C Spence 2005 Audio-visual simultaneity judgments Percept Psychophys 67 3 531 544 16119399 10.3758/BF03193329 (Pubitemid 40979066)
68. M Zampini DI Shore C Spence 2005 Audiovisual prior entry Neurosci Lett 381 3 217 222 10.1016/j.neulet.2005.01.085 15896473 10.1016/j.neulet.2005.01.085 1:CAS:528:DC%2BD2MXkt1Ggs7s%3D (Pubitemid 40719160)