Neural processing of asynchronous audiovisual speech perception

NeuroImage

Volumn 49, Issue 4, 2010, Pages 3308-3318

  Stevenson, Ryan A ^a   Altieri, Nicholas A ^a   Kim, Sunah ^a   Pisoni, David B ^a   James, Thomas W ^a  

^a INDIANA UNIVERSITY   (United States)

Author keywords

fMRI; Integration; Multisensory; STS; Superior colliculus; Synchrony

Indexed keywords

ADULT; ARTICLE; AUDITORY DISCRIMINATION; AUDITORY STIMULATION; BLOOD OXYGENATION; COGNITION; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEMISPHERIC DOMINANCE; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; LEFT HEMISPHERE; MALE; NORMAL HUMAN; OCCIPITAL CORTEX; PRIORITY JOURNAL; RIGHT HEMISPHERE; SPEECH PERCEPTION; STIMULUS RESPONSE; SUPERIOR COLLICULUS; TEMPORAL CORTEX; VISUAL CORTEX; VISUAL DISCRIMINATION; VISUAL STIMULATION;

ADULT; AUDITORY CORTEX; AUDITORY PERCEPTION; EVOKED POTENTIALS, AUDITORY; EVOKED POTENTIALS, VISUAL; FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; NERVE NET; TEMPORAL LOBE; VISUAL CORTEX; VISUAL PERCEPTION;

EID: 73749083278     PISSN: 10538119     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2009.12.001     Document Type: Article

References (59)

1. Beauchamp M.S. See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr. Opin. Neurobiol. 15 (2005) 145-153
2. Beauchamp M.S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3 (2005) 93-113
3. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., and Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat. Neurosci. 7 (2004) 1190-1192
4. Beauchamp M.S., Lee K.E., Argall B.D., and Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41 (2004) 809-823
5. Bertelson P., and Radeau M. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29 (1981) 578-584
6. Bertelson P., Pavani F., Ladavas E., Vroomen J., and de Gelder B. Ventriloquism in patients with unilateral visual neglect. Neuropsychologia 38 (2000) 1634-1642
7. Bertelson P., Vroomen J., de Gelder B., and Driver J. The ventriloquist effect does not depend on the direction of deliberate visual attention. Percept. Psychophys. 62 (2000) 321-332
8. Brainard D.H. The Psychophysics Toolbox. Spat. Vis. 10 (1997) 433-436
9. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10 (2000) 649-657
10. Conrey B., and Pisoni D.B. Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J. Acoust. Soc. Am. 119 (2006) 4065-4073
11. Conrey B.L., and Pisoni D.B. Detection of auditory-visual asynchrony in speech and nonspeech signals. In: Pisoni D.B. (Ed). Research on Spoken Language Processing (2004), Indiana University, Bloomington 71-94
12. Dixon N.F., and Spitz L. The detection of auditory visual desynchrony. Perception 9 (1980) 719-721
13. Forman S.D., Cohen J.D., Fitzgerald M., Eddy W.F., Mintun M.A., and Noll D.C. Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn. Reson. Med. 33 (1995) 636-647
14. Formisano E., Linden D.E., Di Salle F., Trojano L., Esposito F., Sack A.T., Grossi D., Zanella F.E., and Goebel R. Tracking the mind's image in the brain I: time-resolved fMRI during visuospatial mental imagery. Neuron 35 (2002) 185-194
15. Gaver W.W. What in the world do we hear? An ecological approach to auditory event perception. Ecol. Psychol. 5 (1993) 1-29
16. Georgopoulos A.P., Lurito J.T., Petrides M., Schwartz A.B., and Massey J.T. Mental rotation of the neuronal population vector. Science 243 (1989) 234-236
17. Glover G.H. Deconvolution of impulse response in event-related BOLD fMRI. Neuroimage 9 (1999) 416-429
18. Grant, K.W., Greenberg, S., 2001. Speech intelligibility derived from asynchronous processing of auditory-visual information. International Conference of Auditory-Visual Speech Processing, Santa Cruz, CA, pp. 132-137.
19. Grant K.W., Van Wassenhove V., and Poeppel D. Detection of auditory (cross-spectral) and auditory-visual (cross-modal) synchrony. Speech Commun. 44 (2004) 43-53
20. Grant K.W., Walden B.E., and Seitz P.F. Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration. J. Acoust. Soc. Am. 103 (1998) 2677-2690
21. Green K.P., and Kuhl P.K. The role of visual information in the processing of place and manner features in speech perception. Percept. Psychophys. 45 (1989) 34-42
22. Hamilton R.H., Shenton J.T., and Coslett H.B. An acquired deficit of audiovisual speech processing. Brain Lang. 98 (2006) 66-73
23. James T.W., Stevenson R.A., and Kim S. Assessing multisensory integration with additive factors and functionsl MRI (2009), The International Society for Psychophysics, Dublin, Ireland
24. Kim, S., James, T.W., in press. Enhanced Effectiveness in visuo-haptic object-selective brain regions with increasing stimulus saliency. Hum. Brain Mapp. doi:10.1002/hbm.20897.
25. King A.J., and Palmer A.R. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60 (1985) 492-500
26. Lachs L., and Hernandez L.R. Update: The Hoosier Audiovisual Multitalker Database. In: Pisoni D.B. (Ed). Research on spoken language processing (1998), Speech Research Laboratory, Indiana University, Bloomington, IN 377-388
27. Laurienti P.J., Kraft R.A., Maldjian J.A., Burdette J.H., and Wallace M.T. Semantic congruence is a critical factor in multisensory behavioral performance. Exp. Brain Res. 158 (2004) 405-414
28. Luce P.A., and Pisoni D.B. Recognizing spoken words: the neighborhood activation model. Ear Hear. 19 (1998) 1-36
29. Macaluso E., George N., Dolan R., Spence C., and Driver J. Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21 (2004) 725-732
30. McGrath M., and Summerfield Q. Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. J. Acoust. Soc. Am. 77 (1985) 678-685
31. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
32. Meredith M.A., Nemitz J.W., and Stein B.E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7 (1987) 3215-3229
33. Miller L.M., and D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 25 (2005) 5884-5893
34. Munhall K.G., Gribble P., Sacco L., and Ward M. Temporal constraints on the McGurk effect. Percept. Psychophys. 58 (1996) 351-362
35. Noesselt T., Rieger J.W., Schoenfeld M.A., Kanowski M., Hinrichs H., Heinze H.J., and Driver J. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J. Neurosci. 27 (2007) 11431-11441
36. Pelli D.G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10 (1997) 437-442
37. Puce A., Allison T., Bentin S., Gore J.C., and McCarthy G. Temporal cortex activation in humans viewing eye and mouth movements. J. Neurosci. 18 (1998) 2188-2199
38. Richter W., Somorjai R., Summers R., Jarmasz M., Menon R.S., Gati J.S., Georgopoulos A.P., Tegeler C., Ugurbil K., and Kim S.G. Motor area activity during mental rotation studied by time-resolved single-trial fMRI. J. Cogn. Neurosci. 12 (2000) 310-320
39. Ross L.A., Saint-Amour D., Leavitt V.M., Javitt D.C., and Foxe J.J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb. Cortex 17 (2007) 1147-1153
40. Saxe R., Brett M., and Kanwisher N. Divide and conquer: a defense of functional localizers. Neuroimage 30 (2006) 1088-1096 discussion. 1097-1089
41. Scott S.K., Blank C.C., Rosen S., and Wise R.J. Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123 Pt. 12 (2000) 2400-2406
42. Sheffert S.M., Lachs L., and Hernandez L.R. The Hooiser Audiovisual Multitalker Database. In: Pisoni D.B. (Ed). Research on spoken language processing (1996), Speach Research Laboratory, Indiana University, Bloomington, IN 578-583
43. Soto-Faraco S., and Alsius A. Deconstructing the McGurk-MacDonald illusion. J. Exp. Psychol. Hum. Percept. Perform. 35 (2009) 580-587
44. Stein B.E., Meredith M.A., and Wallace M.T. The visually responsive neuron and beyond: multisensory integration in cat and monkey. Prog. Brain Res. 95 (1993) 79-90
45. Steinmetz R. Human perception of jitter and media synchronization. IEEE J. Sel. Areas Commun. 14 (1996) 61-72
46. Stekelenburg J.J., Vroomen J., and de Gelder B. Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neurosci. Lett. 357 (2004) 163-166
47. Stevenson R.A., and James T.W. Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44 (2009) 1210-1223
48. Stevenson R.A., Geoghegan M.L., and James T.W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp. Brain Res. 179 (2007) 85-95
49. Stevenson R.A., Kim S., and James T.W. An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp. Brain Res. 198 (2009) 183-194
50. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26 (1954) 212-215
51. Summerfield Q. Some Preliminaries to a Comprehensive Account of Audio-visual Speech Perception. In: Dodd B., and Campbell B.A. (Eds). Hearing by Eye: The Psychology of Lip Reading (1987), Lawrence Erlbaum Associates Ltd., Publishers, London, UK 3-52
52. Talaraich J., and Tournoux P. Co-planar stereotaxic atlas of the human brain (1988), Theime Medical Publishers, New York, New York
53. Thirion B., Pinel P., Meriaux S., Roche A., Dehaene S., and Poline J.B. Analysis of a large fMRI cohort: statistical and methodological issues for group analyses. Neuroimage 35 (2007) 105-120
54. van Atteveldt N.M., Formisano E., Blomert L., and Goebel R. The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb. Cortex 17 (2007) 962-974
55. van Wassenhove V., Grant K.W., and Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. U. S. A. 102 (2005) 1181-1186
56. van Wassenhove V., Grant K.W., and Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45 (2007) 598-607
57. Vroomen J., Bertelson P., and de Gelder B. The ventriloquist effect does not depend on the direction of automatic visual attention. Percept. Psychophys. 63 (2001) 651-659
58. Werner, S., Noppeney, U., in press. Superadditive responses in the superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral. Cortex. doi:10.1093/cercor/bhp248.
59. Wheaton K.J., Thompson J.C., Syngeniotis A., Abbott D.F., and Puce A. Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex. Neuroimage 22 (2004) 277-288