Temporal window of integration in auditory-visual speech perception

Neuropsychologia

Volumn 45, Issue 3, 2007, Pages 598-607

  van Wassenhove, Virginie ^a,b   Grant, Ken W ^c   Poeppel, David ^d  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^c WALTER REED ARMY MEDICAL CENTER   (United States)

^d Bldg 142   (United States)

Author keywords

Analysis by synthesis; McGurk illusion; Multisensory; Psychophysics; Time

Indexed keywords

ADULT; ARTICLE; AUDIO RECORDING; AUDITORY RESPONSE; AUDITORY VISUAL SPEECH PERCEPTION; FEMALE; HEARING; HUMAN; HUMAN EXPERIMENT; ILLUSION; LIP READING; MALE; MCGURK ILLUSION; NORMAL HUMAN; SENSORY STIMULATION; SENSORY SYSTEM; SENSORY SYSTEM EXAMINATION; SPEECH PERCEPTION; STIMULUS RESPONSE; TASK PERFORMANCE; VIDEORECORDING; VISION;

ACOUSTIC STIMULATION; ADULT; AUDITORY PERCEPTION; FEMALE; FUNCTIONAL LATERALITY; HUMANS; JUDGMENT; MALE; PHOTIC STIMULATION; SPEECH; TIME PERCEPTION; VISION; VISUAL PERCEPTION;

EID: 33845468167     PISSN: 00283932     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2006.01.001     Document Type: Article

References (59)

1. Arai, T., & Greenberg, S. (1997). The temporal properties of spoken Japanese are similar to those of English. In Proceedings of Eurospeech (pp. 1011-1014).
2. Arnold D.H., Johnston A., and Nishida S. Timing sight and sound. Vision Research 45 (2005) 1275-1284
3. Boemio A., Fromm S., Braun A., and Poeppel D. Hierarchical and asymmetric temporal sensitivity in human auditory cortices. Nature Neuroscience 8 3 (2005) 389-395
4. Buchner H., Gobbelé R., Wagner M., Fuchs M., Waberski T.D., and Beckmann R. Fast visual evoked potential input in to human area V5. Neuroreport 8 (1997) 2419-2422
5. Calvert G.A., Campbell R., and Brammer M. Evidence from functional magnetic resonance imaging of Crossmodal binding in the human heteromodal cortex. Current Biology 10 (2000) 649-657
6. Campbell C.S., and Massaro D.W. Perception of visible speech: influence of spatial quantization. Perception 26 5 (1997) 627-644
7. Celesia G.G. Organization of auditory cortical areas in man. Brain 99 (1976) 403-414
8. Conrey B.L., and Pisoni D.B. Audiovisual asynchrony detection for speech and nonspeech signals. Proceedings of the audio-visual speech processing workshop (AVSP-2003). St.-Jorioz, France (2003) 25-30
9. Dixon N.F., and Spitz L. The detection of auditory visual desynchrony. Perception 9 (1980) 719-721
10. Engel G.R., and Ehrenstein W.H. Visual-auditory distance constancy. Nature 234 (1971) 308-308
11. Ffytche D.H., Guy C.N., and Zeki S. The parallel visual motion inputs into areas V1 and V5 of human cerebral cortex. Brain 118 (1995) 1375-1394
12. Fujisaki W., Shimojo S., Kashino M., and Nishida S. Recalibration of audio-visual simultaneity. Nature Neuroscience 7 7 (2004) 773-778
13. Grant K.W., and Greenberg S. Speech intelligibility derived from asynchronous processing of auditory-visual information. Proceedings of the audio-visual speech processing workshop (AVSP-2001). Aalborg, Denmark (2001) 132-137
14. Grant K., Walden B.E., and Seitz P.F. Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration. Journal of the Acoustical Society of America 103 (1998) 2677-2690
15. Greenberg, S. (1996). Understanding speech understanding: towards a unified theory of speech perception. In Proceedings of the ESCA workshop on the "Auditory Basis of Speech Perception", Keele University, United Kingdom, 1-8.
16. Greenberg S. A multi-tier theoretical framework for understanding spoken language. In: Greenberg S., and Ainsworth W.A. (Eds). Listening to speech: an auditory perspective (2005), Lawrence Erlbaum Associates, Mahwah, NJ 411-433
17. Helfer K.S. Auditory and auditory-visual perception of clear and conversational speech. Journal of Speech, Language and Hearing Research 40 (1997) 432-443
18. Hirsh I.J., and Sherrick Jr. C.E. Perceived order in different sense modalities. Journal of Experimental Psychology 62 (1961) 423-432
19. Howard III M.A., Volkov I.O., Abbas P.J., Damasio H., Ollendieck M.C., and Granner M.A. A chronic microelectrode investigation of the tonotopic organization of human auditory cortex. Brain Research 724 (1996) 260-264
20. Jones J.A., and Munhall K. The effects of separating auditory and visual sources on audiovisual integration of speech. Canadian Acoustics 25 (1997) 13-19
21. Jordan T.R., McCotter M.V., and Thomas S.M. Visual and audiovisual speech perception with color and gray-scale facial images. Perception and Psychophysics 62 7 (2000) 1394-1404
22. Kopinska A., and Harris L.R. Simultaneity constancy. Perception 33 (2004) 1049-1060
23. Lakatos P., Pincze Z., Fu K.M., Javitt D.C., Karmos G., and Schroeder C.E. Timing of pure tone and noise-evoked responses in macaque auditory cortex. Neuroreport 16 9 (2005) 933-937
24. Lewald J., and Guski R. Auditory-visual integration as a function of distance: no compensation for sound-transmission time in human perception. Neuroscience Letters 357 (2004) 119-122
25. Liberman A.M., and Whalen D.H. On the relation of speech to language. Trends in Cognitive Sciences 4 (2000) 187-196
26. Liégeois-Chauvel C., Musolino A., and Chauvel P. Localization of the primary auditory cortex area in man. Brain 114 (1991) 139-153
27. Loveless N., Levänen S., Jousmäki V., Sams M., and Hari R. Temporal integration in auditory sensory memory: neuromagnetic evidence. Electroencephalography and Clinical Neurophysiology 100 (1996) 220-228
28. MacDonald J., Andersen S., and Bachmann T. Hearing by eye: how much spatial degradation can be tolerated?. Perception 29 10 (2000) 1155-1168
29. MacLeod A., and Summerfield Q. A procedure for measuring auditory and audio-visual speech-reception thresholds for sentences in noise: rationals, evaluation, and recommendations for use. British Journal of Audiology 24 (1990) 29-43
30. Massaro D.W., Cohen M.M., and Smeele P.M. Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America 100 (1996) 1777-1786
31. McGrath M., and Summerfield Q. Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America 77 (1985) 678-684
32. McGurk H., and McDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-747
33. Meredith M.A., Nemitz J.W., and Stein B.E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. The Journal of Neuroscience 7 100 (1987) 3215-3229
34. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13 (2002) 417-425
35. Munhall K., Gribble P., Sacco L., and Ward M. Temporal constraints on the McGurk effect. Perception and Psychophysics 58 (1996) 351-362
36. Navarra J., Vatakis A., Zampini M., Soto-Faraco S., Humphreys W., and Charles Spence C. Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Research 25 (2005) 499-507
37. Näätänen R. Attention and brain function (1992), Lawrence Erlbaum Associates, Hillsdale, NJ
38. Pandey P.C., Kunov H., and Abel S.M. Disruptive effects of auditory signal delay on speech perception with lipreading. The Journal of Auditory Research 26 (1986) 27-41
39. Poeppel D. The analysis of speech in different temporal integration windows: cerebral lateralization as 'asymmetric sampling in time'. Speech Communication 41 (2003) 245-255
40. Rosen S. Temporal information in speech: acoustic, auditory, and linguistic aspects. Philosophical Transactions of the Royal Society of London B 336 (1992) 367-373
41. Rosenblum L.D., and Saldaña H.M. An audiovisual test of kinematic primitives for visual speech perception,. Journal of Experimental Psychology, Human Perception and Performance 22 2 (1996) 318-331
42. Schroeder C.E., Mehta A.D., and Givre S.J. A spatiotemporal profile of visual system activation revealed by current source density analysis in the awake macaque. Cerebral Cortex 8 7 (1998) 575-592
43. Sams M., and Aulanko R. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127 (1991) 141-147
44. Stein B.E., and Meredith A.M. The merging of the senses (1993), MIT Press, Cambridge, MA
45. Steinschneider M., Schroeder C.E., Arezzo J.C., and Vaughan Jr. H.G. Speech-evoked activity in primary auditory cortex: effects of voice onset time. Electroencephalography and Clinical Neurophysiology 92 (1994) 30-43
46. Steinschneider M., Volkov I.O., Noh M.D., Garell P.C., and Howard M.A. Temporal encoding of the voice onset time phonetic parameter by field potentials recorded directly from human auditory cortex. Journal of Neurophysiology 82 (1999) 2346-2357
47. Sugita Y., and Suzuki Y. Implicit estimation of sound-arrival time. Nature 421 (2003) 911
48. Sumby W., and Pollack I. Visual contributions to speech intelligibility in noise. Journal of the Acoustical Society of America 26 (1954) 212-215
49. Summerfield Q. Some preliminaries to a comprehensive account of audio-visual speech perception. In: Dodd B., and Campbell R. (Eds). Hearing by eye: the psychology of lip-reading (1987), Erlbaum, London 3-51
50. Tuomainen J., Andersen T., Tiippana K., and Sams M. Audio-visual speech perception is special. Cognition 96 (2005) B13-B22
51. van Wassenhove, V. (2004). Neural correlates of auditory-visual speech desynchronization. In 'Analysis-by-synthesis' in auditory-visual speech perception: a forward model of multisensory integration (pp. 119-158). PhD Thesis, University of Maryland, College Park.
52. van Wassenhove V., Grant K.W., and Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of United States of America 102 4 (2005) 1181-1186
53. Vroomen J., Keetels M., deGelder B., and Bertelson P. Recalibration of temporal order perception by exposure to audio-visual asynchrony. Brain Research. Cognitive Brain Research 22 1 (2004) 32-35
54. Winkler I., Czigler I., Jaramillo M., Paavilainen P., and Näätänen R. Temporal constraints of auditory event synthesis: evidence from ERPs. Neuroreport 9 (1998) 495-499
55. Yabe H., Koyama S., Kakigi R., Gunji A., Tervaniemi M., Sato Y., et al. Automatic discriminative sensitivity inside temporal window of sensory memory as a function of time. Cognitive Brain Research 12 (2001) 39-48
56. Yabe H., Tervaniemi M., Reinikainen K., and Näätänen R. Temporal window of integration revealed by MMN to sound omission. Neuroreport 8 (1997) 1971-1974
57. Yabe H., Winkler I., Czigler I., Koyama S., Kakigi R., Sutoh T., et al. Organizing sound sequences in the human brain: the interplay of auditory streaming and temporal integration. Brain Research 897 (2001) 222-227
58. Zampini M., Shore D.I., and Spence C. Audiovisual temporal order judgments. Experimental Brain Research 152 (2003) 198-210
59. Zampini, M., Guest, S., Shore, D. I., & Spence, C. (in press). Audiovisual simultaneity judgments.