Psychophysics of the McGurk and Other Audiovisual Speech Integration Effects

Journal of Experimental Psychology: Human Perception and Performance

Volumn 37, Issue 4, 2011, Pages 1193-1209

  Jiang, Jintao ^a   Bernstein, Lynne E ^a  

^a HOUSE EAR INSTITUTE   (United States)

Author keywords

Audiovisual speech perception; Congruent and incongruent; Factor analysis; Quantitative stimulus measures

Indexed keywords

ARTICLE; AUDITORY STIMULATION; HUMAN; LANGUAGE; PERCEPTIVE DISCRIMINATION; PHOTOSTIMULATION; PSYCHOPHYSICS; REFERENCE VALUE; SPEECH PERCEPTION; VISION;

ACOUSTIC STIMULATION; DISCRIMINATION (PSYCHOLOGY); HUMANS; LANGUAGE; PHOTIC STIMULATION; PSYCHOPHYSICS; REFERENCE VALUES; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 79961150200     PISSN: 00961523     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0023100     Document Type: Article

References (66)

1. Bernstein, L. E. (in press-a). Multisensory information integration for communication and speech. In B. E. Stein (Ed.), The new handbook of multisensory integration. Cambridge, MA: MIT Press.
2. Bernstein, L. E. (in press-b). Visual speech perception. In G. Bailly, P. Perrier, & E. Vatiokis-Bateson (Eds.), Audiovisual speech processing. Cambridge, England: Cambridge University Press.
3. Bernstein L.E., Auer E.T., Chaney B., Alwan A., Keating P.A. Development of a facility for simultaneous recordings of acoustic, optical (3-D motion and video), and physiological speech data. Journal of the Acoustical Society of America 2000, 107:2887.
4. Bernstein L.E., Auer E.T., Moore J.K. Audiovisual speech binding: Convergence or association?. Handbook of multisensory processes 2004, 203-223. MIT Press, Cambridge, MA. G.A. Calvert, C. Spence, B.E. Stein (Eds.).
5. Bernstein L.E., Lu Z.-L., Jiang J. Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing. Brain Research 2008, 1242:172-184.
6. Binnie C.A., Montgomery A.A., Jackson P.L. Auditory and visual contributions to the perception of consonants. Journal of Speech & Hearing Research 1974, 17:619-630.
7. Blumstein S.E., Myers E.B., Rissman J. The perception of voice onset time: An fMRI investigation of phonetic category structure. Journal of Cognitive Neuroscience 2005, 17:1353-1366.
8. Braida L.D. Crossmodal integration in the identification of consonant segments. Quarterly Journal of Experimental Psychology: Human Experimental Psychology 1991, 43(A):647-677.
9. Breeuwer M., Plomp R. Speechreading supplemented with formant-frequency information from voiced speech. Journal of the Acoustical Society of America 1985, 77:314-317.
10. Callan D.E., Callan A.M., Honda K., Masaki S. Single-sweep EEG analysis of neural processes underlying perception and production of vowels. Cognitive Brain Research 2000, 10:173-176.
11. Campbell C.S., Massaro D.W. Perception of visible speech: Influence of spatial quantization. Perception 1997, 26:627-644.
12. Campbell R. The processing of audio-visual speech: Empirical and neural bases. Philosophical Transactions of the Royal Society of London B: Biological Sciences 2008, 363:1001-1010.
13. Chandrasekaran C., Trubanova A., Stillittano S., Caplier A., Ghazanfar A.A. The natural statistics of audiovisual speech. PLoS Computational Biology 2009, 5:e1000436.
14. Craig M.S., van Lieshout P., Wong W. A linear model of acoustic-to-facial mapping: Model parameters, data set size, and generalization across speakers. Journal of the Acoustical Society of America 2008, 124:3183-3190.
15. Fowler C. Speech as a supramodal or amodal phenomenon. Handbook of multisensory processes 2004, 189-202. MIT Press, Cambridge, MA. G. Calvert, C. Spence, B.E. Stein (Eds.).
16. Grant K.W., Braida L.D. Evaluating the articulation index for auditory-visual input. Journal of the Acoustical Society of America 1991, 89:2952-2960.
17. Grant K.W., Greenberg S., Poeppel D., van Wassenhove V. Effects of spectro-temporal asynchrony in auditory and auditory-visual speech processing. Seminars in Hearing 2004, 25:241-255.
18. Green K.P., Kuhl P.K. The role of visual information in the processing of place and manner features in speech perception. Perception & Psychophysics 1989, 45:34-42.
19. Green K.P., Kuhl P.K., Meltzoff A.N., Stevens E.B. Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception & Psychophysics 1991, 50:524-536.
20. Green K.P., Norrix L.W. Acoustic cues to place of articulation and the McGurk effect: The role of release bursts, aspiration, and formant transitions. Journal of Speech, Language, & Hearing Research 1997, 40:646-665.
21. Hasson U., Skipper J.I., Nusbaum H.C., Small S.L. Abstract coding of audiovisual speech: Beyond sensory representation. Neuron 2007, 56:1116-1126.
22. Hosokawa M., Okazaki S., Kawakubo Y., Maekawa H., Ozaki H. Topographic change of ERP due to discrimination of CV syllables with various vowel durations. International Congress Series 2002, 1232:53-57.
23. Hutchison E.R., Blumstein S.E., Myers E.B. An event-related fMRI investigation of voice-onset time discrimination. Neuroimage 2008, 40:342-352.
24. Jiang J., Alwan A., Keating P.A., Auer E.T., Bernstein L.E. On the relationship between face movements, tongue movements, and speech acoustics. EURASIP Journal on Applied Signal Processing 2002, 2002:1174-1188.
25. Jiang J., Auer E.T., Alwan A., Keating P.A., Bernstein L.E. Similarity structure in visual speech perception and optical phonetic signals. Perception & Psychophysics 2007, 69:1070-1083.
26. Kailath T., Sayed A.H., Hassibi B. Linear estimation 2000, Prentice Hall, Upper Saddle River, NJ.
27. Kim J.-O., Mueller C.W. Factor analysis: Statistical methods and practical issues 1978, Sage, Beverly Hills, CA.
28. Liberman A.M., Cooper F.S., Shankweiler D.P., Studdert-Kennedy M. Perception of the speech code. Psychological Review 1967, 74:431-461.
29. MacDonald J., McGurk H. Visual influences on speech perception processes. Perception & Psychophysics 1978, 24:253-257.
30. Manuel S.Y., Repp B.H., Studdert-Kennedy M., Liberman A.M. Exploring the " McGurk Effect. " Journal of the Acoustical Society of America 1983, 74(S1):S66.
31. Massaro D.W. Speech perception by ear and eye: A paradigm for psychological inquiry 1987, Lawrence Erlbaum Associates, Hillsdale, NJ.
32. Massaro D.W. Perceiving talking faces: From speech perception to a behavioral principle 1998, MIT Press, Cambridge, MA.
33. Massaro D.W. Speechreading: Illusion or window into pattern recognition. Trends in Cognitive Sciences 1999, 3:310-317.
34. Massaro D.W., Cohen M.M., Smeele P.M.T. Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America 1996, 100:1777-1786.
35. McGrath M., Summerfield Q. Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America 1985, 77:678-685.
36. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264(5588):746-748.
37. Miller L.M., d'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 2005, 25:5884-5893.
38. Munhall K.G., Gribble P., Sacco L., Ward M. Temporal constraints on the McGurk effect. Perception & Psychophysics 1996, 58:351-362.
39. Munhall K.G., Kroos C., Jozan G., Vatikiotis-Bateson E. Spatial frequency requirements for audiovisual speech perception. Perception & Psychophysics 2004, 66:574-583.
40. Munhall K.G., Vatikiotis-Bateson E. The moving face during speech communication. Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech 1998, 123-139. Psychology Press, East Sussex, United Kingdom. R. Campbell, B. Dodd, D. Burnham (Eds.).
41. Nearey T.M. Speech perception as pattern recognition. Journal of the Acoustical Society of America 1997, 101:3241-3254.
42. Nock, H. J., Iyengar, G., & Neti, C. (2002, December). Assessing face and speech consistency for monologue detection in video. Paper presented at the ACM International Conference on Multimedia, Juan-les-Pins, France.
43. Ojanen V., Möttönen R., Pekkola J., Jaaskelainen I.P., Joensuu R., Autti T., Sams M. Processing of audiovisual speech in Broca's area. NeuroImage 2005, 25:333-338.
44. Papanicolaou A.C., Castillo E., Breier J.I., Davis R.N., Simos P.G., Diehl R.L. Differential brain activation patterns during perception of voice and tone onset time series: A MEG study. Neuroimage 2003, 18:448-459.
45. Pekkola J., Ojanen V., Autti T., Jaaskelainen I.P., Möttönen R., Tarkianen A., Sams M. Primary auditory cortex activation by visual speech: An fMRI study at 3T. NeuroReport 2005, 16:125-128.
46. Ponton C.W., Bernstein L.E., Auer E.T. Mismatch negativity with visual-only and audiovisual speech. Brain Topography 2009, 21:207-215.
47. Rabiner L.R. A tutorial on Hidden Markov Models and selected applications in speech recognition. Proceedings of The IEEE 1989, 77:257-286.
48. Rosenblum L.D. Speech perception as a multimodal phenomenon. Current Directions in Psychological Science 2008, 17:405-409.
49. Rosenblum L.D., Miller R.M., Sanchez K. Lip-read me now, hear me better later: Cross-modal transfer of talker-familiarity effects. Psychological Science 2007, 18:392-396.
50. Rouger J., Fraysse B., Deguine O., Barone P. McGurk effects in cochlear-implanted deaf subjects. Brain Research 2008, 1188:87-99.
51. Sams M., Aulanko R., Hamalainen M., Hari R., Lounasmaa O.V., Lu S.T., Simola J. Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 1991, 127:141-145.
52. Sekiyama K. Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception & Psychophysics 1997, 59:73-80.
53. Sekiyama K., Tohkura Y. McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. Journal of the Acoustical Society of America 1991, 90:1797-1805.
54. Simos P.G., Diehl R.L., Breier J.I., Molis M.R., Zouridakis G., Papanicolaou A.C. MEG correlates of categorical perception of a voice onset time continuum in humans. Cognitive Brain Research 1998, 7:215-219.
55. Skipper J.I., van Wassenhove V., Nusbaum H.C., Small S.L. Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex 2007, 17:2387-2399.
56. Stevens K.N. Acoustic phonetics 1998, MIT Press, Cambridge, MA.
57. Sugamura N., Itakura F. Speech analysis and synthesis methods developed at ECL in NTT: From LPC to LSP. Speech Communication 1986, 5:199-215.
58. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 1954, 26:212-215.
59. Summerfield Q. Some preliminaries to a comprehensive account of audio-visual speech perception. Hearing by eye: The psychology of lip-reading 1987, 3-52. Erlbaum, London, England. B. Dodd, R. Campbell (Eds.).
60. Summerfield Q., McGrath M. Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology A, Human Experimental Psychology 1984, 36:51-74.
61. Trébuchon-Da Fonseca A., Giraud K., Badier J.-M., Chauvel P., Liégeois-Chauvel C. Hemispheric lateralization of voice onset time (VOT) comparison between depth and scalp EEG recordings. NeuroImage 2005, 27:1-14.
62. van Wassenhove V., Grant K.W., Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 2007, 45:598-607.
63. Walker S., Bruce V., O'Malley C. Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect. Perception & Psychophysics 1995, 57:1124-1133.
64. Welch R.B., Warren D.H. Intersensory interactions. Handbook of perception and human performance, Volume I: Sensory processes and perception 1986, 25-36. Wiley, New York, NY. K.R. Boff, L. Kaufman, J.P. Thomas (Eds.).
65. Yamamoto E., Nakamura S., Shikano K. Lip movement synthesis from speech based on Hidden Markov Models. Speech Communication 1998, 26:105-115.
66. Yehia H., Rubin P., Vatikiotis-Bateson E. Quantitative association of vocal-tract and facial behavior. Speech Communication 1998, 26:23-43.