Perception of asynchronous and conflicting visual and auditory speech

Journal of the Acoustical Society of America

Volumn 100, Issue 3, 1996, Pages 1777-1786

  Massaro, Dominic W ^a   Cohen, Michael M ^a   Smeele, Paula M T ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; HEARING; HUMAN; HUMAN EXPERIMENT; PERCEPTION; PRIORITY JOURNAL; SPEECH PERCEPTION; VISION;

EID: 0029814916     PISSN: 00014966     EISSN: None     Source Type: Journal    
DOI: 10.1121/1.417342     Document Type: Article

References (48)

1. Bernstein, L. E. and Eberhardt, S. P. (1986). Johns Hopkins Lipreading Corpus I-II: Disc I (The Johns Hopkins University, Baltimore).
2. Binnie, C. A., Montgomery, A. A., and Jackson, P. L. (1974). "Auditory and visual contributions Io the perception of selected English consonants for normally hearing and hearing-impaired listeners," in Visual and Audio-visual Perception of Speech, edited by H. Birk Nielsen and E. Kampp, Scand. Audiol. Suppl. 4, 181-209.
3. Braida, L. D. (1991). "Cross-modal integration in the identification of consonant segments," T. Q. J. Exp. Psychol. 43A, 647-677.
4. Breeuwer, M., and Plomp, R. (1984). "Speechreading supplemented with frequency-selective sound-pressure information," J. Acoust. Soc. Am. 76, 686-691.
5. Campbell, R., and Dodd, B. (1980). "Hearing by eye," Q. J. Exp. Psychol. 32, 85-99.
6. Chandler, J. P. (1969). "Subroutine STEPIT - Finds local minima of a smooth function of several parameters," Behav. Sci. 14, 81-82.
7. Cohen, M. M., and Massaro, D. W. (1990). "Synthesis of visible speech," Behav. Res. Methods, Instrum. Comput. 22, 260-263.
8. Cohen, M. M., and Massaro, D. W. (1993). "Modeling coarticulation in synthetic visual speech," in Models and Techniques in Computer Animation, edited by N. M. Thalmann and D. Thalmann (Springer-Verlag, Tokyo), pp. 139-156.
9. Cohen, M. M., and Massaro, D. W. (1994). "Development and experimentation with synthetic visible speech," Behav. Res. Methods Instrum. Comput. 26, 260-265.
10. Erber, N. P. (1972). "Auditory, visual, and auditory-visual recognition of consonants by children with normal and impaired hearing," J. Speech Hear. Res. 15, 423-422.
11. Gouraud, H. (1971). "Continuous shading of curved surfaces," IEEE Trans. Comput. C-20(6), 623-628.
12. Green, K. P., and Kuhl, P. K. (1989). "The role of visual information in the processing of place and manner features in speech perception," Percept. Psychophys. 45, 34-42.
13. Klatt, D. H. (1980). "Software for a cascade/parallel formant synthesizer," J. Acoust. Soc. Am. 67, 971-995.
14. Koenig, E. (1965). (Cited by Pandey et al.)
15. Kohfeld, D. L. (1971). "Simple reaction time as a function of stimulus intensity in decibels of light and sound," J. Exp. Psychol. 88, 251-257.
16. Massaro, D. W. (1985). "Attention and preception: An information-integration perspective," Acta Psychol. 60, 211-241.
17. Massaro, D. W. (1987). Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry (Erlbaum, Hillsdale, NJ).
18. Massaro, D. W. (1989). "Multiple Book Review of Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry," Behav. Brain Sci. 12, 741-794.
19. Massaro, D. W. (1990). "A Fuzzy logical Model of Speech Perception," Proceedings of the XXIV International Congress of Psychology, in Human Information Processing: Measures, Mechanisms, and Models, edited by D. Vickers and P. L. Smith (North-Holland, Amsterdam), pp. 367-379.
20. Massaro, D. W. (1992). "Broadening the domain of the fuzzy logical model of perception," in Cognition, Conceptual, and Methodological Issues, edited by H. L. Pick, Jr., P. Van den Broek, and D. C. Knill (American Psychological Association, Washington, DC), pp. 51-84.
21. Massaro, D. W., and Cohen, M. M. (1983). "Evaluation and integration of visual and auditory information in speech perception," J. Exp. Psychol. Human Percept. Perform. 9, 753-771.
22. Massaro, D. W., and Cohen, M. M. (1990). "Perception of synthesized audible and visible speech," Psychol. Sci. 1, 55-63.
23. Massaro, D. W., and Cohen, M. M. (1993). "Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables," Speech Commun. 13, 127-134.
24. Massaro, D. W., and Cohen, M. M. (1995). "Perceiving talking faces," Curr. Directions Psychol. Sci. 3, 104-109.
25. Massaro, D. W., and Cohen, M. M. (in press). "Perceiving speech from inverted faces," Percept. Psychophys.
26. Massaro, D. W., and Loftus, G. (in press). "Sensory and perceptual storage: Data and theory," in Handbook of Perception and Cognition, edited by E. L. Bjork and R. A. Bjork (Academic, San Diego), Vol. 10.
27. Massaro, D. W., Tsuzaki, M., Cohen, M. M., Gesi, A., and Heredia, R. (1993). "Bimodal speech perception: An examination across languages," J. Phon. 21, 445-478.
28. Massaro, D. W., Cohen, M. M., and Smeele, P. M. T. (1995). "Cross-linguistic comparisons in the integration of visual and auditory speech," Mem. Cognit. 23, 113-131.
29. McGill, W. J. (1961). "Loudness and reaction time," Acta Psychol. 19, 193-199.
30. McGrath, M., and Summerfield, Q. (1985). "Intermodal timing relations and audio-visual speech recognition by normal-hearing adults," J. Acoust. Soc. Am. 77, 678-685.
31. McGurk, H., and MacDonald, J. (1976). "Hearing lips and seeing voices," Nature 264, 746-748.
32. Mills, A. E., and Thiem, R. (1980). "Auditory-visual fusions and illusions in speech perception," Linguistische Berichte 68/80, 85-108.
33. Montgomery, A. A., and Jackson, P. L. (1983). "Physical characteristics of the lips underlying vowel lipreading performance," J. Acoust. Soc. Am. 73, 2134-2144.
34. Munhall, K. G., Gribble, P., Sacco, L. and Ward, M. (1996). "Temporal constraints on the McGurk effect," Percept. Psychophys. 58, 351-362.
35. Pandey, P. C., Kunov, H., and Abel, S. M. (1986), "Disruptive effects of auditory signal delay on speech perception with lipreading," J. Aud. Res. 26, 27-41.
36. Parke, F. I. (1975). "A model for human faces that allows speech synchronized animation," Comput. Graphics J. 1(1), 1-4.
37. Parke, F. I. (1982). "Parameterized models for facial animation," IEEE Comput. Graphics 2(9), 61-68.
38. Pearce, A., Wyvill, B., Wyvill, G., and Hill, D. (1986). "Speech and expression: A computer solution to face animation," in Proceedings of Graphics Interface '86, pp. 136-140.
39. Reisberg, D., McLean, J., and Goldfield, A. (1987). "Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli," in Hearing by Eye: The Psychology of Lip-Reading, edited by B. Dodd and R. Campbell (Erlbaum, London), pp. 97-113.
40. Repp, B. H., Manuel, S. Y., Liberman, A. M., and Studdert-Kennedy, M. (1983). "Exploring the 'McGurk Effect'," paper presented at the 24th meeting of the Psychonomic Society, San Diego, California, November 1983, Bull. Psychonom. Soc., 358 (abstract).
41. Risberg, A., and Lubker, J. L. (1978). "Prosody and speechreading," Report STL-OPSR 4/78, Dept. of Speech Communication and Musical Acoustics, Royal Institute of Technology, Stockholm, 1-16.
42. Sekiyama, K., and Tohkura, Y. (1991). "McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility," J. Acoust. Soc. Am. 90, 1797-1805.
43. Smeele, P. M. T., and Sittig, A. C. (1990). "The contribution of vision to speech perception," Proceedings of the 13th International Symposium on Human Factors in Telecommunications, Torino, 525.
44. Smeele, P. M. T., and Sittig, A. C. (1991). "The contribution of vision to speech perception," Proceedings of the 2nd European Conference on Speech Communication and Technology, Eurospeech 91, Genova, 1495-1497.
45. Sumby, W. H., and Pollack, I. (1954). "Visual contribution to speech intelligibility in noise," J. Acoust. Soc. Am. 26, 212-215.
46. Summerfield, A. Q. (1979). "Use of visual information in phonetic perception," Phonetica 36, 314-331.
47. Thompson, L. A., and Massaro, D. W. (1989). "Before you see it, you see its parts: Evidence for feature encoding and integration in preschool children and adults," Cognitive Psychol. 21, 334-362.
48. Vroomen, J. H. M. (1992). "Hearing voices and seeing lips: Investigations in the psychology of lipreading," Dissertation, Katholieke Universiteit Brabant.