The temporal distribution of information in audiovisual spoken-word identification

Attention, Perception, and Psychophysics

Volumn 72, Issue 1, 2010, Pages 209-225

  Jesse, Alexandra ^a   Massaro, Dominic W ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADOLESCENT; ADULT; ARTICLE; FEMALE; HEARING; HUMAN; LINGUISTICS; MALE; PERCEPTIVE DISCRIMINATION; SPEECH PERCEPTION; TIME PERCEPTION; VISION;

ADOLESCENT; ADULT; AUDITORY PERCEPTION; FEMALE; HUMANS; MALE; SIGNAL DETECTION, PSYCHOLOGICAL; SPEECH PERCEPTION; TIME PERCEPTION; VISUAL PERCEPTION; VOCABULARY; YOUNG ADULT;

EID: 77949327376     PISSN: 19433921     EISSN: None     Source Type: Journal    
DOI: 10.3758/APP.72.1.209     Document Type: Article

References (70)

1. ALLOPENNA, P. D., MAGNUSON, J. S., & TANENHAUS, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory A Language, 38, 419-439. doi:10.1006/jmla.1997.2558
2. AUER, E. T., JR. (2002). The influence of the lexicon on speech read word recognition: Contrasting segmental and lexical distinctiveness. Psychonomic Bulletin & Review, 9, 341-347.
3. BAKEMAN, R. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37, 379-384.
4. BENGUEREL, A.-P., & PICHORA-FULLER, M. K. (1982). Coarticulation effects in lipreading. Journal of Speech & Hearing Research, 25, 600-607.
5. BENKf, J. R. (2003). Analysis of English nonsense syllable recognition in noise. Phonetica, 60, 129-157. doi:10.1159/000071450
6. BERNSTEIN, L. E., & EBERHARDT, S. P. (1986). Johns Hopkins lipreading corpus I-II: Disc I [Videodisk]. Baltimore: Johns Hopkins University.
7. BINNIE, C. A., MONTGOMERY, A. A., & JACKSON, P. L. (1974). Auditory and visual contributions to the perception of consonants. Journal of Speech & Hearing Disorders, 17, 619-630.
8. BRANCAZIO, L. (2004). Lexical influences in audiovisual speech perception. Journal of Expertmental Psychology: Human Perception & Performance, 30, 445-463. doi:10.1037/0096-1523.30.3.445
9. BREEUWER, M., & PLOMP, R. (1986). Speechreading supplemented with auditorily presented speech parameters. Journal of the Acoustical Society of America,19, 481-499. doi:10.1121/1.393536
10. CAMPBELL, C. S., & MASSARO, D. W. (1997). Perception of visible speech: Influence of spatial quantization. Perception, 26, 627-644. doi: 10.1068/ p260627
11. CAMPBELL, R., & DODD, B. (1980). Hearing by eye. QuarterlyJournal of Expertmental Psychology, 32, 85-99. doi:10.1080/00335558008248235
12. CATHIARD, M.-A., LALLOUACHE, M. T., MOHAMADI, T., & ABRY, C. (1995). Configurational vs. temporal coherence in audio-visual speech perception. In K. Elenius & P. Branderud (Eds.), Proceedings of the 13th International Congress of Phonetic Sciences (Vol. 3, pp. 218-221). Stockholm: ICPhS.
13. CHOMSKY, N., & HALLE, M. (1968). The sound pattern of English. New York: Harper & Row.
14. COHEN, M. M., MASSARO, D. W., & CLARK, R. (2002). Training a talking head. In D. C. Martin (Ed), Proceedings of the IEEE Fourth International Conference on Multimodal Interfaces (ICMI'02) (pp. 499-510). Pittsburgh. doi:10.1109/ICMI.2002.1167046
15. CUTLER, A., WEBER, A., SMITS, R., & COOPER, N. (2004). Patterns of English phoneme confusions by native and non-native listeners. Journal of the Acoustical Society of America, 116, 3668-3678. doi: 10.1121/1 .1810292
16. DAVIS, M. H., MARSLEN-WILSON, W. D., & GASKELL, M. G. (2002). Leading up the lexical garden path: Segmentation and ambiguity in spoken word recognition. Journal of Expertmental Psychology: Human Perception & Performance, 28, 218-244.
17. DE LA VAUX, S. K., & MASSARO, D. W. (2004). Audiovisual speech gating: Examining information and information processing Cognitive Processing, 5, 106-112. doi:10.1007/s10339-004-0014-2
18. DEMOREST, M. E., & BERNSTEIN, L. E. (1992). Sources of variability in speechreading sentences: A generalizability analysis. Journal of Speech & Hearing Research, 35, 876-891.
19. ERBER, N. P. (1974). Effects of angle, distance, and illumination on visual reception of speech by profoundly deaf children. Journal of Speech & Hearing Research, 17, 99-112.
20. FISHER, C. G. (1968). Confusions among visually perceived consonants. Journal of Speech & Hearing Research, 15, 474-482.
21. GRANT, K. W., & GREENBERG, S. (2001). Speech intelligibility derived from asynchronous processing of auditory-visual information. In D. W. Massaro, J. Light, & K. Geraci (Eds.), Proceedings of Auditory-Visual Speech Processing (AVSP-2001) (pp. 132-137). Santa Cruz, CA; Perceptual Science Laboratory.
22. GRANT, K. W., & WALDEN, B. E. (1996). Evaluating the articulation index for audio-visual consonant recognition. Journal of the Acoustical Society of America, 100, 2415-2424. doi:10.1121/1.417950
23. GREESBERG, S. (1999). Speaking in shorthand- A syllable-centric perspective for understanding pronunciation variation. Speech Communication, 29, 159-176. doi:10.1016/S0167-6393(99)00050-3
24. GREENBERG, S. (2005). A multi-tier framework for understanding spoken language. In S. Greenberg & W. Ainsworth (Eds.), Listening to speech: An auditory perspective (pp. 411-433). Hillsdale, NJ: Erlbaum.
25. GRIMM, W. A. (1966). Perception of segments of English-spoken consonant-vowel syllables. Journal of the Acoustical Society of Americo, 40, 1454-1461. doi.10.1121/1.1910248
26. GROSJEAN, F. (1980). Spoken word recognition processes and the gating paradigm. Perception & Psychophysics, 28, 267-283.
27. GROSJEAN, F. (1996). Gating. Language & Cognitive Processes, 11, 597-604. doi:10.1080/016909696386999
28. JACKSON, P. L. (1988). The theoretical minimal unit for visual speech perception. Volta Review, 90, 99-115.
29. KIEFTE, M. (2003). Temporal information in gated stop consonants. Speech Communication, 40, 315-333. doi:10.1016/S0167-6393(02)00069-9
30. KRICOS, P. B., & LESNER, S. A. (1982). Differences in visual intelligibility across talkers. Volta Review, 84, 219-225.
31. KUC̄ERA, H., & FRANCIS, W. (1967). Computational analysts of presentday American English. Providence, RI: Brown University Press.
32. LANSING, C. R., MCCONKIE, G. W. (2003). Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception & Psychophysics, 65, 536-552.
33. LISKER, L., & Rossi, M. (1992). Auditory and visual cueing of the [± rounded] feature of vowels. Language & Speech, 35, 391-417.
34. LÖFQVIST, A. (1990). Speech as audible gestures. In W. J. Hardcastle & A Marchai (Eds.), Speech production and speech modeling (pp. 289-322). Dordrecht, NL: Kluwer.
35. MACLEOD, A., & SUMMERFIELD, Q. (1987). Quantifying the contribution of vision to speech perception in noise. British Journal ofAudiology, 21, 131-141.doi:10.3109/03005368709077786
36. MAGNUSON, J. S., DIXON, J., TANENHAUS, M. K., & ASLIN, R. N. (2007). The dynamics of lexical competition during spoken word recognition. Cognitive Science, 31, 133-156. doi:10.1207/s15516709cog3101-5
37. MASSARO, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle Cambridge, MA: MIT Press.
38. MASSARO, D. W., & COHEN, M. M. (1993). Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables. Speech Communication, 13, 127-134. doi:10.1016/0167-6393(93)90064-R
39. MASSARO, D. W., COHEN, M. M., & SMEELE, P. M. T. (1996). Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America, 100, 1777-1786. doi:10.1121/ 1.417342
40. MATTYS, S. L., BERNSTEIN, L. E., & AUER, E. T., JR. (2002). Stimulusbased lexical distinctiveness as a general word-recognition mechanism. Perception & Psychophysics, 64, 667-679.
41. MCGURK, H., & MACDONALD, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748. doi:10.1038/264746a0
42. MILLER, G. A., & NICELY, P. A. (1955). An analysis of perceptual confusions among some English consonants. Journal of the Acoustical Society of America, 27, 338-352. doi:10.1121/1.1907526
43. MUNHALL, K. G., GRIBBLE, P., SACCO, L., & WARD, M. (1996). Temporal constraints on the McGurk effect Perception & Psychophysics, 58, 351-362.
44. MUNHALL, K. G., & TOHKURA, Y. (1998). Audiovisual gating and the time course of speech perception. Journal of the Acoustical Society of America, 104, 530-539. doi:10.1121/1.423300
45. NORMS, D., & MCQUEEN, J. M. (2008). Shortlist B: A Bayesian model of continuous speech recognition. Psychological Review, 115, 357-395. doi:10.1037/0033-295X.l15.2.357
46. ÖHMAN, S. E. G. (1966). Perception of segments of VCCV utterances. Journal of the Acomtical Society of America, 40, 979-988. doi:10.1121/ 1.1910222
47. OLEJNIK, S., & ALGINA, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434-447. doi:10.1037/1082-989X.8.4.434
48. OUNI, S., COHEN, M. M., ISHAK, H., & MASSARO, D. W. (2007). Visual contribution to speech perception: Measuring the intelligibility of ani-mated talking heads. EURASIP Journal on Audio, Speech, & Music Processing, 2007 (Art No. 47891). doi: 10.1155/2007/47891
49. OWENS, E., & BLAZEK, B. (1985). Visemes observed by hearing-impaired and normal-hearing adult viewers. Journal of Speech & Hearing Research, 28, 381-393.
50. POLS, L. C. W., & SCHOUTEN, M. E. H. (1978). Identification of deleted consonants. Journal of the Acoustical Society of America, 64, 1333-1337. doi:10.1121/1.382100
51. REISBERO, D., MCLEAN, J., & GOLDFIELD, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 97-113). Hillsdale, NJ: Erlbaum.
52. ROBERT-RlBES, J., SCHWARTZ, J.-L., LALLOUACHE, T., & ESCUDIER, P. (1998). Complementarity and synergy in bimodal speech: Auditory, visual and audio-visual identification of French oral vowels in noise. Journal of the Acoustical Society of America, 103, 3677-3689. doi:10.1121/1.423069
53. SALTZMAN, E. L., & MUNHALL, K. G. (1989). A dynamical approach to gestural patterning in speech production. Ecological Psychology, 1, 1615-1623. doi:10.1207/s15326969eco0104-2
54. SEITZ, P. F., & GRANT, K. W. (1999). Modality, perceptual encoding speed, and time-course of phonetic information. In D. W. Massaro (Ed), Proceedings of Auditory-Visual Speech Processing (AVSP'99). Santa Cruz, CA.
55. SHANNON, C. E. (1948). A mathematical theory of communications. Bell Systems Technical Journal, 27, 379-423.
56. SMEELE, P. M. T. (1994). Perceiving speech: Integrating auditory and visual speech. Unpublished doctoral dissertation, Delft University of Technology, The Netherlands.
57. SMITS, R. (2000). Temporal distribution of information for human consonant recognition in VCV utterances. Journal of Phonetics, 27, 111-135. doi:10.1006/jpho.2000.0107
58. SMITS, R., WARNER, N., MCQUEEN, J. M., & CUTLER, A. (2003). Unfolding of phonetic information over time: A database of Dutch diphone perception. Journal of the Acoustical Society of America, 113, 563-574. doi: 10.1121/1.1525287
59. STEVENS, K. N. (2000). Acoustic phonetics. Cambridge, MA: MIT Press.
60. STEVENS, K. N., & BLUMSTEIN, S. E. (1978). Invariant cues for place of articulation in stop consonants. Journal of the Acoustical Society of America, 64, 1358-1368. doi:10.1121/1.382102
61. SUMBY, W. H., & POLLACK, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215. doi:10.1121/1.1907309
62. SUMMERFIELD, A. Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 3-52). Hillsdale, NJ: Erlbaum.
63. TEKIEU, M. E., & CULLINAN, W. L. (1979). The perception of temporally segmented vowels and consonant-vowel syllables. Journal of Speech & Hearing Research, 22, 103-121.
64. TRAUNMÜLLER, H., & ÖHRSTRÖM, N. (2007). Audiovisual perception of openness and lip rounding in front vowels. Journal of Phonetics, 35, 244-258. doi:10.1016/j.wocn.2006.03.002
65. TYLER, L. K. (1984). The structure of the initial cohort Evidence from gating. Perception & Psychophysics, 36, 417-427.
66. VAN WASSENHOVE, V. (2004). Cortical dynamics of auditory-visual speech: A forward model of multisensory integration. Unpublished doctoral dissertation, University of Maryland.
67. WALDEN, B. E., PROSEK, R. A., & WORTHINGTON, D. W. (1974). Predicting audiovisual consonant recognition performance of hearingimpaired adults. Journal of Speech & Hearing Research, 17, 270-278.
68. WANG, M. D., & BILGER, R. C. (1973). Consonant confusions in noise: A study of perceptual features. Journal of the Acoustical Society of America, 54, 1248-1266. doi:10.1121/1.1914417
69. YEHIA, H., RUBIN, P., & VATTKIOTIS-BATESON, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 16, 23-43. doi:10.1016/S0167-6393(98)00048-X
70. ZWITSERLOOD, P. (1989). The locus of the effects of sententialsemantic context in spoken-word processing. Cognition, 32, 25-64. doi:10.1016/0010- 0277(89)90013-9