Similarity structure in visual speech perception and optical phonetic signals

Perception and Psychophysics

Volumn 69, Issue 7, 2007, Pages 1070-1083

  Jiang, Jintao ^a,b   Auer Jr , Edward T ^a,c   Alwan, Abeer ^b   Keating, Patricia A ^b   Bernstein, Lynne E ^a  

^a HOUSE EAR INSTITUTE   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF KANSAS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 36148950995     PISSN: 00315117     EISSN: None     Source Type: Journal    
DOI: 10.3758/BF03193945     Document Type: Article

References (73)

1. ANDERSSON, U., & LIDESTAM, B. (2005). Bottom-up driven speechreading in a speechreading expert: The case of AA (JK023). Ear & Hearing, 26, 214-224.
2. ARNOLD, P., & HILL, F. (2001). Bisensory augmentation: A speechreading advantage when speech is clearly audible and intact. British Journal of Psychology, 92, 339-355.
3. ASHBY, F. G., MADDOX, W. T., & LEE, W. W. (1994). On the dangers of averaging across subjects when using multidimensional scaling or the similarity-choice model. Psychological Science, 5, 144-151.
4. AUER, E. T., JR. (2002). The influence of the lexicon on speech read word recognition: Contrasting segmental and lexical distinctiveness. Psychonomic Bulletin & Review, 9, 341-347.
5. AUER, E. T., JR., & BERNSTEIN, L. E. (1997). Speechreading and the structure of the lexicon: Computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. Journal of the Acoustical Society of America, 102, 3704-3710.
6. AUER, E. T., JR., & BERNSTEIN, L. E. (2007). Enhanced visual speech perception in individuals with early-onset hearing impairment. Journal of Speech, Language, & Hearing Research, 50, 1-9.
7. AUER, E. T., JR., BERNSTEIN, L. E., & TUCKER, P. E. (2000). Is subjective word familiarity a meter of ambient language? A natural experiment on effects of perceptual experience. Memory & Cognition, 28, 789-797.
8. BENOI̊T, C., GUIARD-MARIGNY, T., LE GOFF, B., & ADJOUDANI, A. (1996). Which components of the face do humans and machines best speechread? In D. G. Stork & M. E. Hennecke (Eds.), Speechreading by humans and machines: Models, systems, and applications (pp. 315-328). Berlin: Springer.
9. BERNSTEIN, L. E. (2006). Visual speech perception. In E. Vatikiotis-Bateson, G. Bailly. & P. Perrier (Eds.), Audio-visual speech processing. Cambridge, MA: MIT Press.
10. BERNSTEIN, L. E., AUER, E. T., JR., & TAKAYANAGI, S. (2004). Auditory speech detection in noise enhanced by lipreading. Speech Communication, 44, 5-18.
11. BERNSTEIN, L. E., AUER, E. T., JR., & TUCKER, P. E. (2001). Enhanced speechreading in deaf adults: Can short-term training/practice close the gap for hearing adults? Journal of Speech, Language, & Hearing Research, 44, 5-18.
12. BERNSTEIN, L. E., DEMOREST, M. E., & TUCKER, P. E. (1998). What makes a good speechreader? First you have to find one. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading andauditory-visual speech (pp. 211-228). Hove, U.K.: Psychology Press.
13. BERNSTEIN, L. E., DEMOREST, M. E., & TUCKER, P. E. (2000). Speech perception without hearing. Perception & Psychophysics, 62, 233-252.
14. BREEUWER, M., & PLOMP, R. (1986). Speechreading supplemented with auditorily presented speech parameters. Journal of the Acoustical Society of America, 79, 481-499.
15. CAMPBELL, R. (1996, October). Seeing speech in space and time: Psychological and neurological findings. Paper presented at the ICSLP 1996, Philadelphia.
16. DE GELDER, B., & BERTELSON, P. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences, 7, 460-467.
17. DEMOREST, M. E., & BERNSTEIN, L. E. (1992). Sources of variability in speechreading sentences: A generalizability analysis. Journal of Speech & Hearing Research, 35, 876-891.
18. DODD, B., MCINTOSH, B., & WOODHOUSE, L. (1998). Early lipreading ability and speech and language development of hearing-impaired pre-schoolers. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 229-242). Hove, U.K.: Psychology Press.
19. EDELMAN, S. (1998). Representation is representation of similarities. Behavioral & Brain Sciences, 21, 449-498.
20. ERBER, N. P. (1975). Auditory-visual perception of speech. Journal of Speech & Hearing Disorders, 40, 481-492.
21. ERBER, N. P. (1979). Auditory-visual perception of speech with reduced optical clarity. Journal of Speech & Hearing Research, 22, 212-223.
22. FISHER, C. G. (1968). Confusions among visually perceived consonants. Journal of Speech & Hearing Research, 11, 796-804.
23. FRISCH, S. A., PIERREHUMBERT, J. B., & BROE, M. B. (2004). Similarity avoidance and the OCP. Natural Language & Linguistic Theory, 22, 179-228.
24. GOLDSTONE, R. L. (1999). Similarity. In R. A. Wilson & F. C. Keil (Eds.), The MIT encyclopedia of the cognitive sciences (pp. 763-765). Cambridge, MA: MIT Press.
25. GRANT, K. W (2001). The effect of speechreading on masked detection thresholds for filtered speech. Journal of the Acoustical Society of America, 109, 2272-2275.
26. GRANT, K. W., & SEITZ, P. F. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America, 108, 1197-1208.
27. GRANT, K. W., & WALDEN, B. E. (1996). Evaluating the articulation index for auditory-visual consonant recognition. Journal of the Acoustical Society of America, 100, 2415-2424.
28. GRANT, K. W., WALDEN, B. E., & SEITZ, P. F. (1998). Auditory-visual speech recognition by hearing-impaired subjects: Consonant recognition, sentence recognition, and auditory-visual integration. Journal of the Acoustical Society of America, 103, 2677-2690.
29. IEEE (1969). IEEE recommended practice for speech quality measurements. IEEE Transactions on Audio & Electroacoustics, 17, 225-246.
30. IJSSELDIJK, F. J. (1992). Speechreading performance under different conditions of video image, repetition, and speech rate. Journal of Speech & Hearing Research, 35, 466-471.
31. IVERSON, P., BERNSTEIN, L. E., & AUER, E. T., JR. (1998). Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition. Speech Communication, 26, 45-63.
32. JACKSON, P. L., MONTGOMERY, A. A., & BINNIE, C. A. (1976). Perceptual dimensions underlying vowel lipreading performance. Journal of Speech & Hearing Research, 19, 796-812.
33. JIANG, J., ALWAN, A., KEATING, P., AUER, E. T., JR., & BERNSTEIN, L. E. (2002). On the relationship between face movements, tongue movements, and speech acoustics. EURASIP Journal on Applied Signal Processing, 2002, 1174-1188.
34. JORDAN, T. R., & THOMAS, S. M. (2001). Effects of horizontal viewing angle on visual and audiovisual speech recognition. Journal of Experimental Psychology: Human Perception & Performance, 27, 1386-1403.
35. KAILATH, T., SAYED, A. H., & HASSIBI, B. (2000). Linear estimation. Upper Saddle River, NJ: Prentice Hall.
36. 0. Journal of the Acoustical Society of America, 100, 593-602.
37. KLATT, D. H. (1987). Review of text-to-speech conversion for English. Journal of the Acoustical Society of America, 82, 737-793.
38. KRICOS, P. B., & LESNER, S. A. (1982). Differences in visual intelligibility across talkers. Volta Review, 84, 219-225.
39. KRUSKAL, J. B., & WISH, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage.
40. MACEACHERN, E. (2000). On the visual distinctiveness of words in the English lexicon. Journal of Phonetics, 28, 367-376.
41. MACLEOD, A., & SUMMERFIELD, Q. (1987). Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology, 21, 131-141.
42. MARASSA, L. K., & LANSING, C. R. (1995). Visual word recognition in two facial motion conditions: Full-face versus lips-plus-mandible. Journal of Speech & Hearing Research, 38, 1387-1394.
43. MASSARO, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
44. MATTYS, S. L., BERNSTEIN, L. E., & AUER, E. T., JR. (2002). Stimulus-based lexical distinctiveness as a general word-recognition mechanism. Perception & Psychophysics, 64, 667-679.
45. MCCOTTER, M. V., & JORDAN, T. R. (2003). The role of facial colour and luminance in visual and audiovisual speech perception. Perception, 32, 921-936.
46. MCGRATH, M. (1985). An examination of cues for visual and audiovisual speech perception using natural and computer-generated faces. Unpublished thesis, University of Nottingham.
47. MCGURK, H., & MACDONALD, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
48. MOHAMMED, T., CAMPBELL, R., MACS WEENEY, M., MILNE, E., HANSEN, P., & COLEMAN, M. (2005). Speechreading skill and visual movement sensitivity are related in deaf speechreaders. Perception, 34, 205-216.
49. MONTGOMERY, A. A., & JACKSON, P. L. (1983). Physical characteristics of the lips underlying vowel lipreading performance. Journal of the Acoustical Society of America, 73, 2134-2144.
50. MOODY-ANTONIO, S., TAKAYANAGI, S., MASUDA, A., AUER, E. T., JR., FISHER, L., & BERNSTEIN, L. E. (2005). Improved speech perception in adult congenitally deafened cochlear implant recipients. Otology & Neurotology, 26, 649-654.
51. MUNHALL, K. G., KROOS, C., JOZAN, G., & VATIKIOTIS-BATESON, E. (2004). Spatial frequency requirements for audiovisual speech perception. Perception & Psychophysics, 66, 574-583.
52. MUNHALL, K. G., & VATIKIOTIS-BATESON, E. (1998). The moving face during speech communication. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 123-139). Hove, U.K.: Psychology Press.
53. NOSOFSKY, R. [M.] (1985). Overall similarity and the identification of separable-dimension stimuli: A choice model analysis. Perception & Psychophysics, 38, 415-432.
54. NOSOFSKY, R. M., & STANTON, R. D. (2005). Speeded classification in a probabilistic category structure: Contrasting exemplar-retrieval, decision-boundary, and prototype models. Journal of Experimental Psychology: Human Perception & Performance, 31, 608-629.
55. OWENS, E., & BLAZEK, B. (1985). Visemes observed by hearing-impaired and normal-hearing adult viewers. Journal of Speech & Hearing Research, 28, 381-393.
56. PLANT, G. L. (1980). Visual identification of Australian vowels and diphthongs. Australian Journal ofAudiology, 2, 83-91.
57. REISBERG, D., MCLEAN, J., & GOLDFIELD, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 97-113). London: Erlbaum.
58. REPP, B. H. (1981). On levels of description in speech research. Journal of the Acoustical Society of America, 69, 1462-1464.
59. RÖNNBERG, J. (1995). Perceptual compensation in the deaf and blind: Myth or reality? In R. A. Dixon & L. Bãckman (Eds.), Compensating for psychological deficits and declines: Managing losses and promoting gains (pp. 251-274). Mahwah, NJ: Erlbaum.
60. RÖNNBERG, J., SAMUELSSON, S., & LYXELL, B. (1998). Conceptual constraints in sentence-based lipreading in the hearing impaired. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 143-153). Hove, U.K.: Psychology Press.
61. ROSENBLUM, L. D., & SALDAÑA, H. M. (1998). Time-varying information for visual speech perception. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 61-81). Hove, U.K.: Psychology Press.
62. SCHEINBERG, J. S. (1980). Analysis of speechreading cues using an interleaved technique. Journal of Communication Disorders, 13, 489-492.
63. SHEPARD, R. N. (1965). Approximation to uniform gradients of generalization by monotone transformations of scale. In D. I. Mostofsky (Ed.), Stimulus generalization (pp. 94-110). Stanford, CA: Stanford University Press.
64. SHEPARD, R. N., & CHIPMAN, S. (1970). Second-order isomorphism of internal representations: Shapes of states. Cognitive Psychology, 1, 1-17.
65. SIEGEL, S., & CASTELLAN, N. J., JR. (1988). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.
66. SMEELE, P. M. T., HAHNLEN, L. D., STEVENS, E. B., KUHL, P. K., & MELTZOFF, A. N. (1995). Investigating the role of specific facial information in audio-visual speech perception [Abstract]. Journal of the Acoustical Society of America, 98, 2983.
67. SPSS (2003). SPSS base 12.0 user's guide. Chicago: Author.
68. STEVENS, K. N. (1998). Acoustic phonetics. Cambridge, MA: MIT Press.
69. STEVENS, K. N., MANUEL, S. Y., SHATTUCK- HUFNAGEL, S., & LIU, S. (1992, October). Implementation of a model for lexical access based on features. Paper presented at the ICSLP 1992, Banff, Canada.
70. SUMBY, W. H., & POLLACK, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.
71. SUMMERFIELD, Q., MACLEOD, A., MCGRATH, M., & BROOKE, M. (1989). Lips, teeth, and the benefits of lipreading. In A. W. Young & H. D. Ellis (Eds.), Handbook of research on face processing (pp. 223-233). Amsterdam: North-Holland.
72. THOMAS, S. M., & JORDAN, T. R. (2004). Contributions of oral and extraoral facial movement to visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception & Performance, 30, 873-888.
73. WALDEN, B. E., PROSEK, R. A., MONTGOMERY, A. A., SCHERR, C. K., & JONES, C. J. (1977). Effects of training on the visual recognition of consonants. Journal of Speech & Hearing Research, 20, 130-145.