Visual speech primes open-set recognition of spoken words

Language and Cognitive Processes

Volumn 24, Issue 4, 2009, Pages 580-610

  Buchwald, Adam B ^a,b   Winters, Stephen J ^b,c   Pisoni, David B ^b  

^a NEW YORK UNIVERSITY   (United States)

^b INDIANA UNIVERSITY   (United States)

^c UNIVERSITY OF CALGARY   (Canada)

Author keywords

Audiovisual priming; Lexical access; Visual speech; Word recognition

Indexed keywords

ACCURACY; ADULT; ARTICLE; AUDITORY STIMULATION; FEMALE; HUMAN; HUMAN EXPERIMENT; LANGUAGE PROCESSING; LINGUISTICS; MALE; MENTAL PERFORMANCE; NORMAL HUMAN; PHONETICS; PRIORITY JOURNAL; SPEECH PERCEPTION; VISUAL INFORMATION; VISUAL SPEECH PERCEPTION; VISUAL STIMULATION; WORD RECOGNITION;

EID: 70549097068     PISSN: 01690965     EISSN: 14640732     Source Type: Journal    
DOI: 10.1080/01690960802536357     Document Type: Article

References (82)

1. Auer, E. T., Jr. (2002). The influence of the lexicon on speech read word recognition: Contrasting segmental and lexical distinctiveness. Psychonomic Bulletin and Review, 9(2), 341-347.
2. Auer, E. T., Jr., & Bernstein, L. E. (1997). Speechreading and the structure of the lexicon: Computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. Journal of the Acoustical Society of America, 102(6), 3704-3710.
3. Bergeson, T. R., & Pisoni, D. B. (2004). Audiovisual speech perception in deaf adults and children following cochlear implantation. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes. Cambridge, MA: MIT Press.
4. Bernstein, L. E. (2005). Phonetic processing by the speech perceiving brain. In D. B. Pisoni & R. E. Remez (Eds.), Handbook of speech perception (pp. 79-98). Malden, MA: Blackwell.
5. Bowers, J. S., & Marsolek, C. J. (Eds.) (2003). Rethinking implicit memory. New York: Oxford University Press.
6. Bybee, J. (2001). Frequency and language use. Cambridge: Cambridge University Press.
7. Calvert, G. A., Spence, C, & Stein, B. E. (Eds.) (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press.
8. Cole, R. A., & Jakimik, J. (1980). A model of speech perception. In R. A. Cole (Ed.), Perception and production offluent speech (pp. 133-163). Hillsdale, NJ: Lawrence Erlbaum Associates.
9. Davis, C, & Kim, J. (2001). Repeating and remembering foreign language words: Implications for language teaching systems. Artificial Intelligence Review, 16, 37-47.
10. Davis, C, & Kim, J. (2004). Audio-visual interactions with intact clearly audible speech. Quarterly Journal of Experimental Psychology, 57A(6), 1103-1121.
11. Dodd, B., Oerlemens, M., & Robinson, R. (1989). Cross-modal effects in repetition priming: A comparison of lip-read graphic and heard stimuli. Visible Language, 22, 59-77.
12. Dufour, S., & Peereman, R. (2003). Inhibitory priming effects in auditory word recognition: When the target's competitors conflict with the word prime. Cognition, 88, B33-B44.
13. Foster, K. I., & Davis, C. (1984). Repetition priming and frequency attenuation in lexical access. Journal of Experimental Psychology: Learning, Memory and Cognition, 10, 680-689.
14. Fowler, C. (1986). An event approach to the study of speech perception from a direct-realist perspective. Journal of Phonetics, 14, 3-28.
15. Fowler, C. (1996). Listeners do hear sounds, not tongues. Journal of the Acoustical Society of America, 99, 1730-1741.
16. Fowler, C. (2004). Speech as a supramodal or amodal phenomenon. In G A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes. Cambridge, MA: MIT Press.
17. Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in Cognitive Science, 10, 278-285.
18. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston, MA: Houghton Mifflin.
19. Goldinger, S. D. (1996). Words and voices: episodic traces in spoken word identification and recognition memory. Journal of Experimental Psychology: Learning, Memory and Cognition, 22, 1166-1183.
20. Goldinger, S. D. (1998). Echoes of echoes? An episodic theory of lexical access. Psychological Review, 105(2), 251-279.
21. Goldinger, S. D., & Azuma, T. (2003). Puzzle-solving science: The quixotic quest for units in speech perception. Journal of Phonetics, 31, 305-320.
22. Green, K. P., Kuhl, P. K., Meltzoff, A. N., & Stevens, E. B. (1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception and Psychophysics, 38, 269-276.
23. Grill-Spector, K., Henson, R., & Martin, A. (2006). Repetition and the brain: Neural models of stimulus-specific effects. Trends in Cognitive Science, 10(1), 14-23.
24. Hamilton, R. H., Shenton, J. T., & Coslett, H. B. (2006). An acquired deficit of audiovisual speech processing. Brain and Language, 98, 66-73.
25. Horii, Y, House, A. S., & Hughes, G. W. (1971). A masking noise with speech envelope characteristics for studying intelligibility. Journal of the Acoustical Society of America, 49, 1849-1856.
26. Huttenlocher, D. P., & Zue, V. W. (1984). A model of lexical access from partial phonetic information. Paper presented at the IEEE International Conference on Acoustics, Speech and Signal Processing.
27. Iverson, P., Bernstein, L. E., & Auer, E. T., Jr. (1998). Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition. Speech Communication, 26, 45-63.
28. Johnson, K. (1997). Speech perception without speaker normalization: an exemplar model. In K. Johnson & J. W. Mullenix (Eds.), Talker variability in speech processing (pp. 145-166). San Diego, CA: Academic Press.
29. Johnson, K. (2005). Decisions and mechanisms in exemplar-based phonology. Berkeley, CA: UC Berkeley.
30. Kamachi, M., Hill, H., Lander, K., & Vatikiotis-Bateson, E. (2003).'Putting the face to the voice': Matching identity across modality. Current Biology, 13, 1709-1714.
31. Kerzel, D., & Bekkering, H. (2000). Motor activation from visible speech: Evidence from stimulus response compatibility. Journal of Experimental Psychology: Human Perception and Performance, 26(2), 634-647.
32. Kim, J., & Davis, C. (2003). Task effects in masked cross-script translation and phonological priming. Journal of Memory and Language, 49, 484-99.
33. Kim, J., Davis, C, & Krins, P. (2004). Amodal processing of visual speech as revealed by priming. Cognition, 93(1), B39-B47.
34. Lachs, L. (Ed.). (2002). Vocal tract kinematics and crossmodalspeech information. Bloomington, IN: Speech Research Laboratory, Indiana University.
35. Lachs, L., & Hernandez, L. R. (1998). Update: The Hoosier audiovisual multi-talker database. In Research on spoken language processing progress report no. 22 (pp. 377-388). Bloomington, IN: Speech Research Laboratory, Indiana University.
36. Lachs, L., & Pisoni, D. B. (2004a). Cross-modal source information and spoken word recognition. Journal of Experimental Psychology: Human Perception and Performance, 30(2), 378-396.
37. Lachs, L., & Pisoni, D. B. (2004b). Crossmodal source identification in speech perception. Ecological Psychology, 16(3), 159-187.
38. Lachs, L., Pisoni, D. B., & Kirk, K. I. (2001). Use of audiovisual information in speech perception by prelingually deaf children with cochlear implants: A first report. Ear and Hearing, 22, 236-251.
39. Lachs, L., Weiss, J. W., & Pisoni, D. B. (2000). Use of partial stimulus information by cochlear implant users and listeners with normal hearing in identifying spoken words: Some preliminary analyses. The Volta Review, 102(4), 303-320.
40. Liberman, A. M., & Mattingly, I. G. (1985). The motor theory of speech perception revised. Cognition, 21, 1-36.
41. Luce, P. A., & Pisoni, D. B. (1998). Recognizing spoken words: The neighborhood activation model. Ear and Hearing, 19, 1-36.
42. Marslen-Wilson, W. D., & Zwitserlood, P. (1989). Accessing spoken words: The importance of word onsets. Journal of Experimental Psychology: Human Perception and Performance, 15, 576-585.
43. Massaro, D. W. (1987). Speech perception by ear and eye. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 53-84). Hillsdale, NJ: Lawrence Erlbaum Associates.
44. Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
45. Massaro, D. W., & Cohen, M. M. (1995). Perceiving talking faces. Current Directions in Psychological Science, 4, 104-109.
46. Massaro, D. W., & Cohen, M. M. (1999). Speech perception in hearing-impaired perceivers: Synergy of multiple modalities. Journal of Speech, Language and Hearing Research, 42, 21-41.
47. Massaro, D. W., & Stork, D. G. (1998). Speech recognition and sensory integration: a 240-year-old theorem helps explain how people and machines can integrate auditory and visual information to understand speech. American Scientist, 86, 236-244.
48. Mattys, S. L., Bernstein, L. E., & Auer, E. T., Jr. (2002). Stimulus-based lexical distinctiveness as a general word-recognition mechanism. Perception and Psychophysics, 64(4), 667-679.
49. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748.
50. McLennan, C. T., Luce, P. A., & Charles-Luce, J. (2003). Representation of lexical form. Journal of Experimental Psychology: Learning, Memory and Cognition, 29(4), 539-553.
51. Miller, G. A., & Nicely, P. (1955). An analysis of perceptual confusions among some English consonants. Journal of the Acoustical Society of America, 27(2), 338-352.
52. Morton, J. (1979). Word recognition. In J. Morton & J. C. Marshall (Eds.), Structures and processes (pp. 109-156). Cambridge: MIT Press.
53. Nygaard, L. C., Sommers, M. S., & Pisoni, D. B. (1994). Speech perception as a talker-contingent process. Psychological Science, 5(1), 42-46.
54. Oldfield, R. C. (1966). Things, words, and the brain. Quarterly Journal of Experimental Psychology, 18, 340-353.
55. Palmeri, T. J., Goldinger, S. D., & Pisoni, D. B. (1993). Episodic encoding of voice attributes and recognition memory for spoken words. Journal of Experimental Psychology: Learning, Memory and Cognition, 19, 309-328.
56. Peterson, G. E., & Lehiste, I. (1960). Duration of syllable nuclei in English. Journal of the Acoustical Society of America, 32(6), 693-703.
57. Pierrehumbert, J. (2001). Exemplar dynamics: Word frequency, lenition, and contrast. In J. Bybee & P. Hopper (Eds.), Frequency effects and the emergence of lexical structure (pp. 137-157). Amsterdam: John Benjamins.
58. Pisoni, D. B., & Levi, S. V. (in press). Representations and representational specificity in speech perception and spoken word recognition. In M. G Gaskell (Ed.), Handbook of psycholinguistics. Oxford: Oxford University Press.
59. Port, R., & Leary, A. (2005). Against formal phonology. Language, 81(4), 927-964.
60. Ronquest, R. E., Levi, S. V., & Pisoni, D. B. (2007). Language identification from visual-only speech. In Research on Spoken Language Processing Report No. 28 (Vol. 95-118). Bloomington, IN: Speech Research Laboratory, Indiana University.
61. Rosenblum, L. D. (2005). Primacy of multimodal speech perception. In D. B. Pisoni & R. E. Remez (Eds.), Handbook of speech perception (pp. 51-78). Malden, MA: Blackwell.
62. Rosenblum, L. D., Miller, R. M., & Sanchez, K. (2007). Lip-read me now, hear me better later: Cross-modal transfer of talker-familiarity effects. Psychological Science, 18(5), 392-396.
63. Sheffert, S. M., & Fowler, C. A. (1995). The effects of voice and visible speaker change on memory for spoken words. Journal of Memory and Language, 34, 665-685.
64. Sheffert, S. M., Lachs, L., & Hernandez, L. R. (1997). The Hoosier audiovisual multi-talker database. In Research on spoken language processing progress report no. 21 (pp. 578-583). Bloomington, IN: Speech Research Laboratory, Indiana University.
65. Shipman, D. W., & Zue, V. W. (1982). Properties of large lexicons: Implications for advanced isolated word recognition systems. Paper presented at the IEEE 1982 International Conference on Acoustics, Speech and Signal Processing.
66. Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005). Listening to talking faces: Motor cortical activation during speech perception. Neuroimage, 25, 76-89.
67. Slowiaczek, L. M., Nusbaum, H. C., & Pisoni, D. B. (1987). Phonological priming in auditory word recognition. Journal of Experimental Psychology: Learning, Memory and Cognition, 13(1), 64-75.
68. Soto-Faraco, S., Navarra, J., Weikum, W. M., Vouloumanos, A., Sebastián-Galles, N., & Werker, J. F. (2007). Discriminating languages by speech reading. Perception and Psychophysics, 69(2), 218-237.
69. Sumby W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.
70. Summerfield, A. Q. (1979). Use of visual information in phonetic perception. Phonetica, 36, 314-331.
71. Summerfield, A. Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 3-52). Hillsdale, NJ: Lawrence Erlbaum Associates.
72. Treiman, R. (1986). The division between onsets and rimes in English syllables. Journal of Memory and Language, 25, 476-491.
73. Treiman, R., & Danis, C. (1988). Short-term memory errors for spoken syllables are affected by the linguistic structure of the syllables. Journal of Experimental Psychology: Learning, Memory and Cognition, 14, 145-152.
74. van Wassenhove, V, Grant, K. W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences, 102(4), 1181-1186.
75. van Wassenhove, V, Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neuropsychologia, 45, 598-607.
76. Vatakis, A., & Spence, C. (2007). Crossmodal binding: Evaluating the 'unity assumption' using audiovisual speech stimuli. Perception & Psychophysics, 69, 744-756.
77. Vitevitch, M. S. (2002). Influence of onset density on spoken word recognition. Journal of Experimental Psychology: Human Perception and Performance, 28(2), 270-278.
78. Vitevitch, M. S., & Luce, P. A. (1998). When words compete: Levels of processing in spoken word perception. Psychological Science, 9, 325-329.
79. Vitevitch, M. S., & Luce, P. A. (1999). Probabilistic phonotactics and neighborhood activation in spoken word recognition. Journal of Memory and Language, 40, 374-08.
80. Vitevitch, M. S., Luce, P. A., Pisoni, D. B., & Auer, E. T., Jr. (1999). Phonotactics, neighborhood activation and lexical access for spoken words. Brain and Language, 68, 306-311.
81. Weikum, W. M., Vouloumanos, A., Navarra, J., Soto-Faraco, S., Sebastián-Galles, N., & Werker, J. F. (2007). Visual language discrimination in infancy. Science, 316(5828), 1159.
82. Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26, 23-43.