Speech perception as a multimodal phenomenon

Current Directions in Psychological Science

Volumn 17, Issue 6, 2008, Pages 405-409

  Rosenblum, Lawrence D ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Audiovisual; Lip reading; Multimodal; Speech

Indexed keywords

EID: 57649115619     PISSN: 09637214     EISSN: 14678721     Source Type: Journal    
DOI: 10.1111/j.1467-8721.2008.00615.x     Document Type: Article

References (27)

1. Bernstein, L.E., Auer, E.T., Jr., & Moore, J.K. (2004). (See ss7References). Presents a "late integration" alternative to amodal accounts as well as a different interpretation of the neurophysiological data on multimodal speech perception.
2. Brancazio, L. (2004). (See ss7References). This paper presents experiments showing lexical influences on audiovisual speech responses and discusses multiple explanations.
3. Calvert, G.A., & Lewis, J.W. (2004). Hemodynamic studies of audiovisual interactions. In G.A. Calvert, C. Spence, & B.E. Stein (Eds.), The handbook of multisensory processing (pp. 483-502). Cambridge, MA: MIT Press. Provides an overview of research on neurophysiological responses to speech and nonspeech cross-modal stimuli.
4. Fowler, C.A. (2004). Speech as a supramodal or amodal phenomenon. In G.A. Calvert, C. Spence, & B.E. Stein (Eds.), The handbook of multisensory processing (pp. 189-202). Cambridge, MA: MIT Press. Provides an overview of multimodal speech research and its relation to speech production and the infant multimodal perception literature; also presents an argument for an amodal account of cross-modal speech.
5. Rosenblum, L.D. (2005). (See ss7References). Provides an argument for a primacy of multimodal speech and a modality-neutral (amodal) theory of integration.
6. Bernstein, L.E., Auer, E.T. Jr. Moore, J.K. (2004). Audiovisual speech binding: Convergence or association. In G.A. Calvert, C. Spence B.E. Stein (Eds.), Handbook of multisensory processing (pp. 203 223). Cambridge, MA : MIT Press.
7. Brancazio, L. (2004). Lexical influences in audiovisual speech perception. Journal of Experimental Psychology: Human Perception & Performance, 30, 445 463.
8. Burnham, D., Ciocca, V., Lauw, C., Lau, S. Stokes, S. (2000). Perception of visual information for Cantonese tones. In M. Barlow P. Rose (Eds.), Proceedings of the Eighth Australian International Conference on Speech Science and Technology (pp. 86 91). Canberra : Australian Speech Science and Technology Association.
9. Calvert, G.A., Bullmore, E., Brammer, M.J., Campbell, R., Iversen, S.D., Woodruff, P., et al 1997). Silent lipreading activates the auditory cortex. Science, 276, 593 596.
10. Fowler, C.A. Dekle, D.J. (1991). Listening with eye and hand: Cross-modal contributions to speech perception. Journal of Experimental Psychology: Human Perception & Performance, 17, 816 828.
11. Gentilucci, M. Cattaneo, L. (2005). Automatic audiovisual integration in speech perception. Experimental Brain Research, 167, 66 75.
12. Ghazanfar, A.A., Maier, J.X., Hoffman, K.L. Logothetis, N.K. (2005). Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. The Journal of Neuroscience, 25, 5004 5012.
13. Gordon, M.S. Rosenblum, L.D. (2005). Effects of intra-stimulus modality change on audiovisual time-to-arrival judgments. Perception & Psychophysics, 67, 580 594.
14. Grant, K.W. Seitz, P. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America, 108, 1197 1208.
15. Green, K.P. (1998). The use of auditory and visual information during phonetic processing: Implications for theories of speech perception. In R. Campbell B. Dodd (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 3 25). London : Erlbaum.
16. Massaro, D.W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA : MIT Press.
17. McGurk, H. MacDonald, J.W. (1976). Hearing lips and seeing voices. Nature, 264, 746 748.
18. Munhall, K. Vatikiotis-Bateson, E. (2004). Spatial and temporal constraint on audiovisual speech perception. In G.A. Calvert, C. Spence B.E. Stein (Eds.), The handbook of multisensory processing (pp. 177 188). Cambridge, MA : MIT Press.
19. Musacchia, G., Sams, M., Nicol, T. Kraus, N. (2005). Seeing speech affects acoustic information processing in the human brainstem. Experimental Brain Research, 168, 1 10.
20. Remez, R.E., Fellowes, J.M. Rubin, P.E. (1997). Talker identification based on phonetic information. Journal of Experimental Psychology: Human Perception & Performance, 23, 651 666.
21. Rosenblum, L.D. (2005). The primacy of multimodal speech perception. In D. Pisoni R. Remez (Eds.), Handbook of speech perception (pp. 51 78). Malden, MA : Blackwell.
22. Rosenblum, L.D., Miller, R.M. Sanchez, K. (2007). Lip-read me now, hear me better later: Cross-modal transfer of talker-familiarity effects. Psychological Science, 18, 392 396.
23. Sams, M., Mottonen, R. Sihvonen, T 2005). Seeing and hearing others and oneself talk. Cognitive Brain Research, 23, 429 435.
24. Shimojo, S. Shams, L. (2001). Sensory modalities are not separate modalities: Plasticity and interactions. Current Opinion in Neurobiology, 11, 505 509.
25. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 53 83). London : Erlbaum.
26. Tuomainen, J., Andersen, T.S., Tiippana, K. Sams, M. (2005). Audio-visual speech perception is special. Cognition, 96, B13 B22.
27. Windmann, S. (2004). Effects of sentence context and expectation on the McGurk illusion. Journal of Memory and Language, 50, 212 230.