Eye movement of perceivers during audiovisual speech perception

Perception and Psychophysics

Volumn 60, Issue 6, 1998, Pages 926-940

  Vatikiotis Bateson, Eric ^a   Eigsti, Inge Marie ^b   Yano, Sumio ^c   Munhall, Kevin G ^d  

^a ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

^b UNIVERSITY OF ROCHESTER   (United States)

^c NHK SCIENCE AND TECHNICAL RESEARCH LABORATORIES   (Japan)

^d QUEEN S UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; COMPARATIVE STUDY; EYE FIXATION; EYE MOVEMENT; HUMAN; JAPAN; PERCEPTION; PHYSIOLOGY; SPEECH PERCEPTION; VISION;

EYE MOVEMENTS; FIXATION, OCULAR; HUMANS; JAPAN; PERCEPTUAL MASKING; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 0032134085     PISSN: 00315117     EISSN: None     Source Type: Journal    
DOI: 10.3758/BF03211929     Document Type: Article

References (51)

1. ABRY, C., LALLOUACHE, M.-T., & CATHIARD, M.-A. (1996). How can coarticulation models account for speech sensitivity to audio-visual desynchronization? In D. Stork & M. Hennecke (Eds.), Speechreading by humans and machines (NATO-ASI Series F: Computer and Systems Sciences, Vol. 150, pp. 247-256). Berlin: Springer-Verlag.
2. BENOÎT, C., LALLOUACHE, [M.-] T., MOHAMADI, T., & ABRY, C. (1992). A set of French visemes for visual speech synthesis. In G. Bailly, C. Benoît, & T. R. Sawalis (Eds.), Talking machines: Theories, models, and designs (pp. 335-348). Amsterdam: Elsevier.
3. BERTELSON, P., & RADEAU, M. (1976). Ventriloquism, sensory interaction, and response bias: Remarks on the paper by Choe, Welch, Gilford, and Juola. Perception & Psychophysics, 19, 531-535.
4. BROOKE, N. M., & SUMMERFIELD, A. Q. (1983). Analysis, synthesis, and perception of visible articulatory movements. Journal of Phonetics, 11, 63-76.
5. CARPENTER, R. H. S. (1988). Movements of the eyes (2nd rev. ed.). London: Pion.
6. CARPENTER, R. H. S. (ED.) (1991). Eye movements. London: Macmillan.
7. CATHIARD, M.-A., LALLOUACHE, M.-T., & ABRY, C. (1996). Does movement on the lips mean movement in the mind? In D. Stork & M. Hennecke (Eds.). Speechreading by humans and machines (NATO-ASI Series F: Computer and Systems Sciences, Vol. 150, pp. 211-219). Berlin: Springer-Verlag.
8. DEMOREST, M., & BERNSTEIN, L. (1992). Sources of variability in speechreading sentences: A generalizability analysis. Journal of Speech & Hearing Research. 35, 876-891.
9. DRIVER, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66-68.
10. EIGSTI, I. M., MUNHALL, K. G., YANO, S., & VATIKIOTIS-BATESON, E. (1995). Effects of listener expectation on eye movement behavior during audiovisual perception. Journal of the Acoustical Society of America, 97, 3286.
11. GAILEY, L. (1987). Psychological parameters of lip-reading skill. In R. Dodd & B. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 115-141). Hillsdale, NJ: Erlbaum.
12. GRAY, H. (1977). Gray's anatomy. New York: Crown.
13. GREEN, K. P., & KUHL, P. K. (1989). The role of visual information in the processing of place and manner features in speech perception. Perception & Psychophysics, 45, 34-42.
14. GREEN, K. P., & KUHL, P. K. (1991). Integral processing of visual place and auditory voicing information during phonetic perception. Journal of Experimental Psychology: Human Perception & Performance, 17, 278-288.
15. GRÜSSER, O.-J., & LANDIS, T. (EDS.) (1991). Visual agnosias and other disturbances of visual perception and cognition (Vision and Visual Dysfunction, Vol. 12). London: Macmillan.
16. JACOBS, A., & LÉVY-SCHOEN, A. (1988). Breaking down saccade latency into central and peripheral processing times in a visual dual- task. In G. Lüer, U. Lass, & J. Shallo-Hoffman (Eds.), Eye movement research: Physiological and psychological aspects (pp. 267-285). Lewiston, NY: Hogrefe.
17. JEFFERS, J., & BARLEY, M. (1971). Speechreading (lipreading). Springfield, IL: C. C. Thomas.
18. LANSING, C. R., & MCCONKIE, G. (1994). A new method for speech-reading research: Tracking observer's eye movements. Journal of the Academy of Rehabilative Audiology, 27, 25-43.
19. LEIGH, R. J., & ZEE, D. S. (1991). Oculomotor disorders. In R. H. S. Carpenter (Ed.), Eye movements (Vision and Visual Dysfunction, Vol. 8, pp. 297-319). London: Macmillan.
20. LUETTIN, J., THACKER, N. A., & BEET, S. W. (1996). Active shape models for visual speech feature extraction. In D. Stork & M. Hennecke (Eds.), Speechreading by humans and machines (NATO-ASI Series F: Computer and Systems Sciences, Vol. 150, pp. 383-390). Berlin: Springer-Verlag.
21. MACLEOD, A., & SUMMERFIELD, Q. (1990). A procedure for measuring auditory and audiovisual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use. British Journal of Audiology, 24, 29-43.
22. MASSARO, D. W. (1987). Speech perception by ear and eye: A paradigm for psychological inquiry. Hillsdale, NJ: Erlbaum.
23. MASSARO, D. W., TSUZAKI, M., COHEN, M. M., GESI, A., & HERIDIA, R. (1993). Bimodal speech perception: An examination across languages. Journal of Phonetics, 21, 445-478.
24. MCGURK, C., & MACDONALD, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
25. MONTGOMERY, A. A., & JACKSON, P. L. (1983). Physical characteristics of the lips underlying vowel lipreading performance. Journal of the Acoustical Society of America, 73, 2134-2144.
26. MORAY, N. (1993). Designing for attention. In A. Baddeley & L. Weiskrantz (Eds.), Attention: Selection, awareness, and control. A tribute to Donald Broadbent (pp. 53-72). Oxford: Oxford University Press, Clarendon Press.
27. MUNHALL, K. G., GRIBBLE, P., SACCO, L., & WARD, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58, 351-362.
28. MUNHALL, K. G., & VATIKIOTIS-BATESON, E. (1998). The moving face during speech communication. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye: Part 2. Advances in the psychology of speechreading and auditory-visual speech (pp. 123-139). Sussex: Taylor & Francis, Psychology Press.
29. PETAJAN, E. D. (1985). Automatic lipreading to enhance speech recognition. In Proceedings: Computer Vision and Pattern Recognition (pp. 40-47). San Francisco: IEEE Computer Society Press.
30. POLYAK, S. L. (1941). The retina. Chicago: University of Chicago Press.
31. POSNER, M. I. (1980). Orienting attention. Quarterly Journal of Experimental Psychology, 32, 3-25.
32. REISBERG, D., MCLEAN, J., & GOLDFIELD, A. (1987). Easy to hear but hard to understand. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lipreading (pp. 97-114). Hillsdale, NJ: Erlbaum.
33. REMEZ, R. E., RUBIN, P. E., BERNS, S. M., PARDO, J. S., & LANG, J. M. (1994). On the perceptual organization of speech. Psychological Review, 101, 129-156.
34. ROSENBLUM, L. D., JOHNSON, J. A., & SALDAÑA, H. M. (1996). Visual kinematic information for embellishing speech in noise. Journal of Speech & Hearing Research, 39, 1159-1170.
35. SEKIYAMA, K., & TOHKURA, Y. [I.] (1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, 427-444.
36. SEKIYAMA, K., TOHKURA, Y. I., & UMEDA, M. (1996). A few factors which affect the degree of incorporating lip-read information into speech perception. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings: ICSLP 96 (Vol. 3. pp. 1481-1484). Newcastle, DE: Citation Delaware.
37. SMEELE, P. M. T. (1996). Psychology of human Speechreading. In D. G. Stork & M. E. Hennecke (Eds.), Speechreading by humans and machines (NATO-ASI Series F: Computer and Systems Sciences, Vol. 150, pp. 3-17). Berlin: Springer-Verlag.
38. SUMBY, W. H., & POLLACK, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.
39. SUMMERFIELD, Q. (1979). Use of visual information for phonetic perception. Phonetics, 36, 314-331.
40. SUMMERFIELD, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lipreading (pp. 3-52). Hillsdale, NJ: Erlbaum.
41. SWISHER, M. V., CHRISTIE, K., & MILLER, S. L. (1989). The reception of signs in peripheral vision by deaf persons. Sign Language Studies, 63, 99-125.
42. VATIKIOTIS-BATESON, E., EIGSTI, I. M., & YANO, S. (1994). Listener eye movement behavior during audiovisual perception. Journal of the Acoustical Society of Japan, 94-3, 679-680.
43. VATIKIOTIS-BATESON, E., EIGSTI, I. M., YANO, S., & MUNHALL, K. G. (1996). Perceiver eye motion during audiovisual perception (ATR Tech. Rep. No. TR-203, pp. 1-33). Kyoto: ATR Human Information Processing Research Laboratories.
44. VATIKIOTIS-BATESON, E., MUNHALL, K. G., HIRAYAMA, M., LEE, Y. C., & TERZOPOULOS, D. (1996). The dynamics of audiovisual behavior in speech. In D. Stork & M. Hennecke (Eds.), Speechreading by humans and machines (NATO-ASI Series, Series F, Computers and Systems Sciences, Vol. 150, pp. 221-232). Berlin: Springer-Verlag.
45. VATIKIOTIS-BATESON, E., & YEHIA, H. C. (1996). Physiological modeling of facial motion during speech. Transactions of the Technical Committee on Psychological and Physiological Acoustics, H-96(65), 1-8.
46. VATIKIOTIS-BATESON, E., & YEHIA, H. C. (1997). Unified model of audible-visible speech production. In EuroSpeech '97: 5th European conference on speech communication and technology.
47. VITKOVITCH, M., & BARBER, P. (1994). Effects of video frame rate on subjects' ability to shadow one of two cometing verbal passages. Journal of Speech & Hearing Research, 37, 1204-1210.
48. WOLFF, G. J., PRASAD, K. V., STORK, D. G., & HENNECKE, M. (1994). Lipreading by neural networks: Visual preprocessing, learning and sensory integration. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6 (pp. 1027-1034). San Francisco: Morgan Kaufmann.
49. YAMADA, M. (1993). [Analysis of human visual information processing mechanisms using eye movement] (Tech. Rep. No. 17). Tokyo: Institute of Television Engineers.
50. YARBUS, A. L. (1967). Eye movements and vision. New York: Plenum.
51. YEHIA, H. C. RUBIN, P. E., & VATIKIOTIS-BATESON, E. (in press). Quantitative association of acoustic, facial, and vocal-tract shapes. Speech Communication.