Visual Prosody and Speech Intelligibility: Head Movement Improves Auditory Speech Perception

Psychological Science

Volumn 15, Issue 2, 2004, Pages 133-137

  Munhall, K G ^a,b   Jones, Jeffery A ^b   Callan, Daniel E ^b   Kuratate, Takaaki ^b   Vatikiotis Bateson, Eric ^b,c  

^a QUEEN S UNIVERSITY   (Canada)

^b ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

^c UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 1642405348     PISSN: 09567976     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.0963-7214.2004.01502010.x     Document Type: Article

References (30)

1. Badin, P., Bailly, G., Reveret, L., Baciu, M., Segebarth, C., & Savariaux, C. (2002). Three-dimensional linear articulatory modeling of tongue, lips and face, based on MRI and video images. Journal of Phonetics, 30, 533-553.
2. Bernstein, L.E., Demorest, M.E., & Tucker, P.E. (2000). Speech perception without hearing. Perception & Psychophysics, 62, 233-252.
3. Bernstein, L.E., Eberhardt, S.P., & Demorest, M.E. (1998). Single-channel vibrotactile supplements to visual perception of intonation and stress. Journal of the Acoustical Society of America, 85, 397-405.
4. Cutler, A., Dahan, D., & van Donselaar, W. (1997). Prosody in the comprehension of spoken language: A literature review. Language and Speech, 40, 141-201.
5. Cutler, A., & Otake, T. (1994). Mora or phoneme? Further evidence for language-specific listening. Journal of Memory and Language, 33, 824-844.
6. Cutler, A., & Otake, T. (1999). Pitch accent in spoken-word recognition in Japanese. Journal of the Acoustical Society of America, 105, 1877-1888.
7. Fisher, C.G. (1969). The visibility of terminal pitch contour. Journal of Speech and Hearing Research, 12, 379-382.
8. Graf, H.P., Cosatto, E., Strom, V., & Huang, F.J. (2002, May). Visual prosody: Facial movements accompanying speech. Paper presented at the 5th International Conference on Automatic Face and Gesture Recognition, Washington, DC.
9. Granström, B., House, D., & Lundeberg, M. (1999, August). Prosodic cues in multimodal speech perception. Paper presented at the XIVth International Congress of Phonetic Sciences, San Francisco.
10. Hadar, U., Steiner, T.J., Grant, E.C., & Rose, F.C. (1983). Head movement correlates of juncture and stress at sentence level. Language and Speech, 26, 117-129.
11. Hadar, L., Steiner, T.J., Grant, E.C., & Rose, F.C. (1984). The timing of shifts in head posture during conversation. Human Movement Science, 3, 237-245.
12. Hadar, U., Steiner, T.J., & Rose, F.C. (1984). Involvement of head movement in speech production and its implications for language pathology. Advances in Neurology, 42, 247-261.
13. Hill, H., & Johnston, A. (2001). Categorising sex and identity from the biological motion of faces. Current Biology, 11, 880-885.
14. Horn, B.K.P. (1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, 4, 629-642.
15. House, D. (2002, September). Intonational and visual cues in the perception of interrogative mode in Swedish. Paper presented at the 7th International Conference on Spoken Language Processing, Denver, CO.
16. House, D., Beskow, J., & Granström, B. (2001, September). Timing and interaction of visual cues for prominence in audiovisual speech perception. Paper presented at Eurospeech 2001, Aalborg, Denmark.
17. Jiang, J., Alwan, A., Keating, P.A., Auer, E.T., & Bernstein, L.E. (2002). On the relationship between face movements, tongue movements and speech acoustics. EURASIP Journal on Applied Signal Processing, 11, 1174-1188.
18. Kuratate, T., Yehia, H., & Vatikiotis-Bateson, E. (1998). Kinematics-based synthesis of realistic talking faces. In D. Burnham, J. Robert-Ribes, & E. Vatikiotis-Bateson (Eds.), International Conference on Auditory-Visual Speech Processing (AVSP'98) (pp. 185-190). Terrigal-Sydney, Australia: Causal Productions.
19. Massaro, D.W., Tsuzaki, M., Cohen, M.M., Gesi, A., & Heredia, R. (1993). Bimodal speech perception: An examination across languages. Journal of Phonetics, 21, 445-478.
20. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
21. Nicholson, K.G., Baum, S., Cuddy, L., & Munhall, K.G. (2002). A case of impaired auditory and visual speech prosody perception after right hemisphere damage. Neurocase, 8, 314-322.
22. Risberg, A., & Lubker, J. (1978). Prosody and speechreading. Speech Transmission Laboratory Quarterly Progress Report and Status Report, 4, 1-16.
23. Sekiyama, K., & Tohkura, V. (1991). McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. Journal of the Acoustical Society of America, 90, 1797-1805.
24. Sekiyama, K., & Tohkura, Y. (1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, 427-444.
25. Soto-Faraco, S., Sebastián-Gallés, N., & Cutler, A. (2001). Segmental and suprasegmental mismatch in lexical access. Journal of Memory and Language, 45, 412-432.
26. Srinivasan, R., & Massaro, D. (2002). Synthesis and perception of prosody. Unpublished manuscript, University of California, Santa Cruz.
27. Sumby, W.H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.
28. Thompson, D.M. (1934). On the detection of emphasis in spoken sentences by means of visual, tactual, and visual-tactual cues. Journal of General Psychology, 11, 160-172.
29. Yehia, H.C., Kuratate, T., & Vatikiotis-Bateson, E. (2002). Linking facial animation, head motion and speech acoustics. Journal of Phonetics, 30, 555-568.
30. Yehia, H.C., Rubin, P.E., & Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26, 23-44.