Can you 'read' tongue movements? Evaluation of the contribution of tongue display to speech understanding

Speech Communication

Volumn 52, Issue 6, 2010, Pages 493-503

  Badin, Pierre ^a   Tarabalka, Yuliya ^a   Elisei, Frédéric ^a   Bailly, Gérard ^a  

^a GIPSA LAB   (France)

Author keywords

Audiovisual speech perception; Augmented speech; ElectroMagnetic Articulography (EMA); Lip reading; Tongue reading; Virtual audiovisual talking head

Indexed keywords

AUDIO-VISUAL SPEECH; AUDIOVISUAL SPEECH PERCEPTION; ELECTROMAGNETIC ARTICULOGRAPHY; LIP READING; TALKING HEADS;

AUDIO ACOUSTICS; AUDITION; CLONING; ELECTROMAGNETISM; LINGUISTICS; SIGNAL ANALYSIS; SIGNAL TO NOISE RATIO;

VIRTUAL REALITY;

EID: 77951260444     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2010.03.002     Document Type: Article

References (30)

1. Badin, P., Serrurier, A., 2006. Three-dimensional linear modeling of tongue: Articulatory data and models. In: Yehia, H.C., Demolin, D., Laboissière, R. (Eds.), Seventh International Seminar on Speech Production, ISSP7. Ubatuba, SP, Brazil, UFMG, Belo Horizonte, Brazil, pp. 395-402.
2. Badin, P., Tarabalka, Y., Elisei, F., Bailly, G., 2008. Can you "read tongue movements"? In: Interspeech 2008. Brisbane, Australia, pp. 2635-2638.
3. Bälter, O., Engwall, O., Öster, A.-M., Kjellström, H., 2005. Wizard-of-Oz test of ARTUR - a computer-based speech training system with articulation correction. In: Seventh International ACM SIGACCESS Conference on Computers and Accessibility. Baltimore, pp. 36-43.
4. Benoît C., Guiard-Marigny T., Le Goff B., and Adjoudani A. Which components of the face do humans and machines best speechread?. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines (1996), Springer-Verlag, Berlin 315-328
5. Benoît C., and Le Goff B. Audio-visual speech synthesis from French text: eight years of models, designs and evaluation at the ICP. Speech Communication 26 (1998) 117-129
6. Benoît C., Mohamadi T., and Kandel S. Effects of phonetic context on audio-visual intelligibility of French. Journal of Speech and Hearing Research 37 (1994) 1195-1203
7. Cathiard M.-A., Lallouache M.T., and Abry C. Does movement on the lips mean movement in the mind ?. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines (1996), Springer Verlag, Berlin 211-219
8. Cornett O. Cued speech. American Annals of the Deaf 112 (1967) 3-13
9. Engwall, O., 2008. Can audio-visual instructions help learners improve their articulation? An ultrasound study of short term changes. In: Interspeech 2008. Brisbane, Australia, pp. 2631-2634.
10. Erber N.P. Auditory-visual perception of speech. Journal of Speech and Hearing Disorders XL (1975) 481-492
11. Fagel, S., Madany, K., 2008. A 3-D virtual head as a tool for speech therapy for children. In: Interspeech 2008. Brisbane, Australia, pp. 2643-2646.
12. Grauwinkel, K., Dewitt, B., Fagel, S., 2007. Visual information and redundancy conveyed by internal articulator dynamics in synthetic audiovisual speech. In: Interspeech'2007 - Eurospeech - 9th European Conference on Speech Communication and Technology. Antwerp, Belgium, pp. 706-709.
13. Hoole, P., Nguyen, N., 1997. Electromagnetic articulography in coarticulation research. Forschungsberichte des Instituts für Phonetik und Spachliche Kommunikation der Universität München, FIPKM, vol. 35, pp. 177-184.
14. IJsseldijk F.J. Speechreading performance under different conditions of video image, repetition, and speech rate. Journal of Speech and Hearing Research 35 (1992) 466-471
15. Kröger, B.J., Graf-Borttscheller, V., Lowit, A., 2008. Two- and three-dimensional visual articulatory models for pronunciation training and for treatment of speech disorders. In: Interspeech 2008. Brisbane, Australia, pp. 2639-2642.
16. Massaro, D.W., Bigler, S., Chen, T., Perlman, M., Ouni, S., 2008. Pronunciation training: the role of eye and ear. In: Interspeech 2008. Brisbane, Australia, pp. 2623-2626.
17. Massaro, D.W., Light, J., 2003. Read my tongue movements: bimodal learning to perceive and produce non-native speech /r/ and /l/. In: Eurospeech 2003. Geneva, Switzerland, pp. 2249-2252.
18. Massaro D.W., and Light J. Using visible speech to train perception and production of speech for individuals with hearing loss. Journal of Speech, Language, and Hearing Research 47 (2004) 304-320
19. Mills A.E. The development of phonology in the blind child. In: Dodd B., and Campbell R. (Eds). Hearing by eye: the psychology of lipreading (1987), Lawrence Erlbaum Associates, London 145-161
20. Montgomery D. Do dyslexics have difficulty accessing articulatory information?. Psychological Research 43 2 (1981)
21. Mulford R. First words of the blind child. In: Smith M.D., and Locke J.L. (Eds). The Emergent Lexicon: The Child's Development of a Linguistic Vocabulary (1988), Academic Press, New-York 293-338
22. Narayanan S.S., Nayak K., Lee S., Sethy A., and Byrd D. An approach to real-time magnetic resonance imaging for speech production. Journal of the Acoustical Society of America 115 4 (2004) 1771-1776
23. Odisio M., Bailly G., and Elisei F. Tracking talking faces with shape and appearance models. Speech Communication (Special Issue on Audio Visual Speech Processing) 44 1-4 (2004) 63-82
24. Perkell J.S., Cohen M.M., Svirsky M.A., Matthies M.L., Garabieta I., and Jackson M.T.T. Electromagnetic midsagittal articulometer systems for transducing speech articulatory movements. Journal of the Acoustical Society of America 92 (1992) 3078-3096
25. Serrurier A., and Badin P. A three-dimensional articulatory model of the velum and nasopharyngeal wall based on MRI and CT data. Journal of the Acoustical Society of America 123 4 (2008) 2335-2355
26. Stoel-Gammon C. Prelinguistic vocalizations of hearing-impaired and normally hearing subjects. A comparison of consonantal inventories. Journal of Speech and Hearing Disorders 53 (1988) 302-315
27. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26 2 (1954) 212-215
28. Tye-Murray N., Kirk K.I., and Schum L. Making typically obscured articulatory activity available to speechreaders by means of videofluoroscopy. NCVS Status and Progress Report 4 (1993) 41-63
29. Vihman M.M., Macken M.A., Miller R., Simmons H., and Miller J. From babbling to speech: a re-assessment of the continuity issue. Language 61 2 (1985) 397-445
30. Wik, P., Engwall, O., 2008. Can visualization of internal articulators support speech perception? In: Interspeech 2008. Brisbane, Australia, pp. 2627-2630.