Converting Speech into Lip Movements: A Multimedia Telephone for Hard of Hearing People

IEEE Transactions on Rehabilitation Engineering

Volumn 3, Issue 1, 1995, Pages 90-102

  Lavagetto, Fabio ^a  

^a UNIVERSITY OF GENOA   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ANIMATION; COMPUTATIONAL GEOMETRY; COMPUTER GRAPHICS; COMPUTER SOFTWARE; HANDICAPPED PERSONS; MATHEMATICAL MODELS; NEURAL NETWORKS; SPEECH COMMUNICATION; TELEPHONE HEARING AIDS; TELEPHONE SYSTEMS; USER INTERFACES;

HEARING IMPAIRED PEOPLE; LIP MOVEMENTS; MULTIMEDIA INTERPERSONAL COMMUNICATION; MULTIMEDIA TELEPHONE; NONLINEAR SIGNAL PROCESSING; PILOT TELEMATIC SERVICE; RELAY MEDIATION SERVICES; SPEECH CONVERSION;

HUMAN REHABILITATION EQUIPMENT;

EID: 0029270677     PISSN: 10636528     EISSN: None     Source Type: Journal    
DOI: 10.1109/86.372898     Document Type: Article

References (50)

1. S. Curinga, A. A. Grattarola, and F. Lavagetto, “Synthesis and animation of human faces: Artificial reality in interpersonal video communication” in Proc. IFIP TC 5/WG 5.10 Corf. Modeling and Computer Graphics, Genova, Italy, June 28-July 2, 1993.
2. K. Aizawa, H. Harashima, and T. Saito, “Model-based analysis-synthesis image coding (MBASIC) system for person’s face” Image Commun., vol. 1, pp. 139-152, 1989.
3. Y. Nakaya, Y. C. Chuah, and H. Harashima, “Model-based/waveform hybrid coding for videotelephone images” Proc. ICASSP-91, San Francisco, CA, pp. 2741-2744, 1991.
4. S. Morishima and H. Harashima, “A media conversion from speech to facial image for intelligent man-machine interface” IEEE J. Select. Areas Commun., vol. 9, no. 4, pp. 594-600, 1991.
5. H. H. Bothe, G. Lindner, and F. Rieger, “The development of a computer animation program for the teaching of lipreading” in Proc. 1st TIDE Conf., Bruxelles, pp. 45-49, 1993.
6. S. Murakami and M. Kumazaki, “Lip reading by 3-D features for model based image coding” in Proc. PCS-93, Lausanne (CH), p. 2.5, 1993.
7. D. R. Hill, A. Pearce, and B. Wyvill, “Animating speech: An automated approach using speech synthesised by rules” Visual Computer, vol. 3, pp. 176-186, 1988.
8. F. Lavagetto, D. Arzarello, and M. Caranzano, “Lipreadable frame animation driven by speech parameters” in IEEE Int. Sympos. Speech, Image Processing and Neural Networks, Hong Kong, Apr. 14-16, 1994.
9. S. Morishima, K. Aizawa, and H. Harashima, “Model-based facial image coding controlled by the speech parameter” in Proc. PCS-8S, Turin, vol. 4, no. 4, 1988.
10. A. Q. Summerfield, “Use of visual information for phonetic perception” Phonetica, vol. 36, pp. 314-331, 1979.
11. N. P. Erber, “Auditory, visual and auditory-visual recognition of consonants by children with normal and impaired hearing” J. Speech Hearing Res., vol. 15, pp. 413-422, 1972.
12. E. Owens and B. Blazek, “Visems observed by hearing-impaired and normal-hearing adult viewers” J. Speech Hearing Res., vol. 28, pp. 381-393, 1985.
13. E. Magno-Caldognetto, K. Vagges, N. A. Borghese, and G. Ferrigno, “Automatic analysis of lips and jaw kinematics in VCV sequences” in Proc. Eurospeech'92, pp. 453-456, 1992.
14. E. Magno-Caldognetto, K. Vagges, G. Ferrigno, and M. G. Busá, “Liprounding coarticulation in Italian” in Proc. Eurospeech'92, pp. 61-64, 1992.
15. E. Magno-Caldognetto, K. Vagges, G. Ferrigno, and C. Zmarich, “Articulatory dynamics of lips in Italian /'VpV/ and /'VbV/ sequences” in Proc. Eurospeech'93, 1993.
16. R. Jakobson, G. Fant, and M. Halle, Preliminaries to Speech Analysis. The Distinctive Features and Their Correlates. Cambridge, IA: IIT Press, 1951.
17. N. Chomsky and M. Halle, Sound Pattern of English. New York: Harper and Row, 1968.
18. P. Ladefoged, “Phonetics prerequisites for a distinctive feature theory” in Papers in Linguistics and Phonetics to the Memory of Pierre Delattre. The Hague: Mouton, 1972, pp. 273-285.
19. P. Ladefoged, A Course in Phonetics. New York: Harcourt, Brace, Jovanovich, Inc., 1975.
20. P. Ladefoged, “What are linguistic sounds made of?” Language, vol. 56, pp. 485-502, 1980.
21. K. S. Lashley, “The problem of serial order in behavior” in Cerebral Mechanisms in Behavior, L. A. Jeffres, Ed. New York: 1951, pp. 112-136.
22. H. W. Buckingham and H. Hollien, “A neural model for language and speech” J. Phonetics, vol. 6, p. 283, 1978.
23. A. G. Mlcoch and D. J. Noll, “Speech production models as related to the concept of apraxia of speech” in Speech and Language. Advances in Basic Research and Practise, vol. IV. New York: Academic Press, 1980, pp. 201-238.
24. J. A. S. Kelso, “The process approach to understanding human motor behavior: An introduction” in Human Motor Behavior: An Introduction, J. A. S. Kelso, Ed. Hillsdale, NJ: Lawrence Erlbaum Assoc. Pub., 1982, pp. 3-19.
25. W. A. Wickelgren, “Context-sensitive coding, associative memory and serial order in speech behavior” Psychol. Rev., vol. 76, pp. 1-15, 1969.
26. C. A. Fowler, “Coarticulation and theories of extrinsic timing” J. Phonetics, 1980.
27. S. Oehman, “Numerical models of coarticulation” J ASA, vol. 41, pp. 310-320, 1967.
28. P. F. MacNeilage, “Motor control of serial ordering of speech” Psychol. Rev., vol. 77, pp. 182-196, 1970.
29. J. W. Folkins and J. H. Abbs, “Lip and jaw motor control during speech: Motor reorganization response to external interference” JSHR, vol. 18, pp. 207-220, 1975.
30. J. H. Abbs and G. L. Eilenberg, “Peripheral mechanism of speech motor control” in Contemporary Issue in Experimental Phonetics, N. J. Lass, Ed. 1976, pp. 139-166.
31. J. F. Lubker, “Representation and context sensitivity” in The Cognitive Representation of Speech. Amsterdam: North Holland, 1981, pp. 127-131.
32. T. H. Gay, “Temporal and spatial properties of articulatory movements: Evidence for minimum spreading across and maximum effects within syllable boundaries” in The Cognitive Representation of Speech. Amsterdam: North Holland, 1981, pp. 133-138.
33. N. Bemstein, The Coarticulation and Regulation of Movements. London: Pergamon Press, 1967.
34. R. Hammarberg, “The metaphysics of coarticulation” J. Phonetics, vol. 4, pp. 353-363, 1976.
35. R. D. Kent and F. D. Minifie, “Coarticulation in recent speech production models”J. Phonetics, vol. 5, pp. 115-133, 1977.
36. O. Fujimura, “Elementary gestures and temporal organization. What does an articulatory constraint mean?” in The Cognitive Representation of Speech. Amsterdam: North Holland, 1981, pp. 101-110.
37. A. P. Benguerel and M. K. Pichora-Fuller, “Coarticulation effects in lipreading” J. Speech Hearing Res., vol. 25, pp. 600-607, 1982.
38. S. G. Nooteboom, “The target theory of speech production” IPO Annu. Progr. Rep., vol. 5, pp. 51-55, 1970.
39. K. N. Stevens and J. S. Perkell, “Speech physiology and phonetic features” in Dynamic Aspects of Speech Production. Tokyo: University of Tokyo Press, 1977, pp. 323-341.
40. J. S. Perkell, “Phonetic features and the physiology of speech production” in Language Production. New York: Academic Press, 1980, pp. 337-372.
41. J. S. Perkell, “On the use of feedback in speech production” in The Cognitive Representation of Speech. Amsterdam: North Holland, 1981, pp. 45-52.
42. N. Kugler, J. A. S. Kelso, and M. T. Turvey, “On the concept of coordinative structures as dissipative structures: I. Theoretical line” in Tutorials in Motor Behavior. Amsterdam: North Holland, 1980, pp. 32-47.
43. B. Pinkowski, “LPC spectral moments for clustering acoustic transients” IEEE Trans. Speech Audio Processing, vol. 1, no. 3, pp. 362-368, 1993.
44. B. P. Yuhas, M. H. Goldstein, Jr., and T. J. Sejnowski, “Integration of acoustic and visual speech signal using neural networks” IEEE Commun. Mag., pp. 65-71, 1989.
45. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, “Phoneme recognition using time delay neural networks” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, no. 3, pp. 328-339, 1989.
46. E. Casella, F. Lavagetto, and R. Miani, “A time-delay neural network for speech to lips movements conversion” in Proc. Int. Conf. Artificial Neural Networks, Sorrento, Italy, May 26-27, 1994.
47. P. Ekman and W. V. Friesen, “Facial action coding system” in Consulting Psychologists Press. Palo Alto, CA: Stanford University, 1977.
48. K. Aizawa, H. Harashima, and T. Saito, “Model-based analysis—Synthesis image coding (MBASIC) system for person’s face” Image Commun., vol. 1, pp. 139-152, 1989.
49. F. I. Parke, “Parameterized models for facial animation” IEEE Comput. Graphics, Applicat., vol. 2, pp. 61-68, 1982.
50. E. A. Wan, “Temporal back-propagation for FIR neural networks” in Proc. Int. Joint Conf. Neural Networks, San Diego CA, vol. 1, pp. 575-580, 1990.