A comparison of acoustic coding models for speech-driven facial animation

Speech Communication

Volumn 48, Issue 6, 2006, Pages 598-615

  Kakumanu, Praveen ^a   Esposito, Anna ^b   Garcia, Oscar N ^c   Gutierrez Osuna, Ricardo ^d  

^a Wright State University   (United States)

^b SECOND UNIVERSITY OF NAPLES   (Italy)

^c UNIVERSITY OF NORTH TEXAS   (United States)

^d TEXAS A AND M UNIVERSITY   (United States)

Author keywords

Audio visual mapping; Linear discriminants analysis; Speech driven facial animation

Indexed keywords

ANIMATION; COMPUTER SIMULATION; GESTURE RECOGNITION; LINEAR SYSTEMS; NATURAL FREQUENCIES; SPEECH ANALYSIS;

AUDIO VISUAL MAPPING; LINEAR DISCRIMINANTS ANALYSIS; SPECTRAL ENERGY; SPEECH DRIVEN FACIAL ANIMATION;

FACE RECOGNITION;

EID: 33747766904     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2005.09.005     Document Type: Article

References (63)

1. Abry C., Boe L.J., and Schwartz J.L. Plateaus, catastrophes and the structuring of vowel systems. J. Phonet. 17 (1989) 47-54
2. Arslan L.M., and Talkin D. Codebook based face point trajectory synthesis algorithm using speech input. Speech Commun. 27 (1999) 81-93
3. Arun K.S., Huang T.S., and Blostein S.D. Least-square fitting of two 3-d point sets. IEEE Trans. PAMI 9 5 (1987) 698-700
4. Aversano, G., Esposito, A., Esposito, A., Marinaro, M., 2001. A new text-independent method for phoneme segmentation. In: Proc. IEEE-MWSCAS Conference, Dayton, OH, pp. 516-519.
5. Balan, N., 2003. Analysis and Evaluation of Factors Affecting Speech Driven Facial Animation, MS Thesis, Dept. of Computer Science and Engineering, Wright State University.
6. Benoit, C., Le Goff, B., 1998. audio-visual speech synthesis from French text: eight years of models, designs, and evaluation at the ICP, Speech Commun. 26, 117-129.
7. Bernstein, L.E., Benoit, C., 1996. For speech perception by humans or machines, three senses are better than one. In: Proc. ICSLP, Philadelphia 3, pp. 1477-1480.
8. Beskow, J., 1995. Rule-based visual speech synthesis, Proc. EUROSPEECH, Madrid, Spain 1, pp. 299-302.
9. Boll S.F. Suppression of acoustic noise in speech using spectral subtraction. IEEE Trans. ASSP 27 2 (1979) 112-113
10. Brand, M., 1999. Voice puppetry, Proc. SIGGRAPH, LA, California, pp. 21-28.
11. Bregler, C., Omohundro, S., 1995. Nonlinear image interpolation using manifold learning. In: Tesauro, G., Touretzky, D., Leen, T. (Eds.), Advances in Neural Information Processing Systems 7, MIT press, Cambridge, pp. 401-408.
12. Bryll, R., Ma, X., Quek, F., 1999. Camera calibration utility description, VisLab Tech. Rep., University of Illinois at Chicago.
13. Caldognetto, E.M., Vagges, K., Borghese, N.A., Ferrigno, G., 1989. Automatic analysis of lip and jaw kinematics in VCV sequences. In: Proc. of EUROSPEECH, Paris 2, pp. 453-456.
14. Cohen M., and Massaro D.W. Modeling coarticulation in synthetic visual speech. In: Thalmann N.M., and Thalmann D. (Eds). Models and Techniques in Computer Animation (1993), Springer 141-155
15. Coianiz, T., Torresani, L., Caprile, L., 1995. 2D deformable models for visual speech analysis. In: Stork, D., Hennecke, M. (Eds.), Speech Reading by Man and Machine. Springer, pp. 391-398.
16. Duda R.O., Hart P.E., and Stork D.G. Pattern Classification. second ed. (2001), Wiley, New York
17. Duttweiler, D., Messerschmitt, D., 1976. Nearly instantaneous companding for nonuniformly quantized PCM. In: IEEE Trans. on Comm., COM-24, pp. 864-873.
18. Essa, I., 1995. Analysis, interpretation, and synthesis of facial expression, Ph.D. thesis, MIT Media Arts and Sciences, Cambridge, MA.
19. Ezzat T., and Poggio T. Visual speech synthesis by morphing visemes. J. Comput. Vis. 38 1 (2000) 45-57
20. Finn, K., 1986. An investigation of visible lip information to be used in automatic speech recognition, Ph.D. dissertation, Dept. CS, Georgetown University, Washington, DC.
21. Fu, S., 2002. Visual Mapping Based on Hidden Markov Models, MS Thesis, Dept. of Computer Science and Engineering, Wright State University.
22. Fu, S., Gutierrez-Osuna, R., Esposito, A., Kakumanu, P.K., Garcia, O.N., 2005. Audio/Visual Mapping with Cross-Modal Hidden Markov Models. IEEE Transactions on Multimedia 7, No. 2, April.
23. Garofolo, J., Lamel, L.F., Fisher, W.M., Fiscus, J.G., Pellet, D.S., Dahlgren, N.L, 1988. The DARPA TIMIT CDROM. Available from LDC: .
24. Goldschen, A.J., 1993. Continuous automatic speech recognition by lipreading, Ph.D. thesis, George Washington University.
25. Greenberg S., Carvey H., Hitchcock L., and Chang S. Temporal properties of spontaneous speech-a syllable-centric perspective. J. Phonet. 31 3-4 (2003) 465-485
26. Gutierrez-Osuna, R., Kakumanu, P., Esposito, A., Garcia, O.N., Bojorquez, A., Castillo, J., Rudomin, I., 2002. WSU Technical.Report CS-WSU-02-03, Dayton, OH.
27. Gutierrez-Osuna R., Kakumanu P.K., Esposito A., Garcia O.N., Bojorquez A., Castillo J.L., and Rudomin I.J. Speech-driven Facial Animation with Realistic Dynamics. IEEE Trans. Multimedia 7 1 (2005) 33-42
28. Hermansky H., and Morgan N. RASTA processing of speech. IEEE Trans. SAP 2 4 (1994) 578-589
29. Hong P., Wen Z., and Huang T.S. Real-time speech-driven face animation with expressions using neural networks. IEEE Trans. Neural Networks 13 4 (2002) 916-927
30. Itakura, F., 1975. Line spectrum representation of linear prediction coefficients of speech signal, JASA57, pp. 535 (abstract).
31. Jourlin P., Luettin J., Genoud D., and Wassner H. Acoustic-labial speaker verification. Patt. Rec. Lett. 18 (1997) 853-858
32. Kass M., Witkin A., and Terzopoulos D. Snakes: active contour models. Int. J. Comput. Vis. 1 4 (1988) 321-331
33. Kim H., and Lee H. Interlacing properties of line spectrum pair frequencies. IEEE Trans. SAP 7 (1999) 87-91
34. Klatt, D.H., 1982. Prediction of perceived phonetic distance from critical band spectra: a first step. In: Proc. of ICASSP, Paris, pp. 1278-1281.
35. Kühnert, B., Nolan, F., 1999. The origin of coarticulation, in Coarticulation: theory, data, and techniques. In: Harcastle, W., Helwett, N. (Eds.), Cambridge University Press, pp. 7-29.
36. Lavagetto F. Converting speech into lip movements: a multimedia telephone for hard of hearing people. IEEE Trans. Rehab. Eng. 3 1 (1995) 90-102
37. Lee, Y., Terzopoulos, D., Waters, K., 1995. Realisitc modeling for facial animation. In: Proc. of SIGGRAPH, LA, California, pp. 55-62.
38. Leps¢y S., and Curinga S. Conversion of articulatory parameters into active shape model coefficients for lip motion representation and synthesis. Signal Process. Image Commun. 13 (1998) 209-225
39. Luttin, J., Thacher, N.A., Beet, S.W., 1996. Active shape models for visual speech feature extraction. In: Stork, D., Hennecke, M. (Eds.), Speech-Reading by Man and Machine, vol. 150. Springer, pp. 383-390.
40. Markel J., and Gray A. Linear Prediction of Speech (1976), Springer
41. Massaro D.W. Perceiving Talking Faces: From Speech Perception to a Behavioral Principle (1997), MIT Press
42. Massaro, D.W., Beskow, J., Cohen, M.M., Fry, C.L., Rodriquez, T., 1999. Picture my voice: audio to visual speech synthesis using Artificial Neural Networks. In: Proc. AVSP, Santa Cruz, CA, pp. 133-138.
43. McAllister D.F., Rodman R.D., Bitzer D.K., and Freeman A.S. Speaker independence in automated lip-sync for audio-video communication. Comput. Networks ISDN Syst. 30 (1998) 1975-1980
44. Montgomery A., and Jackson P. Physical characteristics of lips underlying vowel lipreading performances. JASA 73 6 (1983) 2134-2144
45. Morishima S., and Harashima H. A Media conversion from speech to facial image for man-machine interface. IEEE J. Selected Areas Commun. 9 4 (1991) 594-600
46. Nakamura S., and Yamamoto E. Speech-to-lip movements synthesis by maximizing audio-visual joint probability. J. VLSI Signal Proc. 27 (2001) 119-126
47. Parke F.I. Parameterized models for facial animation. IEEE Comput. Graph. Appl. 2 9 (1982) 61-68
48. Parsons T.W. Voice and Speech Processing (1986), McGraw Hill (Chapter 3)
49. Pelachaud C., Badler N.I., and Steedman M. Generating facial expressions for speech. Cognit. Sci. 20 (1996) 1-46
50. Picone J.W. Signal modeling techniques in speech recognition. Proc. IEEE 81 9 (1993) 1215-1247
51. Rabiner L.R., and Schafer R.W. Digital Processing of Speech Signals (1978), Prentice-Hall
52. Rogozan A., and Deléglise P. Adaptive fusion of acoustic and visual sources for automatic speech recognition. Speech Commun. 26 (1998) 149-161
53. Seneff S. A joint synchrony/mean-rate model of auditory speech processing. J. Phonetics 16 1 (1988) 55-76
54. Sharma, S., Vermeulen, P., Hermansky, H., 1998. Combining information from multiple classifiers to speaker verification. In: Proc. RL2C, France, pp. 115-119.
55. Soong F.K., and Juang B.H. Optimal quantization of line LSP parameters. IEEE Trans. SAP 1 (1993) 15-24
56. Summerfield Q. Use of visual information for phonetic perception. Phonetics 36 (1979) 314-331
57. Tekalp, A.M., Ostermann, J., 2000. Face and2-D mesh animation in MPEG-4. In: Sig. Processing: Image Comm. 15, pp. 387-421.
58. Tibrewala, S., Hermansky, H., 1997. Sub-band based recognition of noisy speech. In: Proc. of ICASSP, Munich, Germany, pp. 1255-1258.
59. Tsai R.Y. A versatile camera calibration technique for high-accuracy 3D machine vision metrology. IEEE J. Robot. Automat. 3 (1987) 323-344
60. Waters, K., Frisbie, J., 1995. A coordinated muscle model for speech animation. In: Proc. of Graphics Interface, Ontario, pp. 163-170.
61. Waters, K., Levergood, T., 1993. DECface: an automatic lip synchronization algorithm for synthetic faces, RLE, Cambridge, MA Tech. Rep. CRL 93/4.
62. Wilson D.R., and Martinez T.R. Reduction techniques for exemplar-based learning algorithms. Mach. Learning 38 3 (2000) 257-286
63. Yamamoto E., Nakamura S., and Shikano K. Lip movement synthesis from speech based on Hidden Markov models. Speech Commun. 28 (1998) 105-115