Emphatic visual speech synthesis

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 3, 2009, Pages 459-468

  Melenchón, Javier ^c   Martínez, Elisa ^a   Torre, Fernando De La ^b   Montero, José A ^a  

^a UNIVERSITAT RAMON LLULL   (Spain)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c UNIVERSITAT OBERTA DE CATALUNYA   (Spain)

Author keywords

Audiovisual speech synthesis; Emphatic visual speech; Talking head

Indexed keywords

AUDIOVISUAL SPEECH SYNTHESIS; BEHAVIORAL SYNTHESIS; COMMUNICATION STYLES; EMPHATIC VISUAL-SPEECH; FACIAL SYNTHESIS; HUMAN BEING; HUMAN SUBJECTS; PERCEPTUAL TEST; REAL SAMPLES; RESEARCH AREAS; STATISTICAL INTERACTION; TALKING HEAD; TALKING HEADS; UNIT SELECTION; VISUAL SPEECH SYNTHESIS; VOCAL-TRACTS;

SPEECH SYNTHESIS;

EID: 70350442426     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2010213     Document Type: Article

References (60)

1. E. André, "The generation of multimedia presentations, " in A Handbook of Natural Language Process.ing: Techniques and Applications for the Process.ing of Language as Text, R. Dale, H. Moisl, and H. Somers, Eds. New York: Marcel Dekker, 2000, pp. 305-327.
2. J. Morton and M. Johnson, "Conspec and conlern: A two-process theory of infant face recognition, " Psychol. Rev., vol. 98, no. 2, pp. 164-181, 1991.
3. R. A. Hinde, Non-Verbal Communication. Cambridge, U.K.: Cambridge Univ. Press, 1975.
4. W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, "Face recognition: A literature survey, " ACM Comput. Surv., vol. 35, no. 4, pp. 399-458, 2003.
5. D. Massaro, "Auditory visual speech processing, " in Proc. Eurospeech, Aalborg, Denmark, 2001, pp. 1153-1156.
6. T. Chen, "Audiovisual speech processing: Lip reading and lip synchronization, " IEEE Signal Process. Mag., vol. 18, no. 1, pp. 9-31, Jan. 2001.
7. D. Massaro, J. Beskow, M. Cohen, C. Fry, and T. Rodriguez, "Picture my voice: Audio to visual synthesis using artificial neural networks, " in Proc. AVSP, Santa Cruz, NM, 1999, pp. 133-138.
8. G. Bailly, "Audiovisual speech synthesis, " in Proc. ETRW Speech Synth., 2001 [Online]. Available: citeseer.ist.psu.edu/ bailly03audiovisual.html.
9. J. Ostermann and A. Weissenfeld, "Talking faces-Technologies and applications, " in Proc. ICPR'04: Pattern Recognition, 17th Int. Conf., Washington, DC, 2004, vol. 3, pp. 826-833.
10. F. Parke, "Computer generated animation of faces, " in Proc. ACM'72: ACM Annual Conf., New York, 1972, pp. 451-457.
11. Y. Pei and H. Zha, "Transferring of speech movements from video to 3D face space, " IEEE Trans. Vis. Comput. Graphics, vol. 13, no. 1, pp. 58-69, Jan. 2007.
12. Y. Chang and T. Ezzat, "Transferable videorealistic speech animation, " in Proc. SCA'05: 2005 ACMSIGGRAPH/Eurographics Symp. Comput. Animation, New York, 2005, pp. 143-151.
13. A. Hunt and A. Black, "Unit selection in a concatenative speech synthesis system using a large speech database, " in Proc. ICASSP, Atlanta, GA, 1996, vol. 1, pp. 373-376.
14. M. Cohen and D. Massaro, "Modeling coarticulation in synthetic visual speech, " Models and Techniques in Computer Animation, pp. 139-156, 1993.
15. E. Cantu-Paz, Efficient and Accurate Parallel Genetic Algorithms. Norwell, MA: Kluwer, 2000.
16. F. D. la Torre and M. Black, "Robust parameterized component analysis: Theory and applications to 2d facial appearance models, " Comput. Vis. Image Underst., vol. 91, no. 1-2, pp. 53-71, 2003.
17. P. M. Hall, D. R. Marshall, and R. Martin, "Adding and subtracting eigenspaces with eigenvalue decomposition and singular value decomposition, " IVC, vol. 20, no. 13-14, pp. 1009-1016, Dec. 2002.
18. E. Cosatto, "Sample-based talking-head synthesis, " Ph.D. dissertation, Swiss Federal Inst. of Technol., Lausanne, Switzerland, 2002.
19. T. Ezzat, G. Geiger, and T. Poggio, "Trainable videorealistic speech animation, " ACM Trans. Graph., vol. 21, no. 3, pp. 388-398, 2002.
20. H. McGurk and J. McDonald, "Hearing lips and seeing voices, " Nature, vol. 264, pp. 746-748, 1976.
21. C. Bregler, M. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio, " Proc. SIGGRAPH, pp. 353-360, 1997.
22. J. Beskow, "Talking heads: Models and applications for multimodal speech synthesis, " Ph.D. dissertation, Dept. of Speech, Music, Hearing, KTH, Stockholm, Sweden, 2003.
23. R. Gutiérrez-Osuna, P. Kakumanu, A. Espósito, O. García, A. Bojórquez, J. Castello, and I. Rudomin, "Speech-driven facial animation with realistic dynamics, " IEEE Trans. Multimedia, vol. 7, no. 1, pp. 33-42, Feb. 2005.
24. T. Beier and S. Neely, "Feature-based image metamorphosis, " in Proc. 19th Annu. Conf. Comput. Graphics Interactive Techniques (SIGGRAPH), Chicago, IL, 1992, pp. 35-42.
25. H. Hill, N. Troje, and A. Johnston, "Range- and domain-specific exaggeration of facial speech, " J. Vision, vol. 5, no. 10, pp. 793-807, 2005.
26. J. Beskow, B. Granström, and D. House, "Visual correlates to prominence in several expressive modes, " in Proc. Interspeech, Pittsburg, PA, 2006, pp. 1272-1275.
27. A. Syrdal, "Development of a female voice for a concatenative text-tospeech synthesis system, " Current Topics Acoust. Res., pp. 169-181, 1994.
28. A. Black, "Perfect synthesis for all of the people all of the time, " in Proc. 2002 IEEE Workshop Speech Synth., 2002, pp. 167-170.
29. J. Lasseter, "Principles of animation as applied to 3D character animation, " Comput. Graphics, vol. 21, pp. 35-44, 1987.
30. L. Redman, How To Draw Caricatures. New York: McGraw-Hill, 1984.
31. P. Rautek, I. Viola, and M. Gröller, "Caricaturistic visualization, " IEEE Trans. Vis. Comput. Graphics, vol. 12, no. 5, pp. 1085-1092, Sep./Oct. 2006.
32. B. Lindblom, "Explaining phonetic variation: A sketch of the h and h theory, " Speech Production and Speech Modelling, vol. 55, pp. 403-439, 1990.
33. P. Ekman, About Brows: Emotional and Conversational Signals. Cambridge, U.K.: Cambridge Univ. Press, 1979, pp. 169-248.
34. o variations, " in Proc. Int. Conf. Spoken Lang. Process., 1996, pp. 2175-2179.
35. M. Swerts and E. Krahmer, "Congruent and incongruent audiovisual cues to prominence, " in Proc. Speech Prosody Conf., Nara, Japan, 2004, pp. 69-72.
36. R. Scarborough, P.Keating, M. Baroni, T. Cho, S. Mattys, A. Alwan, E. Auer, and L. Bernstein, "Optical cues to the visual perception of lexical and phrasal stress in English, " in Proc. Int. Conf. Speech Prosody, 2006, pp. 217-220.
37. E. Krahmer and M. Swerts, "Testing the effect of audiovisual cues to prominence via a reaction-time experiment, " in Int. Conf. Spoken Lang. Process., Pittsburg, PA, 2006, 2006, paper 1288-Mon3A3O.4.
38. G. Pourtois, D. Debatisse, P. Despland, and B. de Gelder, "Facial expressions modulate the time course of long latency auditory brain potentials, " Cognitive Brain Res., vol. 14, no. 1, pp. 99-105, 2002.
39. M. Aylett, "Synthesising hyperarticulation in unit selectiontts, " Proc. Interspeech, pp. 2521-2524, 2005.
40. R. Guaus and I. Iriondo, "Diphone based unit selection for catalan text-to-speech synthesis, " in Workshop Text, Speech, Dialogue, Brno, Czech Republic, 2000, pp. 277-282.
41. X. Huang, H.-W. Hon, and R. Reddy, Spoken Language Process.ing: A Guide to Theory, Algorithm, and System Development. Upper Saddle River, NJ: Prentice-Hall, 2001.
42. C. Fisher, "Confusions among visually perceived consonants, " J. Speech Hearing Res., vol. 11, pp. 796-804, 1968.
43. A. Summerfield, Hearing by Eye: The Psychology of Lip-Reading. Hillsdale, NJ: Lawrence Erlbaum Associates, 1987, ch. Some preliminaries to a comprehensive account of audio-visual speech perception, pp. 3-51.
44. T. Cootes, G. Edwards, and C. Taylor, "Active appearance models, " in Proc. Eur. Conf. Comput. Vision (ECCV), Freiburg, Germany, 1998, pp. 581-695.
45. T. Jebara, K. Russell, and A. Pentland, "Mixtures of eigenfeatures for real-time structure from texture, " in Proc. Int. Conf. Computer Vision, Bombay, India, 1998, pp. 128-138.
46. G. Golub and C. V. Loan, Matrix Computations. Baltimore, MD: The Johns Hopkins Univer. Press, 1996.
47. E. Dijkstra, "A note on two problems in connexion with graphs, " Numerische Mathematik, vol. 1, pp. 269-271, 1959.
48. D. Kundur and D. Hatzinakos, "Blind image deconvolution, " IEEE Signal Process. Mag., vol. 13, no. 3, pp. 43-64, May 1996.
49. A. Viterbi, "Error bounds for convolutional codes and an asymptotically optimum decoding algorithm, " IEEE Trans. Inf. Theory, vol. IT-13, pp. 260-269, Apr. 1967.
50. B. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision, " in Proc. Int. Joint Conf. Artif. Intell., 1981, pp. 674-679.
51. M. Black and A. Jepson, "Eigentracking: Robust matching and tracking of articulated objects using a view-based representation, " Int. J. Comput. Vis., vol. 26, no. 1, pp. 63-84, 1998.
52. J. Melenchón, L. Meler, and I. Iriondo, "On-the-fly training, " in Proc. AMDO, Palma de Mallorca, Spain, 2004, pp. 146-153.
53. J. Lim, D. Ross, R. L. ans, and M. H. Yang, "Incremental learning for visual tracking, " Adv. Neural Inf. Process. Syst., pp. 793-800, 2005.
54. A. Ríos, La Transcripción Fonética Automática Del Diccionario Electrónico de Formas Simples Flexivas Del Español: Estudio Fonológico en el Léxico. Barcelono, Spain: Univ. Autónoma de Barcelona, 1999.
55. M. de Vega, C.Álvarez, and M. Carreiras, "Estudio estadístico de la ortografía castellana: La frecuencia silábica, " Cognitiva, vol. 4, no. 1, pp. 75-114, 1992.
56. D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning. Boston, MA: Addison-Wesley, 1989.
57. J. Li, G. Chen, Z. Chi, and C. Lu, "Image coding quality assessment using fuzzy integrals with a three-component image model, " IEEE Trans. Fuzzy Syst., vol. 12, no. 1, pp. 99-106, Feb. 2004.
58. G. Geiger, T. Ezzat, and T. Poggio, "Perceptual evaluation of videorealistic speech, " MIT AIM, 2003, 2003-003.
59. S. Ouni, M. Cohen, H. Ishak, and D. Massaro, "Visual contribution to speech perception: Measuring the intelligibility of animated talking heads, " EURASIP J. Audio, Speech, Music Process., vol. 2007, Article ID 47891.
60. D. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. Boca Raton, FL: Chapman & Hall/CRC, 2007.