Relating objective and subjective performance measures for AAM-based visual speech synthesis

IEEE Transactions on Audio, Speech and Language Processing

Volumn 20, Issue 8, 2012, Pages 2378-2387

  Theobald, Barry John ^a   Matthews, Iain ^b  

^a UNIVERSITY OF EAST ANGLIA   (United Kingdom)

^b DISNEY RESEARCH   (United States)

Author keywords

Active appearance models (AAMs); canonical correlation analysis; visual speech evaluation; visual speech synthesis

Indexed keywords

ACOUSTIC FEATURES; ACTIVE APPEARANCE MODELS; CANONICAL CORRELATION ANALYSIS; DYNAMIC TIME; OBJECTIVE MEASURE; PHONETIC TRANSCRIPTIONS; SMALL REGION; SUBJECTIVE PERFORMANCE; SUBJECTIVE QUALITY; VISUAL SPEECH; VISUAL SPEECH SYNTHESIS;

DEGRADATION; SPEECH SYNTHESIS;

QUALITY CONTROL;

EID: 84865358428     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2012.2202651     Document Type: Article

References (82)

1. G. Bailly, M. Bérar, F. Elisei, and M. Odisio, "Audiovisual speech synthesis," Int. J. Speech Technol., vol. 6, pp. 331-346, 2003.
2. J. Beskow, I. Karlsson, J. Kewley, and G. Salvi, "SYNFACE - A talking head telephone for the hearing-impaired," Computers Helping People With Special Needs, pp. 1178-1186, 2004. (Pubitemid 38939738)
3. D. Massaro, Perceiving Talking Faces. Cambridge, MA: MIT Press, 1998.
4. F. Parke, "Parametric models for facial animation," Comput. Graphics Applicat., vol. 2, no. 9, pp. 61-68, 1982.
5. G. Bailly, F. Elisei, P. Badin, and C. Savariaux, "Degrees of freedom of facial movements in face-to-face conversational speech," in Proc. Int. Workshop Multimodal Corpora, 2006, pp. 33-36.
6. R. Gutierrez-Osuna, P. Kakumanu,A. Esposito, O. Garcia, B. A., and I. Rudomin, "Speech-driven facial animation with realistic dynamics," IEEE Trans. Multimedia, vol. 7, no. 1, pp. 33-42, Feb. 2005. (Pubitemid 40178367)
7. B. Theobald, I. Matthews, M. Mangini, J. Spies, T. Brick, J. Cohn, and S. Boker, "Mapping and manipulating facial expression," Lang. Speech, vol. 52, no. 2/3, pp. 369-386, 2009.
8. K. Kähler, J. Haber, and H. Seidel, "Geometry- basedmusclemodelling for facial animation," Graphics Interface, pp. 27-36, 2001.
9. Y. Lee, D. Terzopoulos, and K. Waters, "Realistic modeling for facial animation," in Proc. SIGGRAPH, 1995, pp. 55-62.
10. L. Nedel and D. Thalmann, "Real time muscle deformations using mass-spring systems," Comput. Grahpics Int., pp. 156-165, 1998.
11. M. Brand, "Voice puppetry," in Proc. SIGGRAPH, Los Angeles, CA, 1999, pp. 21-28.
12. Y. Du and X. Lin, "Realistic mouth synthesis based on shape appearance dependence mapping," Pattern Recogn. Lett., vol. 23, no. 14, pp. 1875-1885, 2002.
13. P. Eisert, S. Chaudhuri, and B. Girod, "Speech driven synthesis of talking head sequences," in Proc. Workshop 3D Image Anal. Synth., 1997, pp. 51-56.
14. G. Englebienne, T. Cootes, and M. Rattray, "A probabilistic model for generating realistic speech movements from speech," in Proc. Adv. Neural Inf. Process. Syst., 2007.
15. G. Feldhoffer, A. Tihanyi, and O. Balázs, "A comparative study of direct and ASR-based modular audio to visual speech systems," The Phonetician, vol. 97/98, pp. 15-24, 2008.
16. P. Hong, Z. Wen, and T. Huang, "Real-time speech-driven expressive synthetic talking faces using neural networks," IEEE Trans. Neural Netw., vol. 13, no. 4, pp. 916-927, Jul. 2002.
17. C. Hsieh and Y. Chen, "Partial linear regression for speech-driven talking head application," Signal Process.: Image Commun., vol. 21, pp. 1-12, 2006. (Pubitemid 41653199)
18. D. Massaro, J. Beskow, M. Cohen, T. Fry, and C. Rodriguez, "Picture my voice: Audio to visual speech synthesis using artificial neural networks," in Proc. Int. Conf. Auditory Vis. Speech Process., 1999.
19. B. Theobald and N. Wilkinson, "Real-time visual speech synthesis using active appearance models," in Proc. Int. Conf. Auditory Vis. Speech Process., 2007.
20. L. Wang, W. Han, X. Qian, and F. Soong, "Synthesizing photo-real talking head via trajectory-guided sample selection," in Proc. Interspeech, 2010.
21. Z. Wen, P. Hong, and T. Huang, "Real time speech driven facial animation using formant analysis," in Proc. Int. Conf. Multimedia Expo, 2001, pp. 817-820.
22. C. Bregler, M. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio," in Proc. SIGGRAPH, 1997, pp. 353-360.
23. S.Deena, S. Hou, and A. Galata, "Visual speech synthesis bymodelling coarticulation dynamics using a non-parametric switching state-space model," in Int. Conf. Multimodal Interfaces, 2010, pp. 1-8.
24. J. Edge and A. Hilton, "Visual speech synthesis from 3D video," in IET Eur. Conf. Vis. Media Product., 2006, pp. 174-179.
25. T. Ezzat and T. Poggio, "Miketalk: A talking facial display based on morphing visemes," in Proc.Comput. Animat. Conf., 1998, pp. 96-103.
26. T. Ezzat, G. Geiger, and T. Poggio, "Trainable videorealistic speech animation," in Proc. SIGGRAPH, 2002, pp. 388-398.
27. O. Govokhina, G. Bailly, G. Breton, and P. Bagshaw, "TDA: A new trainable trajectory formation system for facial animation," in Proc. Interspeech, 2006, pp. 2474-2477.
28. F. Huang, E. Cosatto, and H. Graf, "Triphone based unit selection for concatenative visual speech synthesis," in Proc. Int. Conf. Acoust., Speech, Signal Process., 2002, pp. 2037-2040.
29. W.Mattheyses, L. Latacz, andW. Verhelst, "Active appearancemodels for photorealistic visual speech synthesis," in Proc. Interspeech, 2010.
30. B. Theobald, J. Bangham, I. Matthews, and G. Cawley, "Near- videorealistic synthetic talking faces: Implementation and evaluation," Speech Commun., vol. 44, pp. 127-140, 2004.
31. F. Parke and K.Waters, Computer Facial Animation. Natick,MA: A K Peters, 1996.
32. E. Cosatto and H. Graf, "Photo-realistic talking-heads fromimage samples," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 152-163, Jun. 2000.
33. F. Elisei,M. Odisio, G. Bailly, and P. Badin, "Creating and controlling video-realistic talking heads," in Proc. Int. Conf. Auditory Vis. Speech Process., 2001, pp. 90-97.
34. T. Hawkins, A. Wenger, C. Tchou, A. Gardner, F. Goransson, and P. Debevec, "Animatable facial reflectance fields," in Proc. Eurograph. Symp. Rendering, Jun. 2004.
35. V. Blanz, C. Basso, T. Poggio, and T. Vetter, "Reanimating faces in images and video," in Proc. Eurographics, 2003, pp. 641-650.
36. T. Cootes, G. Edwards, and C. Taylor, "Active appearance models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 681-685, Jun. 2001. (Pubitemid 32585279)
37. G. Bailly, N. Campbell, and M. Mbius, "ISCA special session: Hot topics in speech synthesis," in Proc. Eurospeech, 2003, pp. 37-40.
38. B. Theobald, S. Fagel, F. Elsei, and G. Bailly, "LIPS2008: Visual speech synthesis challenge," in Proc. Interspeech, 2008, pp. 1875-1878.
39. G. Geiger, T. Ezzat, and T. Poggio, "Perceptual evaluation of video-realistic speech," Cambrige,MA, Tech. Rep. CBCL Paper 224/AIMemo 2003-003, 2003, MIT.
40. H. Yehia, R. P., and E. Vatikiotis-Bateson, "Quantitative association of vocal-tract and facial behaviour," Speech Commun., vol. 26, pp. 23-43, 1998. (Pubitemid 128381217)
41. P. Hong, Z. Wen, T. Huang, and H. Shum, "Real-time speech-driven 3D face animation," in Proc. 3D Data Process. Vis. Transmission Symp., 2002, pp. 713-716.
42. E. Yamamoto, S.Nakamura, and K. Shikano, "Lipmovement synthesis from speech based on hidden Markov models," in Proc. Int. Conf. Face and Gesture, 1998, pp. 154-159.
43. Z. Deng, U. Neumann, J. Lewis, T. Kim, and S. Narayanan, "Expressive facial animation synthesis by learning speech coarticulation and expression spaces," IEEE Trans. Vis. Comput. Graph., vol. 12, no. 6, pp. 1523-1534, 2006. (Pubitemid 44523095)
44. O. Engwall, "Evaluation of a system for concatenative articulatory visual speech synthesis," in Proc. Int. Conf. Spoken Lang. Process., 2002, pp. 665-668.
45. K. Liu and J. Ostermann, "Realistic facial animation system for interactive services," in Proc. Interspeech, 2008, pp. 2330-2333.
46. J. Ma, R. Cole, B. Pellom, W. Ward, and B. Wise, "Accurate visible speech synthesis based on concatenating variable length motion capture data," IEEE Trans. Vis. Comput. Graphics, vol. 12, no. 2, pp. 266-276, Mar.-Apr. 2006.
47. E. Bevacqua and C. Pelachaud, "Expressive audio-visual speech," Comput. Animation and Virtual Worlds, vol. 15, pp. 297-304, 2004. (Pubitemid 41108008)
48. S. King and R. Parent, "Creating speech-synchronized animation," IEEE Trans. Vis. Comput. Graphics, vol. 11, no. 3, pp. 341-352, May-Jun. 2005. (Pubitemid 40542054)
49. J. Melenchon, E. Martinez, F. De la Torre, and J. Montero, "Emphatic visual speech synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 3, pp. 459-468, Mar. 2009.
50. B. Theobald and N. Wilkinson, "A probabilistic trajectory synthesis system for synthesising visual speech," in Proc. Interspeech, 2008, pp. 2310-2313.
51. J. Tao, L. Xin, and Y. Panrong, "Realistic visual speech synthesis based on hybrid concatenation method," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 3, pp. 469-477, 2009.
52. S. Fu, R. Gutierrez-Osuna, A. Esposito, P. Kakumanu, and O. Garcia, "Audio/visual mapping with cross-modal hidden Markov models," IEEE Trans. Multimedia, vol. 7, no. 2, pp. 243-252, Apr. 2005. (Pubitemid 40454754)
53. S. Nakamura and E.Yamamoto, "Speech-to-lip movement synthesis by maximizing audio-visual joint probability based on the EM algorithm," J. VLSI Signal Process., vol. 27, pp. 119-126, 2001. (Pubitemid 32190760)
54. J. Beskow, "Trainable articulatory control models for visual speech synthesis," J. Speech Technol., vol. 4, no. 7, pp. 335-349, 2004.
55. R. Carlson and B. Granström, "Data-driven multimodal synthesis," Speech Commun., vol. 47, pp. 182-193, 2005. (Pubitemid 41231589)
56. X. Zhuang, L. Wang, F. Soong, and M. Hasegawa-Johnson, "A minimum converted trajectory error (mcte) approach to high quality speech-to-lips conversion," in Proc. Interspeech, 2010.
57. D. Cosker, D. Marshall, P. Rosin, and Y. Hicks, "Speech driven facial animation using a hidden Markov coarticulation model," in Proc. Int. Conf. Pattern Recogn., 2004, pp. 128-131.
58. L. Arslan and D. Talkin, "3D face point trajectory synthesis using an automatically derived visual phoneme similarity matrix," in Proc. Int. Conf. Auditory-Vis. Speech Process., 1998, pp. 175-180.
59. G. Bailly, G. Gibert, and M. Odisio, "Evaluation of movement generation systems using the point-light technique," in Proc. IEEE Workshop Speech Synth., 2002, pp. 27-30.
60. T. Kuratate, K.Munhall, P. Rubin, E. Vatikiotis-Bateson, and H. Yehia, "Audio-visual synthesis of talking faces from speech production correlates," in Proc. Eurospeech, 1999, vol. 3, pp. 1279-1282.
61. J. Dongmei, X. Lei, Z. Rongchun, W. Verhelst, I. Ravyse, and H. Sahli, "Acoustic viseme modelling for speech driven animation: A case study," in Proc. Workshop Model-Based Process. Coding of Audio, 2002, pp. 49-52.
62. S. Cadavid, M. Abdel-Mottaleb, D. Messinger, M. Mahoor, and L. Bahrick, "Detecting local audio-visual synchrony in monologues utilizing vocal pitch and facial landmark trajectories," in Proc. British Mach. Vis. Conf., 2009.
63. I. Pandzic, J. Ostermann, and D. Millen, "User evaluation: Synthetic talking faces for interactive services," Vis. Comput., vol. 15, pp. 330-340, 1999. (Pubitemid 30504265)
64. S. Fagel, G. Bailly, and F. Elisei, "Intelligibility of natural and 3D-cloned German speech," in Proc. Int. Conf. Auditory-Vis. Speech Process., 2007.
65. C. Benoît and B. Le Goff, "Audio-visual speech synthesis from french text: Eight years of models, designs and evaluation at the ICP," Speech Commun., vol. 26, pp. 117-129, 1998. (Pubitemid 128381224)
66. M. Železný, K. Zdeněk, P. Císař, and M. Jindřich, "Design, implementation and evaluation of the czech realistic audio-visual speech synthesis," Signal Process., vol. 86, pp. 3657-3673, 2006.
67. C. Benoît, M. Grice, and V. Hazan, "The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using semantically unpredictable sentences," Speech Commun., vol. 18, no. 4, pp. 381-392, 1996. (Pubitemid 126362996)
68. H. McGurk and J. MacDonald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, 1976.
69. D. Cosker, D. Marshall, P. Rosin, S. Paddock, and S. Rushton, "Towards perceptually realistic talking heads: Models, metrics and Mcgurk," ACM Trans. Appl. Percept., vol. 2, no. 3, pp. 270-285, 2005.
70. P. Dey, S. Maddock, and R. Nicolson, "Evaluation of a viseme-driven talking head," in Proc. 8th Theory Practice Comput. Graphics Conf., 2010, pp. 139-142.
71. S. Fagel and C. Clemens, "An articulation model for audiovisual speech synthesis - Determination, adjustment, evaluation," Speech Commun., vol. 44, pp. 141-154, 2004.
72. S. Fagel, "MASSY speaks English: Adaptation and evaluation of a talking head," Proc. Interspeech, 2008.
73. K. Liu and J. Ostermann, "Optimization of an image-based talking head system," EURASIP J. Audio, Speech, Music Process., vol. 2009, 2009.
74. W. Mattheyses, L. Latacz, and W. Verhelst, "Optimized photorealistic audiovisual speech synthesis using active appearance modeling," in Proc. Int. Conf. Auditory-Vis. Speech Process., 2010, pp. 148-153.
75. K. Choi and J. Hwang, "Automatic creation of a talking head from a video sequence," IEEE Trans. Multimedia, vol. 7, no. 4, pp. 628-637, Aug. 2005. (Pubitemid 41311415)
76. W. Mattheyses, L. Latacz, and W. Verhelst, "On the importance of audiovisual coherence for the perceived quality of synthesized visual speech," EURASIP J. Audio, Speech, Music Process., vol. 2009, pp. 1-12, 2009.
77. G. Takacs, "Direct, modular and hybrid audio to visual speech conversion methods - A comparative study," in Proc. Interspeech, 2009.
78. B. Theobald, "Visual speech synthesis using shape and appearance models," Ph.D. dissertation, Univ. of East Anglia, Norwich, UK, 2003.
79. H. Bredin and G. Chollet, "Audio-visual speech synchrony measure for talking-face identity verification," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Apr. 2007, vol. 2, pp. 233-236.
80. C. de Boor, "Calculation of the smoothing spline with weighted roughness measure," Math. Models Methods in Appl. Sci., vol. 11, no. 1, pp. 33-41, 2001. (Pubitemid 33686843)
81. B. Theobald, N. Wilkinson, and I. Matthews, "On evaluating synthesised visual speech," in Proc. Int. Conf. Auditory Vis. Speech Process., 2008, pp. 7-12.
82. D. Wakerly, W. Mendenhall, and R. Scheaffer, Mathematical Statistics with Applications, ser. Duxbury Advanced Series. Pacific Grove, CA: Duxbury, 2002.