VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track

Computer Graphics Forum

Volumn 34, Issue 2, 2015, Pages 193-204

  Garrido, P ^a   Valgaerts, L ^a   Sarmadi, H ^a   Steiner, I ^b   Varanasi, K ^c   Perez P ^c   Theobalt, C ^a  

^a MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

^b SAARLAND UNIVERSITY   (Germany)

^c Technicolor   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; TRANSLATION (LANGUAGES);

3D SHAPE MODEL; AUDIO ANALYSIS; COMPLEX PROCESSES; HIGH QUALITY; RETRIEVAL METHODS; TV PRODUCTION; VISUAL DISCOMFORT; VISUAL QUALITIES;

VISUAL LANGUAGES;

EID: 84932116100     PISSN: 01677055     EISSN: 14678659     Source Type: Journal    
DOI: 10.1111/cgf.12552     Document Type: Conference Paper

References (50)

1. Anderson R., Stenger B., Wan V., Cipolla R.,: Expressive visual text-to-speech using active appearance models. In Proc. CVPR (2013), pp. 3382-3389. 3, 10
2. Blanz V., Basso C., Vetter T., Poggio T.,: Reanimating faces in images and video. CGF (Proc. EUROGRAPHICS) 22, 3 (2003), 641-650. 3
3. Bregler C., Covell M., Slaney M.,: Video Rewrite: Driving visual speech with audio. In ACM TOG (Proc. SIGGRAPH) (1997), pp. 353-360. 3, 10, 11
4. Burke E.K., Gendreau M., Hyde M., Kendall G., Ochoa G., Ozcan E., QU R.,: Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society 64, 12 (2013), 1695-1724. 6
5. Beeler T., Hahn F., Bradley D., Bickel B., Beardsley P., Gotsman C., Sumner R.W., Gross M.,: High-quality passive facial performance capture using anchor frames. In ACM TOG (Proc. SIGGRAPH) (2011), vol. 30, pp. 75:1-75:10. 3
6. Burke E., Hyde M., Kendall G., Ochoa G., OZCAN E., WOODWARD J.R.,: A Classification of Hyper-heuristic Approaches. Kluwer, 2010. 6
7. Brand M.,: Voice puppetry. In ACM TOG (Proc. SIGGRAPH) (1999), pp. 21-28. 2, 11
8. Boersma P., Weenink D.,: Praat, a system for doing phonetics by computer. Glot International 5, 9/10 (2001), 341-345. 7
9. Bouaziz S., Wang Y., Pauly M.,: Online modeling for realtime facial animation. In ACM TOG (Proc. SIGGRAPH) (2013), vol. 32, pp. 40:1-40:10. 3
10. Chang Y., Ezzat T.,: Transferable video-realistic speech animation. In Proc. SCA (2005), pp. 29-31. 3
11. Cao Y., Faloutsos P., Kohler E., Pighin F.,: Real-time speech motion synthesis from recorded motions. In Proc. SCA (2004), pp. 347-355. 3
12. Cao C., Hou Q., Zhou K.,: Displaced dynamic expression regression for real-time facial tracking and animation. In ACM TOG (Proc. SIGGRAPH) (2014), vol. 33, p. 43. 3
13. Deng Z., NEumann U.,: eFASE: expressive facial animation synthesis and editing with phoneme-isomap controls. In Proc. SCA (2006), pp. 251-260. 3
14. Dale K., Sunkavalli K., Johnson M.K., Vlasic D., Matusik W., Pfister H.,: Video face replacement. In ACM TOG (Proc. SIGGRAPH Asia) (2011), vol. 30, pp. 130:1130:10. 3, 10t
15. Ezzat T., Geiger G., Poggio T.,: Trainable video-realistic speech animation. In ACM TOG (Proc. SIGGRAPH) (2002), pp. 388-398. 3
16. Garrido P., Castro C.,: A flexible and adaptive hyper-heuristic approach for (dynamic) capacitated vehicle routing problems. Fundamenta Informaticae 119, 1 (2012), 29-60. 6
17. Garrido P., Valgaerts L., Rehmsen O., Thormaehlen T., Perez P., Theobalt C.,: Automatic face reenactment. In Proc. CVPR (2014). 3, 10
18. Garrido P., Valgaerts L., WU C., Theobalt C.,: Reconstructing detailed dynamic face geometry from monocular video. In ACM TOG (Proc. SIGGRAPH Asia) (2013), vol. 32, pp. 158:1-158:10. 3, 4, 5, 7
19. Kilborn R.,: Speak my language: Current attitudes to television subtitling and dubbing. Media Culture Society 15, 4 (1993), 641-660. 2
20. Kawai M., Iwao T., Mima D., Maejima A., Morishima S.,: Data-driven speech animation synthesis focusing on realistic inside of the mouth. Journal of Information Processing 22, 2 (2014), 401-409. 3
21. Kshirsagar S., Magnenat-Thalmann N.,: Visyl-lable based speech animation. In CGF (Proc. EUROGRAPHICS) (2003), vol. 22, pp. 632-640. 3
22. Kemelmacher-Shlizerman I., Sankar A., Shechtman E., Seitz S.M.,: Being John Malkovich. In Proc. ECCV (2010), pp. 341-353. 3
23. Lewis J.P., Anjyo K., Rhee T., Zhang M., Pighin F., Deng Z.,: Practice and theory of blendshape facial models. In EUROGRAPHICS STAR report (2014), pp. 199-218. 7
24. LE Goff B., Guiard-Marigny T., Cohen M., Benoit C.,: Real-time analysis-synthesis and intelligibility of talking faces. In ESCA/IEEE Workshop on Speech Synthesis (1994), pp. 53-56. 2
25. Lesner S.A., Kricos P.B.,: Visual vowel and diphthong perception across speakers. Journal of the Academy of Rehabilitative Audiology 14 (1981), 252-258. 2
26. Liu K., Ostermann J.,: Realistic facial expression synthesis for an image-based talking head. In Proc. ICME (July 2011), pp. 1-6. 3, 10
27. Levine S., Theobalt C., Koltun V.,: Real-time prosody-driven synthesis of body language. In ACM TOG (Proc. SIGGRAPH Asia) (2009), vol. 28, pp. 172:1-172:10. 2
28. Li K., XU F., Wang J., Dai Q., Liu Y.,: A data-driven approach for facial expression synthesis in video. In Proc. CVPR (2012), pp. 57-64. 3
29. MA J., Cole R.A., Pellom B.L., Ward W., Wise B.,: Accurate visible speech synthesis based on concatenating variable length motion capture data. IEEE TVCG 12, 2 (2006), 266-276. 3
30. MCGurk H., MacDonald J.,: Hearing lips and seeing voices. Nature 264, 5588 (1976), 746-748. 2
31. Noh J., Neumann U.,: Expression cloning. In ACM TOG (Proc. SIGGRAPH) (2001), pp. 277-288. 3
32. Owens E., Blazek B.,: Visemes observed by the hearing-impaired and normal-hearing adult viewers. Journal of Speech, Language, and Hearing Research 28 (1986), 381-393. 2
33. Pighin F., Szeliski R., Salesin D.,: Resynthesizing facial animation through 3D model-based tracking. In Proc. CVPR (1999), pp. 143-150. 3
34. Slaney M., Covell M.,: FaceSync: A linear operator for measuring synchronization of video facial images and audio tracks. In Proc. NIPS (2000), pp. 814-820. 2
35. Song M., Dong Z., Theobalt C., Wang H., Liu Z., Seidel H.P.,: A generic framework for efficient 2-D and 3-D facial expression analogy. IEEE Trans. Multimedia 9, 7 (2007), 1384-1395. 3
36. Saragih J.M., Lucey S., Cohn J.F.,: Deformable model fitting by regularized landmark mean-shift. IJCV 91, 2 (2011), 200-215. 3
37. Seol Y., Lewis J.P., Seo J., Choi B., Anjyo K., Noh J.,: Spacetime expression cloning for blendshapes. ACM TOG 31, 2 (2012), 14. 11
38. Sumby W., Pollack I.,: Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26, 2 (1954), 212-215. 2
39. Sifakis E., Selle A., Robinson-Mosher A.L., Fedkiw R.,: Simulating speech with a physics-based facial muscle model. In Proc. SCA (2006), pp. 261-270. 3
40. Schödl A., Szeliski R., Salesin D., Essa I.A.,: Video textures. In ACM TOG (Proc. SIGGRAPH) (2000), pp. 489-498. 3
41. Summerfield Q.,: Lipreading and audio-visual speech perception. Philosophical Transactions of the Royal Society Series B: Biological Sciences 335, 1273 (1992), 71-78. 2
42. Theobald B.-J., Matthews I.A., Cohn J.F., Boker S.M.,: Real-time expression cloning using appearance models. In Proc. ICMI (2007), pp. 134-139. 3
43. Taylor S.L., Mahler M., Theobald B.-J., Matthews I.,: Dynamic units of visual speech. In Proc. SCA (2012), pp. 275-284. 2, 3
44. Vlasic D., Brand M., Pfister H., Popovic J.,: Face transfer with multilinear models. In ACM TOG (Proc. SIGGRAPH) (2005), vol. 24, pp. 426-433. 3
45. Weise T., BOUAZIZ S., Li H., Pauly M.,: Realtime performance-based facial animation. In ACM TOG (Proc. SIGGRAPH) (2011), vol. 30, pp. 77:1-77:10. 3
46. Williams L.,: Performance driven facial animation. In ACM TOG (Proc. SIGGRAPH) (1990), vol. 24, pp. 235-242. 3
47. Xiang-Chai J., Xiao J., Hodgins J.,: Vision based control of 3D facial animation. In Proc. SCA (2003), pp. 193-206. 3
48. XU F., Liu Y., Stoll C., Tompkin J., Bharaj G., Dai Q., Seidel H.-P., Kautz J., Theobalt C.,: Video-based characters: Creating new human performances from a multiview video database. In ACM TOG (Proc. SIGGRAPH) (2011), vol. 30, pp. 32:1-32:10. 3
49. Young S., Evermann G., Gales M., Hain T., Kershaw D., Liu X.A., Moore G., Odell J., Ollason D., Povey D., Valtchev V., Woodland P.,: The HTK Book. Cambridge University Engineering Department, 2006. 7
50. Yehia H., Rubin P., Vatikiotis-Bateson E.,: Quantitative association of vocal-tract and facial behavior. Speech Communication 26, 1-2 (1998), 23-43. 2, 3