Photorealistic facial texture inference using deep neural networks

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

Volumn 2017-January, Issue , 2017, Pages 2326-2335

  Saito, Shunsuke ^a,b,c   Wei, Lingyu ^a,b,c   Hu, Liwen ^a,b   Nagano, Koki ^c   Li, Hao ^a,b,c  

^a Santa Monica   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c University of Southern California   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; FREQUENCY ESTIMATION; ITERATIVE METHODS; NEURAL NETWORKS; PATTERN RECOGNITION; RENDERING (COMPUTER GRAPHICS); THREE DIMENSIONAL COMPUTER GRAPHICS;

ANALYSIS TECHNIQUES; CONVEX COMBINATIONS; DEEP CONVOLUTIONAL NEURAL NETWORKS; FACE RECONSTRUCTION; FEATURE CORRELATION; HIGH FREQUENCY HF; HIGH-RESOLUTION TEXTURES; INITIAL ESTIMATION;

DEEP NEURAL NETWORKS;

EID: 85041916909     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2017.250     Document Type: Conference Paper

References (53)

1. B. Amberg, A. Blake, and T. Vetter. On compositional image alignment, with an application to active appearance models. In IEEE CVPR, pages 1714-1721, 2009.
2. J. T. Barron and J. Malik. Shape, albedo, and illumination from a single image of an unknown object. CVPR, 2012.
3. T. Beeler, B. Bickel, P. Beardsley, B. Sumner, and M. Gross. High-quality single-shot capture of facial geometry. ACM Trans. on Graphics (Proc. SIGGRAPH), 29(3):40:1-40:9, 2010.
4. Berkeley Vision and Learning Center. Caffenet, 2014. https://github.com/BVLC/caffe/tree/master/models/bvlc-reference-caffenet.
5. V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187-194, 1999.
6. J. Booth, A. Roussos, S. Zafeiriou, A. Ponniah, and D. Dunaway. A 3d morphable model learnt from 10, 000 faces. In IEEE CVPR, 2016.
7. C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Faceware-house: A 3d facial expression database for visual computing. IEEE TVCG, 20(3):413-425, 2014.
8. C. Cao, H. Wu, Y. Weng, T. Shao, and K. Zhou. Real-time facial animation with image-based dynamic avatars. ACM Transactions on Graphics (TOG), 35(4):126, 2016.
9. P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, and W. Sarokin. Acquiring the Reflectance Field of a Human Face. In SIGGRAPH, 2000.
10. R. Donner, M. Reiter, G. Langs, P. Peloschek, and H. Bischof. Fast active appearance model search using canonical correlation analysis. IEEE TPAMI, 28(10):1690-1694, 2006.
11. C. N. Duong, K. Luu, K. G. Quach, and T. D. Bui. Beyond principal components: Deep boltzmann machines for face modeling. In IEEE CVPR, pages 4786-4794, 2015.
12. G. J. Edwards, C. J. Taylor, and T. F. Cootes. Interpreting face images using active appearance models. In Proceedings of the 3rd. International Conference on Face and Gesture Recognition, FG '98, pages 300-. IEEE Computer Society, 1998.
13. A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH'01, pages 341-346. ACM, 2001.
14. A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In IEEE ICCV, pages 1033-, 1999.
15. L. A. Gatys, M. Bethge, A. Hertzmann, and E. Shechtman. Preserving color in neural artistic style transfer. CoRR, abs/1606.05897, 2016.
16. L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In IEEE CVPR, pages 2414-2423, 2016.
17. A. Ghosh, G. Fyffe, B. Tunwattanapong, J. Busch, X. Yu, and P. Debevec. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graph., 30(6):129:1-129:10, 2011.
18. A. Ghosh, T. Hawkins, P. Peers, S. Frederiksen, and P. Debevec. Practical modeling and acquisition of layered facial reflectance. In ACM Transactions on Graphics (TOG), Volume 27, page 139. ACM, 2008.
19. A. Golovinskiy, W. Matusik, H. Pfister, S. Rusinkiewicz, and T. Funkhouser. A statistical model for synthesis of detailed facial geometry. ACM Trans. Graph., 25(3):1025-1034, 2006.
20. P. Graham, B. Tunwattanapong, J. Busch, X. Yu, A. Jones, P. Debevec, and A. Ghosh. Measurement-based Synthesis of Facial Microgeometry. In EUROGRAPHICS, 2013.
21. A. Haro, B. Guenterz, and I. Essay. Real-time, Photorealistic, Physically Based Rendering of Fine Scale Human Skin Structure. In S. J. Gortle and K. Myszkowski, editors, Eurographics Workshop on Rendering, 2001.
22. A. E. Ichim, S. Bouaziz, and M. Pauly. Dynamic 3d avatar creation from hand-held video input. ACM Trans. Graph., 34(4):45:1-45:14, 2015.
23. V. Kazemi and J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In IEEE CVPR, pages 1867-1874, 2014.
24. I. Kemelmacher-Shlizerman. Internet-based morphable model. IEEE ICCV, 2013.
25. I. Kemelmacher-Shlizerman and R. Basri. 3d face reconstruction from a single image using a single reference face shape. IEEE TPAMI, 33(2):394-405, 2011.
26. V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: Image and video synthesis using graph cuts. In ACM SIGGRAPH 2003 Papers, SIGGRAPH'03, pages 277-286. ACM, 2003.
27. C. Li, K. Zhou, and S. Lin. Intrinsic face image decomposition with human face priors. In ECCV (5)'14, pages 218-233, 2014.
28. H. Li, L. Trutoiu, K. Olszewski, L. Wei, T. Trutna, P.-L. Hsieh, A. Nicholls, and C. Ma. Facial performance sensing head-mounted display. ACM Transactions on Graphics (Proceedings SIGGRAPH 2015), 34(4), July 2015.
29. C. Liu, H.-Y. Shum, and W. T. Freeman. Face hallucination: Theory and practice. Int. J. Comput. Vision, 75(1):115-134, 2007.
30. D. S. Ma, J. Correll, and B. Wittenbrink. The chicago face database: A free stimulus set of faces and norming data. Behavior Research Methods, 47(4):1122-1135, 2015.
31. W.-C. Ma, T. Hawkins, P. Peers, C.-F. Chabert, M. Weiss, and P. Debevec. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Eurographics Symposium on Rendering, 2007.
32. S. P. Mallick, T. E. Zickler, D. J. Kriegman, and P. N. Belhumeur. Beyond lambert: Reconstructing specular surfaces using color. In IEEE CVPR, pages 619-626, 2005.
33. I. Matthews and S. Baker. Active appearance models revisited. Int. J. Comput. Vision, 60(2):135-164, 2004.
34. U. Mohammed, S. J. D. Prince, and J. Kautz. Visio-lization: Generating novel facial images. In ACM SIGGRAPH 2009 Papers, pages 57:1-57:8. ACM, 2009.
35. K. Nagano, G. Fyffe, O. Alexander, J. Barbič, H. Li, A. Ghosh, and P. Debevec. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics (Proceedings SIGGRAPH 2015), 34(4), 2015.
36. K. Olszewski, J. J. Lim, S. Saito, and H. Li. High-fidelity facial and speech animation for vr hmds. ACM Transactions on Graphics (Proceedings SIGGRAPH Asia 2016), 35(6), December 2016.
37. D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In IEEE CVPR, 2016.
38. P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter. A 3d face model for pose and illumination invariant face recognition. In Advanced video and signal based surveillance, 2009. AVSS'09. Sixth IEEE International Conference on, pages 296-301. IEEE, 2009.
39. R. Ramamoorthi and P. Hanrahan. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 497-500. ACM, 2001.
40. D. Ramanan. Face detection, pose estimation, and landmark localization in the wild. In IEEE CVPR, pages 2879-2886, 2012.
41. S. Romdhani. Face image analysis using a multiple features fitting strategy. PhD thesis, University of Basel, 2005.
42. S. Romdhani and T. Vetter. Estimating 3d shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In CVPR (2), pages 986-993, 2005.
43. S. Saito, T. Li, and H. Li. Real-time facial segmentation and performance capture from rgb input. In ECCV, 2016.
44. F. Shi, H.-T. Wu, X. Tong, and J. Chai. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graph., 33(6):222:1-222:13, 2014.
45. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.
46. Solid Angle, 2016. http://www.solidangle.com/arnold/.
47. S. Suwajanakorn, I. Kemelmacher-Shlizerman, and S. M. Seitz. Total moving face reconstruction. In ECCV, pages 796-812. Springer, 2014.
48. The Digital Human League. Digital Emily 2.0, 2015. http://gl.ict.usc.edu/Research/DigitalEmily2/.
49. J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In IEEE CVPR, 2016.
50. M. Turk and A. Pentland. Eigenfaces for recognition. J. Cognitive Neuroscience, 3(1):71-86, 1991.
51. L.-Y. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH'00, pages 479-488, 2000.
52. T. Weyrich, W. Matusik, H. Pfister, B. Bickel, C. Donner, C. Tu, J. McAndless, J. Lee, A. Ngan, H. W. Jensen, and M. Gross. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006), 25(3):1013-1024, 2006.
53. X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li. Face alignment across large poses: A 3d solution. CoRR, abs/1511.07212, 2015.