Painting style transfer for head portraits using convolutional neural networks

ACM Transactions on Graphics

Volumn 35, Issue 4, 2016, Pages

  Selim, Ahmed ^a   Elgharib, Mohamed ^b   Doyle, Linda ^a  

^a TRINITY COLLEGE DUBLIN   (Ireland)

^b QATAR COMPUTING RESEARCH INSTITUTE   (Qatar)

Author keywords

Deformations; Gain maps; Gatys; NPAR; Painting transfer; Portrait; Spatial constraints; VGG; Video

Indexed keywords

CONVOLUTION; DEFORMATION; INTERACTIVE COMPUTER GRAPHICS; NEURAL NETWORKS;

GATYS; NPAR; PORTRAIT; SPATIAL CONSTRAINTS; VIDEO;

PAINTING;

EID: 84980004921     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2897824.2925968     Document Type: Conference Paper

References (56)

1. ASHIKHMIN, M. 2001. Synthesizing natural textures. In ACM Symposium on Interactive 3D Graphics, 217-226.
2. ASHIKHMIN, N. 2003. Fast texture transfer. Computer Graphics and Applications 23, 4, 38-43.
3. BAE, S., PARIS, S., AND DURAND, F. 2006. Two-scale tone management for photographic look. In ACM SIGGRAPH, 637-645.
4. BARRODALE, I., SKEA, D., BERKLEY, M., KUWAHARA, R., AND POECKERT, R. 1993. Warping digital images using thin plate splines. Pattern Recognition 26, 2, 375-376.
5. BEIER, T., AND NEELY, S. 1992. Feature-based image metamorphosis. ACM SIGGRAPH 26, 2, 35-42.
6. CHEN, H., XU, Y.-Q., SHUM, H.-Y., ZHU, S.-C., AND ZHENG, N.-N. 2001. Example-based facial sketch generation with nonparametric sampling. In International Conference on Computer Vision, 433-438.
7. CHEN, H., LIANG, L., QING XU, Y., YEUNG SHUM, H., AND NING ZHENG, N. 2002. Example-based automatic portraiture. In Asian Conference on Computer Vision.
8. CHEN, H., ZHENG, N., LIANG, L., LI, Y., XU, Y.-Q., AND SHUM, H.-Y. 2002. Pictoon: a personalized image-based cartoon system. In ACM Multimedia, 171-178.
9. CHEN, H., LIU, Z., ROSE, C., XU, Y., SHUM, H.-Y., AND SALESIN, D. 2004. Example-based composite sketching of human portraits. In International Symposium on Nonphotorealistic Animation and Rendering, 95-153.
10. COLLOMOSSE, J., AND HALL, P. 2002. Painterly rendering using image salience. In Eurographics, 122-128.
11. COLLOMOSSE, J., AND HALL, P. 2005. Genetic paint: A search for salient paintings. In EvoMUSART, 437-447.
12. COOTES, T. F., TAYLOR, C. J., COOPER, D. H., AND GRAHAM, J. 1995. Active shape model: Their training and application. Computer Vision and Image Understanding 61, 1, 38-59.
13. COOTES, T. F., EDWARDS, G. J., AND TAYLOR, C. J. 1998. Active appearance models. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Springer, 484-498.
14. DIPAOLA, S. 2007. Painterly rendered portraits from photographs using a knowledge-based approach. In SPIE: Human Vision and Imaging.
15. EFROS, A. A., AND FREEMAN, W. T. 2001. Image quilting for texture synthesis and transfer. In ACM SIGGRAPH, 341-346.
16. EFROS, A., AND LEUNG, T. 1999. Texture synthesis by nonparametric sampling. In International Conference on Computer Vision, 1033-1038.
17. GATYS, L. A., ECKER, A. S., AND BETHGE, M. 2015. A neural algorithm of artistic style. CoRR abs/1508.06576.
18. GOOCH, B., COOMBE, G., AND SHIRLEY, P. 2002. Artistic vision: Painterly rendering using computer vision techniques. In International Symposium on Non-photorealistic Animation and Rendering, 83-88.
19. GOOCH, B., REINHARD, E., AND GOOCH, A. 2004. Human facial illustrations: Creation and psychophysical evaluation. SIGGRAPH 23, 1, 27-44.
20. HAEBERLI, P. 1990. Paint by numbers: Abstract image representations. SIGGRAPH Computer Graphics and Interactive Techniques 24, 4, 207-214.
21. HASHIMOTO, R., JOHAN, H., AND NISHITA, T. 2003. Creating various styles of animations using example-based filtering. In Computer Graphics International, 312-317.
22. HAYS, J., AND ESSA, I. 2004. Image and video based painterly animation. In International Symposium on Non-photorealistic Animation and Rendering (NPAR), 113-120.
23. HERTZMANN, A., JACOBS, C. E., OLIVER, N., CURLESS, B., AND SALESIN, D. H. 2001. Image analogies. In SIGGRAPH, 327-340.
24. HERTZMANN, A. 1998. Painterly rendering with curved brush strokes of multiple sizes. In SIGGRAPH, 453-460.
25. HERTZMANN, A. 2001. Paint by relaxation. In Computer Graphics International, 47-54.
26. JOHNSON, J., 2015. Torch implementation of neural style algorithm. https://github.com/jcjohnson/neural-style.
27. KIM, S. Y., MACIEJEWSKI, R., ISENBERG, T., ANDREWS, W. M., CHEN, W., SOUSA, M. C., AND EBERT, D. S. 2009. Stippling by example. In International Symposium on Non-Photorealistic Animation and Rendering, 41-50.
28. KRIZHEVSKY, A., SUTSKEVER, I., AND HINTON, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 1097-1105.
29. KYPRIANIDIS, J. E., COLLOMOSSE, J., WANG, T., AND ISENBERG, T. 2013. State of the art: A taxonomy of artistic stylization techniques for images and video. IEEE Transactions on Visualization and Computer Graphics 19, 5, 866-885.
30. LEE, H., SEO, S., RYOO, S., AND YOON, K. 2010. Directional texture transfer. In International Symposium on Non-Photorealistic Animation and Rendering, 43-48.
31. LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2008. A closedform solution to natural image matting. IEEE Transactions on PAMI 30, 2, 228-242.
32. LI, Y., SHARAN, L., AND ADELSON, E. H. 2005. Compressing and companding high dynamic range images with subband architectures. SIGGRAPH 24, 3, 836-844.
33. LIN, L. 2010. Painterly animation using video semantics and feature correspondence. In NPAR, 73-80.
34. LITWINOWICZ, P. 1997. Processing images and video for an impressionist effect. In SIGGRAH Computer Graphics and Interactive Techniques, 407-414.
35. LIU, C., YUEN, J., TORRALBA, A., SIVIC, J., AND FREEMAN, W. T. 2008. Sift flow: Dense correspondence across different scenes. In European Conference on Computer Vision, 28-42.
36. LOWE, D. G. 1999. Object recognition from local scale-invariant features. In International Conference on Computer Vision, 1150-1157.
37. LU, J., SANDER, P. V., AND FINKELSTEIN, A. 2010. Interactive painterly stylization of images, videos and 3d animations. In ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 127-134.
38. MAHENDRAN, A., AND VEDALDI, A. 2015. Understanding deep image representations by inverting them. In Computer Vision and Pattern Recognition (CVPR), 5188-5196.
39. MCKONE, E., KANWISHER, B., AND DUCHAINE, B. 2007. Can generic expertise explain special processing for faces? Trends in Cognitive Science 1, 8-15.
40. MEIER, B. J. 1996. Painterly rendering for animation. In Computer Graphics and Interactive Techniques, 477-484.
41. MENG, M., ZHAO, M., AND ZHU, S.-C. 2010. Artistic papercut of human portraits. In ACM International Conference on Multimedia (Short Paper), 931-934.
42. O'DONOVAN, P., AND HERTZMANN, A. 2012. Anipaint: Interactive painterly animation from video. IEEE Transactions on Visualization and Computer Graphics 18, 3, 475-487.
43. PITIE, F., KOKARAM, A., AND DAHYOT, R. 2005. N-dimensional probability density function transfer and its application to color transfer. In International Conference on Computer Vision, 1434-1439.
44. REINHARD, E., ADHIKHMIN, M., GOOCH, B., AND SHIRLEY, P. 2001. Color transfer between images. Computer Graphics and Applications 21, 5, 34-41.
45. SARAGIH, J. M., LUCEY, S., AND COHN, J. 2009. Face alignment through subspace constrained mean-shifts. In International Conference of Computer Vision, 1034-1041.
46. SHIH, Y., PARIS, S., BARNES, C., FREEMAN, W. T., AND DURAND, F. 2014. Style transfer for headshot portraits. SIGGRAPH 33, 4, 148:1-148:14.
47. SIMONYAN, K., AND ZISSERMAN, A. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556.
48. SINHA, P., BALAS, B., OSTROVSKY, Y., AND RUSSELL, R. 2006. Face recognition by humans: Nineteen results all computer vision researchers should know about. Proceedings of the IEEE 94, 11, 1948-1962.
49. SU, S. L., DURAND, F., AND AGRAWALA, M. 2005. De-emphasis of distracting image regions using texture power maps. In International Workshop on Texture Analysis and Synthesis in conjunction with ICCV'05, 119-124.
50. TAIGMAN, Y., YANG, M., RANZATO, M., AND WOLF, L. 2014. Deepface: Closing the gap to human-level performance in face verification. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 1701-1708.
51. WANG, X., AND TANG, X. 2009. Face photo-sketch synthesis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 31, 11, 1955-1967.
52. WANG, B., WANG, W., YANG, H., AND SUN, J. 2004. Efficient example-based painting and synthesis of 2d directional texture. IEEE Transactions on Visualization and Computer Graphics 10, 3, 266-277.
53. WANG, T., COLLOMOSSE, J., SLATTER, D., CHEATLE, P., AND GREIG, D. 2010. Video stylization for digital ambient displays of home movies. In International Symposium on Non-Photorealistic Animation and Rendering, 137-146.
54. WANG, T., COLLOMOSSE, J., HUNTER, A., AND GREIG, D. 2013. Learnable stroke models for example-based portrait painting. In British Machine Vision Conference, 36.1-36.11.
55. ZENG, K., ZHAO, M., XIONG, C., AND ZHU, S.-C. 2009. From image parsing to painterly rendering. SIGGRAPH 29, 1 (Dec.), 2:1-2:11.
56. ZHAO, M., AND ZHU, S.-C. 2011. Portrait painting using active templates. In Non-Photorealistic Animation and Rendering (NPAR), 117-124.