Visual atribute transfer through deep image analogy

ACM Transactions on Graphics

Volumn 36, Issue 4, 2017, Pages

  Liao, Jing ^a   Yao, Yuan ^b   Yuan, Lu ^a   Hua, Gang ^a   Kang, Sing Bing ^c  

^a MICROSOFT RESEARCH ASIA   (China)

^b SHANGHAI JIAO TONG UNIVERSITY   (China)

^c MICROSOFT RESEARCH   (United States)

Author keywords

Deep matching; Image analogy; Transfer

Indexed keywords

INTERACTIVE COMPUTER GRAPHICS; SEMANTICS;

COARSE-TO-FINE STRATEGY; DEEP MATCHING; DENSE CORRESPONDENCES; IMAGE ANALOGY; NEAREST NEIGHBORS; SEMANTIC STRUCTURES; TRANSFER; VISUAL INFORMATION;

COLOR IMAGE PROCESSING;

EID: 85030765921     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/3072959.3073683     Document Type: Conference Paper

References (65)

1. Mike Tyka Alexander Mordvintsev, Christopher Olah. 2015. Inceptionism: Going Deeper into Neural Networks. (2015). https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html
2. Xiaobo An and Fabio Pellacini. 2010. User-Controllable Color Transfer. Computer Graphics Forum (2010).
3. Michael Ashikhmin. 2003. Fast texture transfer. IEEE Comput. Graph. and Appl. 23, 4 (2003).
4. Simon Baker, Daniel Scharstein Jr.Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1 (2011), 1-31.
5. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graph. (Proc. of SIGGRAPH) 28, 3 (2009).
6. Connelly Barnes, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2010. The Generalized PatchMatch Correspondence Algorithm. In Proc. ECCV.
7. Connelly Barnes, Fang-Lue Zhang, Liming Lou, Xian Wu, and Shi-Min Hu. 2015. PatchTable: Eficient Patch Queries for Large Datasets and Applications. ACM Trans. Graph. (Proc. of SIGGRAPH) 34, 4 (2015).
8. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation by Example. 32, 4 (2013).
9. T. Brox, C. Bregler, and J. Malik. 2009. Large displacement optical flow. In Proc. CVPR.
10. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV.
11. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017. StyleBank: An Explicit Representation for Neural Image Style Transfer. In Proc. CVPR.
12. Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, and Shi-Min Hu. 2009. Sketch2Photo: Internet Image Montage. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 28, 5 (2009).
13. Li Cheng, S.V.N. Vishwanathan, and Xinhua Zhang. 2008. Consistent image analogies using semi-supervised learning. In CVPR.
14. Hamilton Y. Chong, Steven J. Gortler, and Todd Zickler. 2008. A Perception-based Color Space for Illumination-invariant Image Processing. ACM Trans. Graph. (Proc. of SIGGRAPH) 27, 3 (2008).
15. Cusano Claudio, Gasparini Francesca, and Schettini Raimondo. 2012. Color transfer using semantic image annotation, Vol. 8299.
16. Kevin Dale, Micah K. Johnson, Kalyan Sunkavalli, Wojciech Matusik, and Hanspeter Pfister. 2009. Image restoration using online photo collections. In Proc. ICCV.
17. V. Dumoulin, J. Shlens, and M. Kudlur. 2016. A Learned Representation For Artistic Style. arXiv preprint arXiv:1610.07629 (2016).
18. Alexei A Efros and William T Freeman. 2001. Image quilting for texture synthesis and transfer. In Proc. ACM SIGGRAPH. 341-346.
19. Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008. Edgepreserving decompositions for multi-scale tone and detail manipulation. In ACM Trans. Graph. (Proc. of SIGGRAPH), Vol. 27. ACM.
20. Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier. 2016. Split and Match: Example-based Adaptive Patch Sampling for Unsupervised Style Transfer. In Proc. CVPR.
21. Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In Proc. of NIPS.
22. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. A Neural Algorithm of Artistic Style. In Proc. CVPR.
23. Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid Dense Correspondence with Applications for Image Enhancement. ACM Trans. Graph. (Proc. of SIGGRAPH) 30, 4 (2011).
24. Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. 2015. Proposal Flow. arXiv preprint arXiv:1511.05065 (2015).
25. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In Proc. ACM SIGGRAPH.
26. Yinlin Hu, Rui Song, and Yunsong Li. 2016. Eficient Coarse-To-Fine PatchMatch for Large Displacement Optical Flow. In Proc. CVPR.
27. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. Proc. ECCV (2016).
28. Micah K Johnson, Kevin Dale, Shai Avidan, Hanspeter Pfister, William T Freeman, and Wojciech Matusik. 2011. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. on Visualization and Computer Graphics 17, 9 (2011).
29. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proc. of NIPS.
30. Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. 2014. Transient Attributes for High-level Understanding and Editing of Outdoor Scenes. ACM Trans. Graph. (Proc. of SIGGRAPH) 33, 4 (2014).
31. Hochang Lee, Sanghyun Seo, Seungtaek Ryoo, and Kyunghyun Yoon. 2010. Directional Texture Transfer.
32. Chuan Li and Michael Wand. 2016a. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In Proc. CVPR.
33. Chuan Li and Michael Wand. 2016b. Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks. arXiv preprint arXiv:1604.04382 (2016).
34. Jing Liao, Rodolfo S. Lima, Diego Nehab, Hugues Hoppe, Pedro V. Sander, and Jinhui Yu 2014. Automating Image Morphing Using Structural Similarity on a Halfway Domain. ACM Trans. Graph. 33, 5 (2014).
35. Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. SIFT Flow: Dense Correspondence Across Scenes and Its Applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011).
36. Jonathan L Long, Ning Zhang, and Trevor Darrell. 2014. Do convnets learn correspondence?. In Proc. of NIPS. 1601-1609.
37. Or Lotan and Michal Irani. 2016. Needle-Match: Reliable Patch Matching Under High Uncertainty. In Proc. CVPR.
38. David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (2004).
39. Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. arXiv preprint arXiv:1703.07511 (2017).
40. B. D. Lucas and T. Kanade. 1981. An iterative image registration technique with an application to stereo vision. In Proc. of Imaging Understanding Workshop.
41. Francois Pitie, Anil C. Kokaram, and Rozenn Dahyot. 2005. N-Dimensional Probablility Density Function Transfer and Its Application to Colour Transfer. In Proc. ICCV.
42. Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. 2015. Deep Visual Analogy-Making. In Proc. of NIPS.
43. Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. 2001. Color Transfer Between Images. IEEE Comput. Graph. and Appl. 21, 5 (2001).
44. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211-252.
45. B. C. Russell, J. Sivic, J. Ponce, and H. Dessales. 2011. Automatic Alignment of Paintings and Photographs Depicting a 3D Scene. In 3D Representation for Recognition.
46. Ahmed Selim, Mohamed Elgharib, and Linda Doyle. 2016. Painting Style Transfer for Head Portraits Using Convolutional Neural Networks. ACM Trans. Graph. (Proc. of SIGGRAPH) 35, 4 (2016).
47. Eli Shechtman and Michal Irani. 2007. Matching Local Self-Similarities across Images and Videos. In Proc. CVPR.
48. Xiaoyong Shen, Xin Tao, Chao Zhou, Hongyun Gao, and Jiaya Jia. 2016. Regional Foremost Matching for Internet Scene Images. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 35, 6 (2016).
49. YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand. 2014. Style transfer for headshot portraits. ACM Trans. Graph. (Proc. of SIGGRAPH) 33, 4 (2014).
50. Yichang Shih, Sylvain Paris, Frédo Durand, and William T Freeman. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. (Proc. of SIGGRAPH) 32, 6 (2013).
51. Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros. 2011. Data-driven Visual Similarity for Cross-domain Image Matching. ACM Trans. Graph. (Proc. of SIGGRAPH) 30, 6 (2011).
52. Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. 2008. Summarizing visual data using bidirectional similarity. In CVPR.
53. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
54. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local Color Transfer via Probabilistic Segmentation by Expectation-Maximization. In Proc. CVPR.
55. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. arXiv preprint arXiv:1603.03417 (2016).
56. Philippe Weinzaepfel, Jérôme Revaud, Zaid Harchaoui, and Cordelia Schmid. 2013. DeepFlow: Large displacement optical flow with deep matching. In Proc. ICCV.
57. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring Color to Greyscale Images. Proc. ACM SIGGRAPH 21, 3 (2002).
58. Fuzhang Wu, Weiming Dong, Yan Kong, Xing Mei, Jean-Claude Paul, and Xiaopeng Zhang. 2013. Content-Based Colour Transfer. Computer Graphics Forum (2013).
59. Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. 2014. Beyond pascal: A benchmark for 3d object detection in the wild. In Applications of Computer Vision (WACV), 2014 IEEE Winter Conference on. IEEE, 75-82.
60. Gehua Yang, Charles V Stewart, Michal Sofka, and Chia-Ling Tsai. 2007. Registration of challenging image pairs: Initialization, estimation, and decision. IEEE transactions on pattern analysis and machine intelligence 29, 11 (2007).
61. Hongsheng Yang, Wen-Yan Lin, and Jiangbo Lu. 2014. DAISY Filter Flow: A Generalized Discrete Approach to Dense Correspondences. In Proc. CVPR.
62. Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proc. ECCV.
63. Wei Zhang, Chen Cao, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang. 2013. Style Transfer Via Image Component Analysis. IEEE Trans. on Multimedia (2013).
64. Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qixing Huang, and Alexei A. Efros. 2016. Learning Dense Correspondence via 3D-guided Cycle Consistency. In Proc. CVPR.
65. Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. 1994. L-BFGS-B: Fortran Subroutines for Large-Scale Bound Constrained Optimization. (1994).