High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn , Issue , 2018, Pages 8798-8807

  Wang, Ting Chun ^a   Liu, Ming Yu ^a   Zhu, Jun Yan ^b   Tao, Andrew ^a   Kautz, Jan ^a   Catanzaro, Bryan ^a  

^a NVIDIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SEMANTICS;

ADVERSARIAL NETWORKS; HIGH RESOLUTION IMAGE; HIGH-RESOLUTION PHOTOS; OBJECT APPEARANCE; OBJECT CATEGORIES; OBJECT MANIPULATION; SEGMENTATION INFORMATIONS; SEMANTIC LABELS;

COMPUTER VISION;

EID: 85055134349     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2018.00917     Document Type: Conference Paper

References (64)

1. M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein GAN. In International Conference on Machine Learning (ICML), 2017. 2
2. K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3
3. M. Brown, D. G. Lowe, et al. Recognising panoramas. In IEEE International Conference on Computer Vision (ICCV), 2003. 4
4. P. Burt and E. Adelson. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications, 31 (4): 532-540, 1983. 2, 4
5. Q. Chen and V. Koltun. Photographic image synthesis with cascaded refinement networks. In IEEE International Conference on Computer Vision (ICCV), 2017. 1, 2, 3, 4, 5, 6, 8
6. T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu. Sketch2photo: Internet image montage. ACM Transactions on Graphics (TOG), 28 (5): 124, 2009. 3
7. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The Cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 6, 7
8. P. Costa, A. Galdran, M. I. Meyer, M. Niemeijer, M. Abràmoff, A. M. Mendonça, and A. Campilho. End-toend adversarial retinal image synthesis. IEEE Transactions on Medical Imaging, 2017. 7
9. E. Denton, S. Chintala, A. Szlam, and R. Fergus. Deep generative image models using a Laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2015. 2, 4
10. H. Dong, S. Yu, C. Wu, and Y. Guo. Semantic image synthesis via adversarial learning. In IEEE International Conference on Computer Vision (ICCV), 2017. 2
11. A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS), 2016. 2, 3, 5
12. I. Durugkar, I. Gemp, and S. Mahadevan. Generative multiadversarial networks. In International Conference on Learning Representations (ICLR), 2016. 4
13. L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2, 3, 5
14. M. Gharbi, G. Chaurasia, S. Paris, and F. Durand. Deep joint demosaicking and denoising. ACM Transactions on Graphics (TOG), 35 (6): 191, 2016. 3
15. A. Ghosh, V. Kulharia, V. Namboodiri, P. H. Torr, and P. K. Dokania. Multi-agent diverse generative adversarial networks. ArXiv preprint arXiv: 1704. 02906, 2017. 5
16. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2014. 2
17. J. T. Guibas, T. S. Virdi, and P. S. Li. Synthetic medical images from dual generative adversarial networks. ArXiv preprint arXiv: 1709. 01872, 2017. 7
18. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 4
19. X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie. Stacked generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 4
20. S. Iizuka, E. Simo-Serra, and H. Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36 (4): 107, 2017. 4
21. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-toimage translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 3, 5, 6
22. J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV), 2016. 2, 3, 5, 7
23. M. Johnson, G. J. Brostow, J. Shotton, O. Arandjelovic, V. Kwatra, and R. Cipolla. Semantic photo synthesis. In Computer Graphics Forum, volume 25, pages 407-413. Wiley Online Library, 2006. 3
24. T. Kaneko, K. Hiramatsu, and K. Kashino. Generative attribute controller with conditional filtered generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2
25. L. Karacan, Z. Akata, A. Erdem, and E. Erdem. Learning to generate images of outdoor scenes from attributes and semantic layouts. ArXiv preprint arXiv: 1612. 00215, 2016. 2, 5
26. T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ArXiv preprint arXiv: 1710. 10196, 2017. 2
27. D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2013. 6
28. J.-F. Lalonde, D. Hoiem, A. A. Efros, C. Rother, J. Winn, and A. Criminisi. Photo clip art. In ACM Transactions on Graphics (TOG), volume 26, page 3. ACM, 2007. 3
29. A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther. Autoencoding beyond pixels using a learned similarity metric. In International Conference on Machine Learning (ICML), 2016. 5
30. V. Le, J. Brandt, Z. Lin, L. Bourdev, and T. S. Huang. Interactive facial feature localization. In European Conference on Computer Vision (ECCV), 2012. 6
31. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 5
32. J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan. Perceptual generative adversarial networks for small object detec-tion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2
33. M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-toimage translation networks. In Advances in Neural Information Processing Systems (NIPS), 2017. 3
34. M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2016. 3
35. J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 3
36. X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley. Least squares generative adversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017. 6
37. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. In International Conference on Learning Representations (ICLR), 2016. 2
38. M. Mirza and S. Osindero. Conditional generative adversarial nets. ArXiv preprint arXiv: 1411. 1784, 2014. 3
39. P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from RGBD images. In European Conference on Computer Vision (ECCV), 2012. 6, 7
40. D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2, 3
41. A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations (ICLR), 2015. 2
42. O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015. 3
43. L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60 (1-4): 259-268, 1992. 2
44. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems (NIPS), 2016. 2
45. P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays. Scribbler: Controlling deep image synthesis with sketch and color. ArXiv preprint arXiv: 1612. 00835, 2016. 2, 3
46. A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb. Learning from simulated and unsupervised images through adversarial training. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3
47. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ArXiv preprint arXiv: 1409. 1556, 2014. 2
48. B. M. Smith, L. Zhang, J. Brandt, Z. Lin, and J. Yang. Exemplar-based face parsing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013. 6, 7, 8
49. Y. Taigman, A. Polyak, and L. Wolf. Unsupervised crossdomain image generation. In International Conference on Learning Representations (ICLR), 2017. 3
50. S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. ArXiv preprint arXiv: 1707. 04993, 2017. 2
51. H.-Y. F. Tung, A. W. Harley, W. Seto, and K. Fragkiadaki. Adversarial inverse graphics networks: Learning 2D-to-3D lifting and image-to-image translation from unpaired supervision. In IEEE International Conference on Computer Vision (ICCV), 2017. 3
52. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems (NIPS), 2016. 2
53. X. Wang and A. Gupta. Generative image modeling using style and structure adversarial networks. In European Conference on Computer Vision (ECCV), 2016. 2
54. Z. Yi, H. Zhang, P. T. Gong, et al. DualGAN: Unsupervised dual learning for image-to-image translation. In IEEE International Conference on Computer Vision (ICCV), 2017. 3
55. H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 4
56. R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European Conference on Computer Vision (ECCV), 2016. 3
57. R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros. Real-time user-guided image colorization with learned deep priors. In ACM SIGGRAPH, 2017. 3
58. Z. Zhang, Y. Song, and H. Qi. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2
59. H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 6
60. J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. In International Conference on Learning Representations (ICLR), 2017. 2
61. B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ADE20K dataset. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 6, 7, 8
62. J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros. Generative visual manipulation on the natural image manifold. In European Conference on Computer Vision (ECCV), 2016. 2, 3
63. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017. 3, 6
64. J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-toimage translation. In Advances in Neural Information Processing Systems (NIPS), 2017. 2, 5, 6