Cross-view image synthesis using geometry-guided conditional GANs

Computer Vision and Image Understanding

Volumn 187, Issue , 2019, Pages

  Regmi, Krishna ^a   Borji, Ali ^b  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

^b MarkableAI   (United States)

Author keywords

Cross view; GANs; Homography; Synthesis; View translation

Indexed keywords

PIXELS; SYNTHESIS (CHEMICAL);

ADVERSARIAL NETWORKS; CROSS-VIEW; GANS; GEOMETRY CONSTRAINTS; HOMOGRAPHIES; MODEL COMPARISON; PIXEL INFORMATION; SYNTHESIS METHOD;

ANTENNAS;

EID: 85070368114     PISSN: 10773142     EISSN: 1090235X     Source Type: Journal    
DOI: 10.1016/j.cviu.2019.07.008     Document Type: Article

References (46)

1. Ardeshir, S., Borji, A., Ego2top: Matching viewers in egocentric and top-view videos. Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, the Netherlands, October 11-14, 2016, Proceedings, Part V, 2016, 253–268, 10.1007/978-3-319-46454-1_16.
2. Borji, A., 2018. Pros and cons of gan evaluation measures. arXiv preprint arXiv:1802.03446.
3. Che, T., Li, Y., Jacob, A.P., Bengio, Y., Li, W., 2016. Mode regularized generative adversarial networks. arXiv e-prints abs/1612.02136. URL: https://arxiv.org/abs/1612.02136.
4. Collobert, R., Kavukcuoglu, K., Farabet, C., Torch7: A matlab-like environment for machine learning. BigLearn, NIPS Workshop, 2011.
5. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B., The cityscapes dataset for semantic urban scene understanding. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
6. Dosovitskiy, A., Springenberg, J.T., Tatarchenko, M., Brox, T., Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 39 (2017), 692–705 URL: http://lmb.informatik.uni-freiburg.de/Publications/2017/DTB17.
7. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets. Advances in Neural Information Processing Systems, 2014, 2672–2680.
8. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., Gans trained by a two time-scale update rule converge to a local nash equilibrium. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., (eds.) Advances in Neural Information Processing Systems, Vol. 30, 2017, Curran Associates, Inc., 6626–6637.
9. Hinton, G.E., Osindero, S., Teh, Y.W., A fast learning algorithm for deep belief nets. Neural Comput. 18 (2006), 1527–1554, 10.1162/neco.2006.18.7.1527.
10. Ioffe, S., Szegedy, C., Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, 2015, JMLR.org, 448–456 URL: http://dl.acm.org/citation.cfm?id=30451183045167.
11. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks. CVPR, 2017.
12. Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J., Learning to discover cross-domain relations with generative adversarial networks. Precup, D., Teh, Y.W., (eds.) Proceedings of the 34th International Conference on Machine Learning, PMLR, 2017, International Convention Centre, Sydney, Australia, 1857–1865 URL: http://proceedings.mlr.press/v70/kim17a.html.
13. Kossaifi, J., Tran, L., Panagakis, Y., Pantic, M., 2017. Gagan: Geometry-aware generative adverserial networks. arXiv preprint arXiv:1712.00684.
14. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Commun. ACM 60 (2017), 84–90, 10.1145/3065386 URL: http://doi.acm.org/10.1145/3065386.
15. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W., Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2017, 105–114, 10.1109/CVPR.2017.19.
16. Lin, T.Y., Belongie, S., Hays, J., Cross-view image geolocalization. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
17. Lin, T., Cui, Y., Belongie, S.J., Hays, J., Learning deep representations for ground-to-aerial geolocalization. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, 2015, 5007–5015, 10.1109/CVPR.2015.7299135.
18. Lin, G., Milan, A., Shen, C., Reid, I., Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. CVPR, 2017.
19. Mathieu, M., Couprie, C., LeCun, Y., 2015. Deep multi-scale video prediction beyond mean square error. CoRR abs/1511.05440. URL: http://arxiv.org/abs/1511.05440, arXiv:1511.05440.
20. Mh, S.H.M.F.R., Lee, N.G.H., Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, 7258–7267.
21. Mirza, M., Osindero, S., 2014. Conditional generative adversarial nets. CoRR abs/1411.1784. URL: http://arxiv.org/abs/1411.1784.
22. Nguyen, T., Le, T., Vu, H., Phung, D., Dual discriminator generative adversarial nets. Advances in Neural Information Processing Systems, 2017, 2670–2680.
23. Palazzi, A., Borghi, G., Abati, D., Calderara, S., Cucchiara, R., Learning to map vehicles into bird's eye view. Battiato, S., Gallo, G., Schettini, R., Stanco, F., (eds.) Image Analysis and Processing - ICIAP 2017, 2017, Springer International Publishing, Cham, 233–243.
24. Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C., 2017. Transformation-grounded image generation network for novel 3d view synthesis. CoRR abs/1703.02921. URL: http://arxiv.org/abs/1703.02921.
25. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A., Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 2536–2544.
26. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H., Generative adversarial text to image synthesis. Balcan, M.F., Weinberger, K.Q., (eds.) Proceedings of the 33rd International Conference on Machine Learning, PMLR, 2016, 1060–1069 URL: http://proceedings.mlr.press/v48/reed16.html.
27. Regmi, K., Borji, A., Cross-view image synthesis using conditional gans. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
28. Regmi, K., Shah, M., 2019. Bridging the domain gap for ground-to-aerial image matching. arXiv preprint arXiv:1904.11045.
29. Salakhutdinov, R., Hinton, G.E., Deep boltzmann machines. International Conference on Artificial Intelligence and Statistics (AISTATS), 2009, 3.
30. Salimans, T., Goodfellow, I.J., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training gans. Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, 2016, 2226–2234 URL: http://Papers.Nips.Cc/Paper/6125-Improved-Techniques-for-Training-Gans.
31. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z., Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 2016, 1874–1883, 10.1109/CVPR.2016.207.
32. Smolensky, P., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. 1986, MIT Press, Cambridge, MA, USA, 194–281 (Chapter). Information Processing in Dynamical Systems: Foundations of Harmony Theory URL: http://dl.acm.org/citation.cfm?id=104279.104290.
33. Song, L., Lu, Z., He, R., Sun, Z., Tan, T., Geometry Guided Adversarial Facial Expression Synthesis. 2017.
34. Soran, B., Farhadi, A., Shapiro, L., Action recognition in the presence of one egocentric and multiple static cameras. Asian Conference on Computer Vision, 2014, Springer, 178–193.
35. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 2016, 2818–2826, 10.1109/CVPR.2016.308.
36. Tatarchenko, M., Dosovitskiy, A., Brox, T., Multi-view 3d models from single images with a convolutional network. Leibe, B., Matas, J., Sebe, N., Welling, M., (eds.) Computer Vision – ECCV 2016, 2016, Springer International Publishing, Cham, 322–337.
37. Theis, L., van den Oord, A., Bethge, M., 2016. A note on the evaluation of generative models, in: International Conference on Learning Representations URL: http://arxiv.org/abs/1511.01844.
38. Vo, N.N., Hays, J., Localizing and Orienting Street Views Using Overhead Imagery, Vol. 49. 2016, Springer International Publishing, Cham, 494–509, 10.1007/978-3-319-46448-0_30.
39. Workman, S., Souvenir, R., Jacobs, N., Wide-area image geolocalization with aerial reference imagery. IEEE International Conference on Computer Vision (ICCV), 2015.
40. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H., High-resolution image inpainting using multi-scale neural patch synthesis. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
41. Yeh, R.A., Chen, C., Lim, T.Y., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N., Semantic image inpainting with deep generative models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 5485–5493.
42. Zhai, M., Bessinger, Z., Workman, S., Jacobs, N., Predicting ground-level scene layout from aerial imagery. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
43. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D., Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. ICCV, 2017.
44. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A., Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 2017.
45. Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A., View synthesis by appearance flow. Leibe, B., Matas, J., Sebe, N., Welling, M., (eds.) Computer Vision – ECCV 2016, 2016, Springer International Publishing, Cham, 286–301.
46. Zhu, J.Y., Park, T., Isola, P., Efros, A.A., Unpaired image-to-image translation using cycle-consistent adversarial networkss. Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.