Demystifying MMD GANs

6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings

Volumn , Issue , 2018, Pages

  Binkowski, Mikołaj ^a   Sutherland, Dougal J ^b   Arbel, Michael ^b   Gretton, Arthur ^b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADVERSARIAL NETWORKS; CRITIC NETWORK; GENERATOR PARAMETERS; GRADIENT ESTIMATOR; LEARNING RATES; MATCHING PERFORMANCE; TRAINING ALGORITHMS; TRAINING STRATEGY;

EID: 85083954445     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (62)

1. M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017. arXiv:1701.04862.
2. M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In ICML, 2017. arXiv:1701.07875.
3. S. Arora and Y. Zhang. Do GANs actually learn the distribution? An empirical study, 2017. arXiv:1706.08224.
4. S. Arora, R. Ge, Y. Liang, T. Ma, and Y. Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In ICML, 2017. arXiv:1703.00573.
5. M. G. Bellemare, I. Danihelka, W. Dabney, S. Mohamed, B. Lakshminarayanan, S. Hoyer, and R. Munos. The Cramer distance as a solution to biased Wasserstein gradients, 2017. arXiv:1705.10743.
6. Y. Bengio, G. Mesnil, Y. Dauphin, and S. Rifai. Better mixing via deep representations. In ICML, 2013. arXiv:1207.4404.
7. D. Berthelot, T. Schumm, and L. Metz. BEGAN: Boundary equilibrium generative adversarial networks, 2017. arXiv:1703.10717.
8. P. J. Bickel and E. L. Lehmann. Unbiased estimation in convex families. The Annals of Mathematical Statistics, 40(5):1523–1535, 1969.
9. D. Bouchacourt, P. K. Mudigonda, and S. Nowozin. DISCO nets: DISsimilarity COefficients networks. In NIPS, pp. 352–360. 2016.
10. W. Bounliphone, E. Belilovsky, M. B. Blaschko, I. Antonoglou, and A. Gretton. A test of relative similarity for model selection in generative models. In ICLR, 2016. arXiv:1511.04581.
11. D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In ICLR, 2016. arXiv:1511.07289.
12. I. Danihelka, B. Lakshminarayanan, B. Uria, D. Wierstra, and P. Dayan. Comparison of maximum likelihood and GAN-based training of Real NVPs, 2017. arXiv:1705.05263.
13. G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In UAI, 2015. arXiv:1505.03906.
14. W. Fedus, M. Rosca, B. Lakshminarayanan, A. M. Dai, S. Mohamed, and I. Goodfellow. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. In ICLR, 2018. arXiv:1710.08446.
15. T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. JASA, 102 (477):359–378, 2007.
16. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014. arXiv:1406.2661.
17. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. J. Smola. A kernel two-sample test. JMLR, 13, 2012.
18. I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of Wasserstein GANs. In NIPS, 2017. arXiv:1704.00028.
19. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a Nash equilibrium. In NIPS, 2017. arXiv:1706.08500.
20. R. Huang, S. Zhang, T. Li, and R. He. Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In ICCV, 2017a. arXiv:1704.04086.
21. X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie. Stacked generative adversarial networks. In CVPR, 2017b. arXiv:1612.04357.
22. Y. Jin, K. Zhang, M. Li, Y. Tian, H. Zhu, and Z. Fang. Towards the automatic anime characters creation with generative adversarial networks, 2017. arXiv:1708.05509.
23. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. arXiv:1412.6980.
24. A. Klenke. Probability Theory: A Comprehensive Course. World Publishing Corporation, 2008.
25. A. Krizhevsky. Learning multiple layers of features from tiny images, 2009.
26. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
27. C. Li, D. Alvarez-Melis, K. Xu, S. Jegelka, and S. Sra. Distributional adversarial networks, 2017a. arXiv:1706.09549.
28. C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Póczos. MMD GAN: Towards deeper understanding of moment matching network. In NIPS, 2017b. arXiv:1705.08584.
29. Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In ICML, 2015. arXiv:1502.02761.
30. L. Liu. On the two-sample statistic approach to generative adversarial networks. Master’s thesis, University of Princeton Senior Thesis, April 2017. URL http://arks.princeton.edu/ark:/88435/dsp0179408079v.
31. S. Liu, O. Bousquet, and K. Chaudhuri. Approximation and convergence properties of generative adversarial learning. In NIPS, 2017. arXiv:1705.08991.
32. Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In ICCV, 2015.
33. D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In ICLR, 2017. arXiv:1610.06545.
34. R. Lyons. Distance covariance in metric spaces. The Annals of Probability, 41(5):3051–3696, 2013.
35. B. Mityagin. The zero set of a real analytic function, 2015. arXiv:1512.07276.
36. Y. Mroueh and T. Sercu. Fisher GAN. In NIPS, 2017. arXiv:1705.09675.
37. Y. Mroueh, T. Sercu, and V. Goel. McGan: Mean and covariance feature matching GAN. In ICML, 2017. arXiv:1702.08398.
38. A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429–443, 1997.
39. S. Nowozin, B. Cseke, and R. Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In NIPS, 2016. arXiv:1606.00709.
40. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten-hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 12:2825–2830, 2011.
41. G. Piranian. The Set of Nondifferentiability of a Continuous Function. The American Mathematical Monthly, 73(4):57–61, 1966.
42. A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. arXiv:1511.06434.
43. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006.
44. S. Rosenbaum. Moments of a truncated bivariate normal distribution. JRSS B, 23:405–408, 1961.
45. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In NIPS, 2016. arXiv:1606.03498.
46. D. Sejdinovic, B. K. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and RKHS-based statistics in hypothesis testing. The Annals of Stastistics, 41(5):2263–2291, 2013. arXiv:1207.6076.
47. B. K. Sriperumbudur, K. Fukumizu, A. Gretton, G. R. G. Lanckriet, and B. Schölkopf. Kernel choice and classifiability for RKHS embeddings of probability distributions. In NIPS, 2009a.
48. B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. R. G. Lanckriet. On integral probability metrics, phi-divergences and binary classification, 2009b. arXiv:0901.2698.
49. B. K. Sriperumbudur, A. Gretton, K. Fukumizu, G. R. G. Lanckriet, and B. Schölkopf. Hilbert space embeddings and metrics on probability measures. JMLR, 11:1517–1561, 2010. arXiv:0907.5309.
50. B. K. Sriperumbudur, K. Fukumizu, and G. R. G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. JMLR, 12:2389–2410, 2011. arXiv:1003.0887.
51. B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. R. G. Lanckriet. On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6:1550–1599, 2012.
52. I. Steinwart and A. Christmann. Support Vector Machines. Information Science and Statistics. Springer, 2008.
53. D. J. Sutherland. What are the mean and variance of a 0-censored multivariate normal? Cross Validated answer, 2018. URL https://stats.stackexchange.com/q/326347.
54. D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations, 2017. arXiv:1611.04488.
55. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR, 2014. arXiv:1312.6199.
56. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. In CVPR, 2016. arXiv:1512.00567.
57. G. Székely and M. Rizzo. Testing for equal distributions in high dimension. InterStat, 5, 2004.
58. L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In ICLR, 2016. arXiv:1511.01844.
59. F. Yu, Y. Zhang, S. Song, A. Seff, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop, 2015. arXiv:1506.03365.
60. Z. Zahorski. Sur l’ensemble des points de non-dérivabilité d’une fonction continue. Bulletin de la Société mathématique de France, 2:147–178, 1946.
61. W. Zaremba, A. Gretton, and M. B. Blaschko. B-tests: Low variance kernel two-sample tests. In NIPS, 2013. arXiv:1307.1954.
62. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017. arXiv:1703.10593.