Inference suboptimality in variational autoencoders

35th International Conference on Machine Learning, ICML 2018

Volumn 3, Issue , 2018, Pages 1749-1760

  Cremer, Chris ^a   Li, Xuechen ^a   Duvenaud, David ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPLEX NETWORKS;

APPROXIMATE INFERENCE; AUTOENCODERS; LARGE DATASETS; LATENT VARIABLE MODELS; SUBOPTIMALITY; VARIATIONAL PARAMETERS;

LEARNING SYSTEMS;

EID: 85057248218     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, S. Generating Sentences from a Continuous Space. ArXiv e-prints, November 2015.
2. Burda, Y., Grosse, R., and Salakhutdinov, R. Importance weighted autoencoders. In ICLR, 2016.
3. Clevert, Djork-Arne, Unterthiner, Thomas, and Hochreiter, Sepp. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
4. Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using Real NVP. ICLR, 2017.
5. Geweke, John. Bayesian inference in econometric models using monte carlo integration. Econometrica: Journal of the Econometric Society, pp. 1317-1339, 1989.
6. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010.
7. Gomez, Aidan N, Ren, Mengye, Urtasun, Raquel, and Grosse, Roger B. The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems, pp. 2211-2221, 2017.
8. Grosse, R., Ghahramani, Z., and Adams, R. P. Sandwiching the marginal likelihood using bidirectional monte carlo. arXiv preprint arXiv:1511.02543, 2015.
9. Grosse, Roger B, Ancha, Siddharth, and Roy, Daniel M. Measuring the reliability of mcmc inference with bidirectional monte carlo. In Advances in Neural Information Processing Systems, pp. 2451-2459, 2016.
10. Hoffman, Matthew D. Learning deep latent gaussian models with markov chain monte carlo. In International Conference on Machine Learning, pp. 1510-1519, 2017.
11. Hoffman, Matthew D, Blei, David M, Wang, Chong, and Paisley, John. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303-1347, 2013.
12. Jarzynski, C. Nonequilibrium equality for free energy differences. Physical Review Letters, 78(14):2690, 1997.
13. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
14. Kingma, D.P. and Welling, M. Auto-Encoding Variational Bayes. In ICLR, 2014.
15. Kingma, D.P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improving Variational Inference with Inverse Autoregressive Flow. NIPS, 2016.
16. Krishnan, R. G., Liang, D., and Hoffman, M. On the challenges of learning with inference networks on sparse, high-dimensional data. ArXiv e-prints, October 2017.
17. Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. University of Toronto, 2009.
18. Larochelle, Hugo and Bengio, Yoshua. Classification using discriminative restricted boltzmann machines. In Proceedings of the 25th international conference on Machine learning, pp. 536-543. ACM, 2008.
19. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 1998.
20. Maaløe, L., Sønderby, CK., Sønderby, SK., and Winther, O. Auxiliary Deep Generative Models. ICML, 2016.
21. Neal, R.M. Annealed importance sampling. Statistics and Computing, 2001.
22. Ranganath, R., Tran, D., and Blei, D. M. Hierarchical Variational Models. ICML, 2016.
23. Rezende, D.J. and Mohamed, S. Variational Inference with Normalizing Flows. In ICML, 2015.
24. Rezende, D.J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ICML, 2014.
25. Salakhutdinov, R. and Murray, I. On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference on Machine learning, pp. 872- 879. ACM, 2008.
26. Salimans, T., Kingma, D.P., and Welling, M. Markov chain monte carlo and variational inference: Bridging the gap. In ICML, 2015.
27. Sønderby, Casper Kaae, Raiko, Tapani, Maaløe, Lars, Sønderby, Søren Kaae, and Winther, Ole. Ladder variational autoencoders. In Advances in Neural Information Processing Systems, pp. 3738-3746, 2016.
28. Tomczak, J. M. and Welling, M. Improving Variational Auto-Encoders using Householder Flow. ArXiv e-prints, November 2016.
29. Tomczak, J. M. and Welling, M. Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flow. ArXiv e-prints, June 2017.
30. Turner, R and Sahani, M. Two problems with variational expectation maximisation for time-series models. inference and learning in dynamic models. Cambridge University Press, pp. 104-123, 2011.
31. Wu, Y., Burda, Y., Salakhutdinov, R., and Grosse, R. On the Quantitative Analysis of Decoder-Based Generative Models. ICLR, 2017.
32. Xiao, Han, Rasul, Kashif, and Vollgraf, Roland. Fashionmnist: A novel image dataset for benchmarking machine learning algorithms. github.com/zalandoresearch/fashionmnist, 2017.