Semi-amortized variational autoencoders

35th International Conference on Machine Learning, ICML 2018

Volumn 6, Issue , 2018, Pages 4196-4208

  Kim, Yoon ^a   Wiseman, Sam ^a   Miller, Andrew C ^a   Sontag, David ^b   Rush, Alexander M ^a  

^a HARVARD UNIVERSITY   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; STOCHASTIC SYSTEMS;

AUTO-REGRESSIVE; GENERATIVE MODEL; GRADIENT-BASED OPTIMIZATION; HYBRID APPROACH; INFERENCE NETWORK; TEXT GENERATIONS; VARIATIONAL INFERENCE; VARIATIONAL PARAMETERS;

LEARNING SYSTEMS;

EID: 85057251532     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (67)

1. Amos, Brandon and Kolter, J. Zico. OptNet: Differentiable Optimization as a Layer in Neural Networks. In Proceedings of ICML, 2017.
2. Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Hoffman, Matthew, Pfau, David, Schaul, Tom, and de Freitas, Nando. Learning to Learn by Gradient Descent by Gradient Descent. In Proceedings of NIPS, 2016.
3. Belanger, David, Yang, Bishan, and McCallum, Andrew. End-toend Learning for Structured Prediction Energy Networks. In Proceedings of ICML, 2017.
4. Blei, David M., Kucukelbir, Alp, and McAuliffe, Jon D. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-877, 2017.
5. Bowman, Samuel R., Vilnis, Luke, Vinyal, Oriol, Dai, Andrew M., Jozefowicz, Rafal, and Bengio, Samy. Categorical Reparameterization with Gumbel-Softmax. In Proceedings of CoNLL, 2016.
6. Brakel, Philemon, Stroobandt, Dirk, and Schrauwen, Benjamin. Training Energy-Based Models for Time-Series Imputation. Journal of Machine Learning Research, 14:2771-2797, 2013.
7. Burda, Yuri, Grosse, Roger, and Salakhutdinov, Ruslan. Importance Weighted Autoencoders. In Proceedings of ICLR, 2015a.
8. Burda, Yuri, Grosse, Roger, and Salakhutdinov, Ruslan. Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing. In Proceedings of AISTATS, 2015b.
9. Chen, Xi, Kingma, Diederik P., Salimans, Tim, Duan, Yan, Dhariwal, Prafulla, Schulman, John, Sutskever, Ilya, and Abbeel, Pieter. Variational Lossy Autoencoder. In Proceedings of ICLR, 2017.
10. Cho, Kyunghyun, Raiko, Tapani, Hin, Alexander, and Karhunen, Juha. A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines. In Proceedings of ICANN, 2013.
11. Chung, Junyoung, Kästner, Kyle, Dinh, Laurent, Goel, Kratarth, Courville, Aaron, and Bengio, Yoshua. A Recurrent Latent Variable Model for Sequential Data. In Proceedings of NIPS, 2015.
12. Cremer, Chris, Li, Xuechen, and Duvenaud, David. Inference Suboptimality in Variational Autoencoders. In Proceedings of ICML, 2018.
13. Dieng, Adji B., Wang, Chong, Gao, Jianfeng,, and Paisley, John. TopicRNN: A Recurrent Neural Network With Long-Range Semantic Dependency. In Proceedings of ICLR, 2017.
14. Domke, Justin. Generic Methods for Optimization-based Modeling. In Proceedings of AISTATS, 2012.
15. Fraccaro, Marco, Sonderby, Soren Kaae, Paquet, Ulrich, and Winther, Ole. Sequential Neural Models with Stochastic Layers. In Proceedings of NIPS, 2016.
16. Ghahramani, Zoubin and Beal, Matthew. Propagation algorithms for variational bayesian learning. In Proceedings of NIPS, 2001.
17. Goyal, Anirudh, Sordoni, Alessandro, Cote, Marc-Alexandre, Ke, Nan Rosemary, and Bengio, Yoshua. Z-Forcing: Training Stochastic Recurrent Networks. In Proceedings of NIPS, 2017a.
18. Goyal, Prasoon, Hu, Zhiting, Liang, Xiaodan, Wang, Chenyu, and Xing, Eric. Nonparametric Variational Auto-encoders for Hierarchical Representation Learning. In Proceedings of ICCV, 2017b.
19. Gregor, Karol, Danihelka, Ivo, Graves, Alex, Rezende, Danilo Jimenez, and Wierstra, Daan. DRAW: A Recurrent Neural Network for Image Generation. In Proceedings of ICML, 2015.
20. Gregor, Karol, Besse, Frederic, Rezende, Danilo Jimenez, Danihelka, Ivo, and Wierstra, Daan. Towards Conceptual Compression. In Proceedings of NIPS, 2016.
21. Gulrajani, Ishaan, Kumar, Kundan, Ahmed, Faruk, Taiga, Adrien Ali, Visin, Francesco, Vazquez, David, and Courville, Aaron. PixelVAE: A Latent Variable Model for Natural Images. In Proceedings of ICLR, 2017.
22. Guu, Kelvin, Hashimoto, Tatsunori B., Oren, Yonatan, and Liang, Percy. Generating Sentences by Editing Prototypes. arXiv:1709.08878, 2017.
23. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In Proceedings of CVPR, 2016.
24. Hjelm, R Devon, Cho, Kyunghyun, Chung, Junyoung, Salakhutdinov, Russ, Calhoun, Vince, and Jojic, Nebojsa. Iterative Refinement of the Approximate Posterior for Directed Belief Networks. In Proceedings of NIPS, 2016.
25. Hoffman, Matthew D., Blei, David M., Wang, Chong, and Paisley, John. Stochastic Variational Inference. Journal of Machine Learning Research, 13:1303-1347, 2013.
26. Hornik, Kur, Stinchcombe, Maxwell, and White, Halber. Multilayer Feedforward Networks are Universal Approximators. Neural Networks, 2:359-366, 1989.
27. Hu, Zhiting, Yang, Zichao, Liang, Xiaodan, Salakhutdinov, Ruslan, and Xing, Eric P. Toward Controlled Generation of Text. In Proceedings of ICML, 2017.
28. Jaderberg, Max, Czarnecki, Wojciech Marian, Osindero, Simon, Vinyals, Oriol, Graves, Alex, Silver, David, and Kavukcuoglu, Koray. Decoupled Neural Interfaces using Synthetic Gradients. In Proceedings of ICML, 2017.
29. Johnson, Matthew, Duvenaud, David K., Wiltschko, Alex, Adams, Ryan P., and Datta, Sandeep R. Composing Graphical Models with Neural Networks for Structured Representations and Fast Inference. In Proceedings of NIPS, 2016.
30. Jordan, Michael, Ghahramani, Zoubin, Jaakkola, Tommi, and Saul, Lawrence. Introduction to Variational Methods for Graphical Models. Machine Learning, 37:183-233, 1999.
31. Kim, Yoon, Denton, Carl, Hoang, Luong, and Rush, Alexander M. Structured Attention Networks. In Proceedings of ICLR, 2017.
32. Kingma, Diederik P. and Welling, Max. Auto-Encoding Variational Bayes. In Proceedings of ICLR, 2014.
33. Kingma, Diederik P., Salimans, Tim, and Welling, Max. Improving Variational Inference with Autoregressive Row. In Proceedings of ICLR Workshop, 2016.
34. Krishnan, Rahul G., Shalit, Uri, and Sontag, David. Structured Inference Networks for Nonlinear State Space Models. In Proceedings of AAAI, 2017.
35. Krishnan, Rahul G., Liang, Dawen, and Hoffman, Matthew. On the Challenges of Learning with Inference Networks on Sparse, High-dimensional Data. In Proceedings of AISTATS, 2018.
36. Lake, Brendan M., Salakhutdinov, Ruslan, and Tenenbaum, Joshua B. Human-level Concept Learning through Probabilistic Program Induction. Science, 350:1332-1338, 2015.
37. LeCun, Yann, Simard, Patrice, and Pearlmutter, Barak. Automatic Learning Rate Maximization by On-line Estimation of the Hessians Eigenvectors. In Proceedings of NIPS, 1993.
38. Maclaurin, Dougal, Duvenaud, David, and Adams, Ryan P. Gradient-based Hyperparameter Optimization through Reversible Learning. In Proceedings of ICML, 2015.
39. Marino, Joseph, Yue, Yisong, and Mandt, Stephan. Iterative Amortized Inference. In Proceedings of ICML, 2018.
40. Metz, Luke, Poole, Ben, Pfau, David, and Sohl-Dickstein, Jascha. Unrolled Generative Adversarial Networks. In Proceedings of ICLR, 2017.
41. Miao, Yishu, Yu, Lei, and Blunsom, Phil. Neural Variational Inference for Text Processing. In Proceedings of ICML, 2016.
42. Miao, Yishu, Grefenstette, Edward, and Blunsom, Phil. Discovering Discrete Latent Topics with Neural Variational Inference. In Proceedings of ICML, 2017.
43. Mnih, Andryi and Gregor, Karol. Neural Variational Inference and Learning in Belief Networks. In Proceedings of ICML, 2014.
44. Mueller, Jonas, Gifford, David, and Jaakkola, Tommi. Sequence to Better Sequence: Continuous Revision of Combinatorial Structures. In Proceedings of ICML, 2017.
45. Pearlmutter, Barak A. Fast exact multiplication by the hessian. Neural Computation, 6(1):147-160, 1994.
46. Pu, Yunchen, Gan, Zhe, Henao, Ricardo, Li, Chunyuan, Han, Shaobo, and Carin, Lawrence. VAE Learning via Stein Variational Gradient Descent. In Proceedings of NIPS, 2017.
47. Ranganath, Rajesh, Gerrish, Sean, and Blei, David M. Black Box Variational Inference. In Proceedings of AISTATS, 2014.
48. Ranganath, Rajesh, Tran, Dustin, and Blei, David M. Hierarchical Variational Models. In Proceedings of ICML, 2016.
49. Rezende, Danilo J. and Mohamed, Shakir. Variational Inference with Normalizing Flows. In Proceedings of ICML, 2015.
50. Rezende, Danilo Jimenez, Mohamed, Shakir, and Wierstra, Daan. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of ICML, 2014.
51. Rolfe, Jason Tyler. Discrete Variational Autoencoders. In Proceedings of ICLR, 2017.
52. Sakaya, Joseph and Klami, Arto. Importance Sampled Stochastic Optimization for Variational Inference. In Proceedings of UAI, 2017.
53. Salakhutdinov, Ruslan and Larochelle, Hugo. Efficient Learning of Deep Boltzmann Machines. In Proceedings of AISTATS, 2010.
54. Salimans, Tim, Kingma, Diederik, and Welling, Max. Markov Chain Monte Carlo and Variational Inference: Bridging the Gap. In Proceedings of ICML, 2015.
55. Schmidt, Mark, Roux, Nicolas Le, and Bach, Francis. Minimizing Finite Sums with the Stochastic Average Gradient. arXiv:1309.2388, 2013.
56. Semeniuta, Stanislau, Severyn, Aliaksei, and Barth, Erhardt. A Hybrid Convolutional Variational Autoencoder for Text Generation. In Proceedings of EMNLP, 2017.
57. Serban, Julian Vlad, Sordoni, Alessandro, Ryan Lowe, Laurent Charlin, Pineau, Joëlle, Courville, Aaron, and Bengio, Yoshua. A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. In Proceedings of AAAI, 2017.
58. Sønderby, Casper Kaae, Raiko, Tapani, Maaløe, Lars, Sønderby, Søren Kaae, and Winther, Ole. Ladder Variational Autoencoders. In Proceedings of NIPS, 2016.
59. Stoyanov, Veselin, Ropson, Alexander, and Eisner, Jason. Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure. In Proceedings of AISTATS, 2011.
60. Tomczak, Jakub M. and Welling, Max. VAE with a VampPrior. In Proceedings of AISTATS, 2018.
61. Tran, Dustin, Ranganath, Rajesh, and Blei, David M. The Variational Gaussian Process. In Proceedings of ICLR, 2016.
62. van den Oord, Aaron, Kalchbrenner, Nal, Vinyals, Oriol, Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray. Conditional Image Generation with PixelCNN Decoders. In Proceedings of NIPS, 2016.
63. Wainwright, Martin J. and Jordan, Michael I. Introduction to Variational Methods for Graphical Models. Foundations and Trends in Machine Learning, 1:1-103, 2008.
64. Wang, Wenlin, Gan, Zhe, Wang, Wenqi, Shen, Dinghan, Huang, Jiaji, Ping, Wei, Satheesh, Sanjeev, and Carin, Lawrence. Topic Compositional Neural Language Model. In Proceedings of AISTATS, 2018.
65. Yang, Zichao, Hu, Zhiting, Salakhutdinov, Ruslan, and Berg-Kirkpatrick, Taylor. Improved Variational Autoencoders for Text Modeling using Dilated Convolutions. In Proceedings of ICML, 2017.
66. Zhang, Cheng, Butepage, Judith, Kjellstrom, Hedvig, and Mandt, Stephan. Advances in Variational Inference. arXiv:1711.05597, 2017.
67. Zhao, Shengjia, Song, Jiaming, and Ermon, Stefano. Towards Deeper Understanding of Variational Autoencoding Models. In Proceedings of ICML, 2017.