Z-forcing: Training stochastic recurrent networks

Advances in Neural Information Processing Systems

Volumn 2017-December, Issue , 2017, Pages 6714-6724

  Goyal, Anirudh ^a   Sordoni, Alessandro ^b   Côté, Marc Alexandre ^b   Ke, Nan Rosemary ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b MICROSOFT   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; EQUIVALENCE CLASSES; MODELING LANGUAGES; RECURRENT NEURAL NETWORKS; STOCHASTIC SYSTEMS; STORMS;

COMPETITIVE PERFORMANCE; LATENT VARIABLE MODELS; PROPOSED ARCHITECTURES; RECURRENT MODELS; RECURRENT NETWORKS; RECURRENT NEURAL NETWORK (RNN); STATE OF THE ART; VARIATIONAL INFERENCE;

STOCHASTIC MODELS;

EID: 85046993773     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. Bachman, P. (2016). An architecture for deep, hierarchical generative models. In Advances in Neural Information Processing Systems, pages 4826-4834.
2. Bachman, P. and Precup, D. (2015). Training deep generative models: Variations on a theme.
3. Bayer, J. and Osendorfer, C. (2014). Learning stochastic recurrent networks. arXiv preprint arXiv: 1411.7610.
4. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprint arXiv: 1206.6392.
5. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, S. (2015). Generating sentences from a continuous space. arXiv preprint arXiv: 1511.06349.
6. Burda, Y., Grosse, R., and Salakhutdinov, R. (2015). Importance weighted autoencoders. arXiv preprint arXiv: 1509.00519.
7. Chen, X., and Kingma, D. P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., and Abbeel, P. (2017). Variational lossy autoencoder. Proc. of ICLR.
8. Chung, J., Kastner, K., Dinh, L., Goel, K., and Courville, A. C., and Bengio, Y. (2015). A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pages 2980-2988.
9. Diao, Q., Qiu, M., Wu, C.-Y., and Smola, A. J., Jiang, J., and Wang, C. (2014). Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 193-202.
10. Fraccaro, M., Sønderby, S. K., Paquet, U., and Winther, O. (2016). Sequential neural models with stochastic layers. In Advances in Neural Information Processing Systems, pages 2199-2207.
11. Germain, M., Gregor, K., Murray, I., and Larochelle, H. (2015). Made: Masked autoencoder for distribution estimation. In ICML, pages 881-889.
12. Goyal, A., Lamb, A., Zhang, Y., Zhang, S., and Courville, A. C., and Bengio, Y. (2016). Professor forcing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4601-4609.
13. Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., and Wierstra, D. (2015). Draw: A recurrent neural network for image generation. arXiv preprint arXiv: 1502.04623.
14. Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., and Courville, A. (2016). Pixelvae: A latent variable model for natural images. arXiv preprint arXiv: 1611.05013.
15. Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., and Courville, A. (2017). Pixelvae: A latent variable model for natural images. Proc. of ICLR.
16. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8): 1735-1780.
17. Honkela, A. and Valpola, H. (2004). Variational learning and bits-back coding: an information-theoretic view to bayesian learning. IEEE Transactions on Neural Networks, 15(4): 800-810.
18. Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., and Xing, E. P. (2017). Controllable text generation. arXiv preprint arXiv: 1703.00955.
19. Karl, M., Soelch, M., Bayer, J., and van der Smagt, P. (2016). Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv: 1605.06432.
20. King, S. and Karaiskos, V. (2013). The blizzard challenge 2013. The Ninth Annual Blizzard Challenge, 2013.
21. Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980.
22. Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M. (2014). Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pages 3581-3589.
23. Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. (2016). Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pages 4743-4751.
24. Kingma, D. P. and Welling, M. (2014). Stochastic Gradient VB and the Variational Auto-Encoder. 2nd International Conference on Learning Representationsm (ICLR), pages 1-14.
25. Larochelle, H. and Murray, I. (2011). The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 29-37.
26. Louizos, C. and Welling, M. (2017). Multiplicative normalizing flows for variational bayesian neural networks. arXiv preprint arXiv: 1703.01961.
27. Oord, A. v. d., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel recurrent neural networks. arXiv preprint arXiv: 1601.06759.
28. Raiko, T., Li, Y., Cho, K., and Bengio, Y. (2014). Iterative neural autoregressive distribution estimator nade-k. In Advances in neural information processing systems, pages 325-333.
29. Rezende, D. J. and Mohamed, S. (2015). Variational inference with normalizing flows. arXiv preprint arXiv: 1505.05770.
30. Rumelhart, D. E., and Hinton, G. E., and Williams, R. J. (1988). Learning representations by back-propagating errors. Cognitive modeling, 5(3): 1.
31. Salakhutdinov, R. and Murray, I. (2008). On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference on Machine learning, pages 872-879. ACM.
32. Salimans, T., and Kingma, D. P., and Welling, M. (2014). Markov chain monte carlo and variational inference: Bridging the gap. arXiv preprint arXiv: 1410.6460.
33. Semeniuta, S., Severyn, A., and Barth, E. (2017). A hybrid convolutional variational autoencoder for text generation. arXiv preprint arXiv: 1702.02390.
34. Serban, I. V., II, A. G. O., Pineau, J., and Courville, A. C. (2017a). Piecewise latent variables for neural variational text processing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 422-432.
35. Serban, I. V., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A. C., and Bengio, Y. (2017b). A hierarchical latent variable encoder-decoder model for generating dialogues. In Proc. of AAAI.
36. Uria, B., Côté, M.-A., Gregor, K., Murray, I., and Larochelle, H. (2016). Neural autoregressive distribution estimation. Journal of Machine Learning Research, 17(205): 1-37.
37. Zhao, T., Zhao, R., and Eskenazi, M. (2017). Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv: 1703.10960.