Improved variational autoencoders for text modeling using dilated convolutions

34th International Conference on Machine Learning, ICML 2017

Volumn 8, Issue , 2017, Pages 5917-5928

  Yang, Zichao ^a   Hu, Zhiting ^a   Salakhutdinov, Ruslan ^a   Berg Kirkpatrick, Taylor ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; DECODING; ECONOMIC AND SOCIAL EFFECTS; LEARNING SYSTEMS; MODELING LANGUAGES; SIGNAL ENCODING;

AUTOENCODERS; ENCODING INFORMATIONS; LANGUAGE MODEL; SEMI-SUPERVISED; TEXT MODELING; TRADE OFF;

LONG SHORT-TERM MEMORY;

EID: 85048398320     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. Bayer, Justin and Osendorfer, Christian. Learning stochastic recurrent networks. arXiv preprint arXiv: 1411.7610, 2014.
2. Bowman, Samuel R, Vilnis, Luke, Vinyals, Oriol, Dai, Andrew M, Jozefowicz, Rafal, and Bengio, Samy. Generating sentences from a continuous space. arXiv preprint arXiv: 1511.06349, 2015.
3. Chen, Xi, Kingma, Diederik P, Salimans, Tim, Duan, Yan, Dhariwal, Prafulla, Schulman, John, Sutskever, Ilya, and Abbeel, Pieter. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731, 2016.
4. Chung, Junyoung, Kastner, Kyle, Dinh, Laurent, Goel, Kratarth, Courville, Aaron C, and Bengio, Yoshua. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
5. Dai, Andrew M and Le, Quoc V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, pp. 3079-3087, 2015.
6. Fraccaro, Marco, Sønderby, Søren Kaae, Paquet, Ulrich, and Winther, Ole. Sequential neural models with stochastic layers. In Advances in Neural Information Processing Systems, pp. 2199-2207, 2016.
7. Gregor, Karol, Danihelka, Ivo, Graves, Alex, Rezende, Danilo Jimenez, and Wierstra, Daan. Draw: A recurrent neural network for image generation. arXiv preprint arXiv: 1502.04623, 2015.
8. Gregor, Karol, Besse, Frederic, Rezende, Danilo Jimenez, Danihelka, Ivo, and Wierstra, Daan. Towards conceptual compression. In Advances In Neural Information Processing Systems, pp. 3549-3557, 2016.
9. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
10. Hu, Zhiting, Yang, Zichao, Liang, Xiaodan, Salakhutdinov, Ruslan, and Xing, Eric P. Controllable text generation. arXiv preprint arXiv:1703.00955, 2017a.
11. Hu, Zhiting, Yang, Zichao, Salakhutdinov, Ruslan, and Xing, Eric P. On unifying deep generative models. arXiv preprint arXiv: 1706.00550, 2017b.
12. Jang, Eric, Gu, Shixiang, and Poole, Ben. Categorical rcparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
13. Kalchbrenner, Nal, Espcholt, Lasse, Simonyan, Karen, Oord, Aaron van den, Graves, Alex, and Kavukcuoglu, Koray. Neural machine translation in linear time. arXiv preprint arXiv: 1610.10099, 2016a.
14. Kalchbrenner, Nal, Oord, Aaron van den, Simonyan, Karen, Danihelka, Ivo, Vinyals, Oriol, Graves, Alex, and Kavukcuoglu, Koray. Video pixel networks. arXiv preprint arXiv.1610.00527, 2016b.
15. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
16. Kingma, Diederik P and Welling, Max. Auto-encoding variational bayes. arXiv preprint arXiv.1312.6114, 2013.
17. Kingma, Diederik P, Mohamed, Shakir, Rezende, Danilo Jimenez, and Welling, Max. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581-3589, 2014.
18. Kingma, Diederik P, Salimans, Tim, and Welling, Max. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv: 1606.04934, 2016.
19. Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294-3302, 2015.
20. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
21. Larochelle, Hugo and Lauly, Stanislas. A neural autoregressive topic model. In Advances in Neural Information Processing Systems, pp. 2708-2716, 2012.
22. Le, Quoc V and Mikolov, Tomas. Distributed representations of sentences and documents. In ICML, Volume 14, pp. 1188-1196, 2014.
23. Maddison, Chris J, Mnih, Andriy, and Teh, Yee Whye. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 1611.00712, 2016.
24. Makhzani, Alireza, Shlens, Jonathon, Jaitly, Navdeep, Goodfellow, Ian, and Frey, Brendan. Adversarial autoencoders. arXiv preprint arXiv: 1511.05644, 2015.
25. Miao, Yishu, Yu, Lei, and Blunsom, Phil. Neural variational inference for text processing. In Proc. ICML, 2016.
26. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernocky, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In Interspeech, Volume 2, pp. 3, 2010.
27. Mnih, Andriy and Gregor, Karol. Neural variational inference and learning in belief networks. arXiv preprint arXiv: 1402.0030, 2014.
28. Rezende, Danilo Jimenez and Mohamed, Shakir. Variational inference with normalizing flows. arXiv preprint arXiv: 1505.05770, 2015.
29. Rezende, Danilo Jimenez, Mohamed, Shakir, and Wierstra, Daan. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
30. Salimans, Tim, Kingma, Diederik P, Welling, Max, et al. Markov chain monte carlo and variational inference: Bridging the gap. In ICML, Volume 37, pp. 1218-1226, 2015.
31. Serban, Iulian Vlad, Sordoni, Alessandro, Lowe, Ryan, Charlin, Laurent, Pineau, Joelle, Courville, Aaron, and Bengio, Yoshua. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv preprint arXiv: 1605.06069, 2016.
32. Tang, Duyu, Qin, Bing, and Liu, Ting. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP, pp. 1422-1432, 2015.
33. Van den Oord, Aaron, Dieleman, Sander, Zen, Heiga, Simonyan, Karen, Vinyals, Oriol, Graves, Alex, Kalchbrenner, Nal, Senior, Andrew, and Kavukcuoglu, Koray. Wavenet: A generative model for raw audio. CoRR abs/1609.03499, 2016a.
34. Van den Oord, Aaron, Kalchbrenner, Nal, Espeholt, Lasse, Vinyals, Oriol, Graves, Alex, et al. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems, pp. 4790-4798, 2016b.
35. Yan, Xinchen, Yang, Jimei, Sohn, Kihyuk, and Lee, Honglak. Attribute2image: Conditional image generation from visual attributes. In European Conference on Computer Vision, pp. 776-791. Springer, 2016.
36. Yang, Zichao, Yang, Diyi, Dyer, Chris, He, Xiaodong, Smola, Alex, and Hovy, Eduard. Hierarchical attention networks for document classification. In Proceedings of NAACL-HLT, pp. 1480-1489, 2016.
37. Yu, Fisher and Koltun, Vladlen. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 1511.07122, 2015.
38. Zhang, Biao, Xiong, Deyi, Su, Jinsong, Duan, Hong, and Zhang, Min. Variational neural machine translation. arXiv preprint arXiv: 1605.07869, 2016.
39. Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657, 2015.