Piecewise latent variables for neural variational text processing

EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings

Volumn , Issue , 2017, Pages 422-432

  Serban, Iulian V ^a   Ororbia, Alexander G ^b   Pineau, Joelle ^c   Courville, Aaron ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

^c MCGILL UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; MODELING LANGUAGES; TEXT PROCESSING;

EXPONENTIAL NUMBERS; GRAPHICAL MODEL; LATENT VARIABLE; NATURAL LANGUAGE GENERATION; NATURAL LANGUAGE PROCESSING; NATURAL LANGUAGE TEXT; PIECE-WISE CONSTANTS; VARIATIONAL INFERENCE;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 85060475117     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/d17-1043     Document Type: Conference Paper

References (57)

1. J. L. Ba, J. R. Kiros, and G. E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
2. S. Bangalore, G. Di Fabbrizio, and A. Stent. 2008. Learning the structure of task-driven human–human dialogs. IEEE Transactions on Audio, Speech, and Language Processing, 16(7):1249–1259.
3. D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. JAIR, 3:993–1022.
4. J. Bornschein and Y. Bengio. 2015. Reweighted wake-sleep. In ICLR.
5. S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. 2015. Generating sentences from a continuous space. In Conference on Computational Natural Language Learning.
6. Y. Burda, R. Grosse, and R. Salakhutdinov. 2016. Importance weighted autoencoders. ICLR.
7. A. Cardoso-Cachopo. 2007. Improving Methods for Single-label Text Categorization. PdD Thesis, Instituto Superior Tecnico, Universidade Tecnica de Lisboa.
8. X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. 2017. Variational lossy autoencoder. In ICLR.
9. N. Crook, R. Granell, and S. Pulman. 2009. Unsupervised classification of dialogue acts using a dirichlet process mixture model. In Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 341–348.
10. P. Dayan and G. E. Hinton. 1996. Varieties of helmholtz machine. Neural Networks, 9(8):1385–1403.
11. L. Devroye. 1986. Sample-based non-uniform random variate generation. In Proceedings of the 18th conference on Winter simulation, pages 260–265. ACM.
12. M. Farber. 2016. Amazon’s 'Alexa Prize’ Will Give College Students Up To $2.5M To Create A Social-bot. Fortune.
13. M. Fraccaro, S. K. Sønderby, U. Paquet, and O. Winther. 2016. Sequential neural models with stochastic layers. In NIPS, pages 2199–2207.
14. K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. 2015. DRAW: A recurrent neural network for image generation. In ICLR.
15. G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. 1995. The "wake-sleep” algorithm for unsupervised neural networks. Science, 268(5214):1158–1161.
16. G. E. Hinton and R. Salakhutdinov. 2009. Replicated softmax: an undirected topic model. In NIPS, pages 1607–1614.
17. G. E. Hinton and R. S. Zemel. 1994. Autoencoders, minimum description length and helmholtz free energy. In NIPS, pages 3–10. NIPS.
18. T. Hofmann. 1999. Probabilistic latent semantic indexing. In ACM SIGIR Conference on Research and Development in Information Retrieval, pages 50–57. ACM.
19. E. Jang, S. Gu, and B. Poole. 2017. Categorical reparameterization with gumbel-softmax. In ICLR.
20. M. Johnson, D. K. Duvenaud, A. Wiltschko, R. P. Adams, and S. R. Datta. 2016. Composing graphical models with neural networks for structured representations and fast inference. In NIPS, pages 2946–2954.
21. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37(2):183–233.
22. A. Kannan, K. Kurach, et al. 2016. Smart Reply: Automated Response Suggestion for Email. In KDD.
23. D. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
24. D. P. Kingma, T. Salimans, and M. Welling. 2016. Improving variational inference with inverse autoregressive flow. NIPS, pages 4736–4744.
25. D. P. Kingma and M. Welling. 2014. Auto-encoding variational Bayes. ICLR.
26. H. Larochelle and S. Lauly. 2012. A neural autoregressive topic model. In NIPS, pages 2708–2716.
27. A. B. Lindbo Larsen, S. K. Sønderby, and O. Winther. 2016. Autoencoding beyond pixels using a learned similarity metric. In ICML, pages 1558–1566.
28. S. Lauly, Y. Zheng, A. Allauzen, and H. Larochelle. 2016. Document neural autoregressive distribution estimation. arXiv preprint arXiv:1603.05962.
29. J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. 2016. A diversity-promoting objective function for neural conversation models. In The North American Chapter of the Association for Computational Linguistics (NAACL), pages 110–119.
30. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In Special Interest Group on Discourse and Dialogue (SIGDIAL).
31. Ryan T. Lowe, Nissan Pow, Iulian V. Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training End-to-End Dialogue Systems with the Ubuntu Dialogue Corpus. Dialogue & Discourse, 8(1).
32. Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. 2016. Auxiliary deep generative models. In ICML, pages 1445–1453.
33. C. J. Maddison, A. Mnih, and Y. W. Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR.
34. J. Markoff and P. Mozur. 2015. For Sympathetic Ear, More Chinese Turn to Smartphone Program. New York Times.
35. Y. Miao, L. Yu, and P. Blunsom. 2016. Neural variational inference for text processing. In ICML, pages 1727–1736.
36. A. Mnih and K. Gregor. 2014. Neural variational inference and learning in belief networks. In ICML, pages 1791–1799.
37. R. M. Neal. 1992. Connectionist learning of belief networks. Artificial intelligence, 56(1):71–113.
38. A. G. Ororbia II, C. L. Giles, and D. Reitter. 2015. Online semi-supervised learning with deep hybrid boltzmann machines and denoising autoencoders. arXiv preprint arXiv:1511.06964.
39. R. Pascanu, T. Mikolov, and Y. Bengio. 2012. On the difficulty of training recurrent neural networks. ICML, 28:1310–1318.
40. Rajesh Ranganath, Dustin Tran, and David Blei. 2016. Hierarchical variational models. In ICML, pages 324–333.
41. D. J. Rezende and S. Mohamed. 2015. Variational inference with normalizing flows. In ICML, pages 1530–1538.
42. D. J. Rezende, S. Mohamed, and D. Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In ICML, pages 1278–1286.
43. A. Ritter, C. Cherry, and W. B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 583–593.
44. J. T. Rolfe. 2017. Discrete variational autoencoders. In ICLR.
45. Francisco R Ruiz, Michalis Titsias RC AUEB, and David Blei. 2016. The generalized reparameterization gradient. In NIPS, pages 460–468.
46. R. Salakhutdinov and G. E. Hinton. 2009. Semantic hashing. International Journal of Approximate Reasoning, 50(7):969–978.
47. R. Salakhutdinov and H. Larochelle. 2010. Efficient learning of deep boltzmann machines. In AISTATs, pages 693–700.
48. T. Salimans, D. P Kingma, and M. Welling. 2015. Markov chain monte carlo and variational inference: Bridging the gap. In ICML, pages 1218–1226.
49. R. Sennrich, B. Haddow, and A. Birch. 2016. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL).
50. I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville. 2017a. Multiresolution recurrent neural networks: An application to dialogue response generation. In Thirty-First AAAI Conference (AAAI).
51. I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference (AAAI).
52. I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. 2017b. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference (AAAI).
53. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016b. Generative deep neural networks for dialogue: A short review. In NIPS, Let’s Discuss: Learning Methods for Dialogue Workshop.
54. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2015), pages 196–205.
55. N. Srivastava, R. R Salakhutdinov, and G. E. Hinton. 2013. Modeling documents with deep boltzmann machines. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI), pages 616–624.
56. B. Uria, I. Murray, and H. Larochelle. 2014. A deep and tractable density estimator. In ICML, pages 467–475.
57. K. Zhai and J. D. Williams. 2014. Discovering latent structure in task-oriented dialogues. In Association for Computational Linguistics (ACL), pages 36–46.