TopicRNN: A recurrent neural network with long-range semantic dependency

5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings

Volumn , Issue , 2017, Pages

  Dieng, Adji B ^a   Gao, Jianfeng ^b   Wang, Chong ^b   Paisley, John ^a  

^a Columbia University ^*  (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; MODELING LANGUAGES; SEMANTIC WEB; SEMANTICS; SENTIMENT ANALYSIS; STATISTICS; SYNTACTICS;

FEATURE EXTRACTOR; LATENT DIRICHLET ALLOCATION; LATENT TOPIC MODEL; LONG-RANGE DEPENDENCIES; RECURRENT NEURAL NETWORK (RNN); SEMANTIC DEPENDENCY; SEMANTIC STRUCTURES; STATE OF THE ART;

RECURRENT NEURAL NETWORKS;

EID: 85088228479     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
2. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. journal of machine learning research, 3(Feb):1137-1155, 2003.
3. D. Blei and J. Lafferty. Correlated topic models. Advances in neural information processing systems, 18:147, 2006.
4. D. M. Blei and J. D. Lafferty. Topic models. Text mining: classification, clustering, and applications, 10(71):34, 2009.
5. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022, 2003.
6. C. Chelba and F. Jelinek. Structured language modeling. Computer Speech & Language, 14(4): 283-332, 2000.
7. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
8. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
9. A. M. Dai and Q. V. Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, pages 3079-3087, 2015.
10. J. Gao, J.-Y. Nie, G. Wu, and G. Cao. Dependence language model for information retrieval. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 170-177. ACM, 2004.
11. S. Ghosh, O. Vinyals, B. Strope, S. Roy, T. Dean, and L. Heck. Contextual lstm (clstm) models for large scale nlp tasks. arXiv preprint arXiv:1602.06291, 2016.
12. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
13. Y. Ji, T. Cohn, L. Kong, C. Dyer, and J. Eisenstein. Document context language models. arXiv preprint arXiv:1511.03962, 2015.
14. Y. Ji, G. Haffari, and J. Eisenstein. A latent variable recurrent neural network for discourse relation language models. arXiv preprint arXiv:1603.01913, 2016.
15. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233, 1999.
16. R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.
17. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
18. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
19. Q. V. Le and T. Mikolov. Distributed representations of sentences and documents. In ICML, volume 14, pages 1188-1196, 2014.
20. R. Lin, S. Liu, M. Yang, M. Li, M. Zhou, and S. Li. Hierarchical recurrent neural network for document modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 899-907, 2015.
21. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 142-150. Association for Computational Linguistics, 2011.
22. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330, 1993.
23. Y. Miao, L. Yu, and P. Blunsom. Neural variational inference for text processing. arXiv preprint arXiv:1511.06038, 2015.
24. T. Mikolov and G. Zweig. Context dependent recurrent neural network language model. In SLT, pages 234-239, 2012.
25. T. Mikolov, M. Karafiát, L. Burget, J. Cernocky, ` and S. Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, page 3, 2010.
26. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, ` and S. Khudanpur. Extensions of recurrent neural network language model. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5528-5531. IEEE, 2011.
27. T. Mikolov, A. Joulin, S. Chopra, M. Mathieu, and M. Ranzato. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753, 2014.
28. D. Mimno, H. M. Wallach, E. Talley, M. Leenders, and A. McCallum. Optimizing semantic coherence in topic models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 262-272. Association for Computational Linguistics, 2011.
29. T. Miyato, A. M. Dai, and I. Goodfellow. Adversarial training methods for semi-supervised text classification. stat, 1050:7, 2016.
30. R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. ICML (3), 28:1310-1318, 2013.
31. D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
32. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1): 1929-1958, 2014.
33. I. Sutskever. Training recurrent neural networks. PhD thesis, University of Toronto, 2013.
34. H. M. Wallach. Topic modeling: beyond bag-of-words. In Proceedings of the 23rd international conference on Machine learning, pages 977-984. ACM, 2006.
35. H. M. Wallach, D. M. Mimno, and A. McCallum. Rethinking lda: Why priors matter. In Advances in neural information processing systems, pages 1973-1981, 2009.