A fast and simple algorithm for training neural probabilistic language models

Proceedings of the 29th International Conference on Machine Learning, ICML 2012

Volumn 2, Issue , 2012, Pages 1751-1758

  Mnih, Andriy ^a   Teh, Yee Whye ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CONTINUOUS DISTRIBUTION; DATA SETS; LANGUAGE MODEL; LOG LIKELIHOOD; MICROSOFT RESEARCHES; N-GRAM MODELS; PROBABILISTIC LANGUAGE; SENTENCE COMPLETIONS; SIMPLE ALGORITHM; TREEBANKS;

COMPUTATIONAL LINGUISTICS; LEARNING SYSTEMS;

ALGORITHMS;

EID: 84867118996     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Bengio, Yoshua and Senécal, Jean-Sébastien. Quick training of probabilistic neural nets by importance sampling. In AISTATS'03, 2003.
2. Bengio, Yoshua and Senécal, Jean-Sébastien. Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Transactions on Neural Networks, 19(4):713-722, 2008.
3. Bengio, Yoshua, Ducharme, Réjean, and Vincent, Pascal. A neural probabilistic language model. In NIPS, pp. 932-938, 2000.
4. Bengio, Yoshua, Ducharme, Rejean, Vincent, Pascal, and Jauvin, Christian. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
5. Berger, A.L., Pietra, V.J.D., and Pietra, S.A.D. A maximum entropy approach to natural language processing. Computational linguistics, 22(1):39-71, 1996.
6. Bird, S., Klein, E., and Loper, E. Natural language processing with Python. O'Reilly Media, 2009.
7. Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, 2008.
8. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS2010), 2010.
9. Gutmann, M.U. and Hyvärinen, A. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13:307-361, 2012.
10. Maas, A.L. and Ng, A.Y. A probabilistic model for semantic word vectors. In NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning, 2010.
11. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010.
12. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Černockỳ, J. Empirical evaluation and combination of advanced language modeling techniques. In Proceedings of Interspeech, pp. 605-608, 2011.
13. Mnih, A. and Hinton, G. Three new graphical models for statistical language modelling. Proceedings of the 24th International Conference on Machine Learning, pp. 641-648, 2007.
14. Mnih, A., Yuecheng, Z., and Hinton, G. Improving a statistical language model through non-linear prediction. Neurocomputing, 72(7-9):1414-1418, 2009.
15. Mnih, Andriy and Hinton, Geoffrey. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, volume 21, 2009.
16. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In AISTATS'05, pp. 246-252, 2005.
17. Pihlaja, M., Gutmann, M., and Hyvärinen, A. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI2010)., 2010.
18. Schwenk, Holger and Gauvain, Jean-Luc. Training neural network language models on very large corpora. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 201-208, 2005.
19. Socher, R., Lin, C.C., Ng, A.Y., and Manning, C.D. Parsing natural scenes and natural language with recursive neural networks. In International Conference on Machine Learning (ICML), 2011.
20. Stolcke, A. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pp. 901-904, 2002.
21. Turian, J., Ratinov, L., and Bengio, Y. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384-394, 2010.
22. Zweig, G. and Burges, C.J.C The Microsoft Research Sentence Completion Challenge. Technical Report MSR-TR-2011-129, Microsoft Research, 2011.