A Neural Probabilistic Language Model

Journal of Machine Learning Research

Volumn 3, Issue 6, 2003, Pages 1137-1155

  Bengio, Yoshua ^a   Ducharme, Réjean ^a   Vincent, Pascal ^a   Jauvin, Christian ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Artificial neural networks; Curse of dimensionality; Distributed representation; Statistical language modeling

Indexed keywords

MATHEMATICAL MODELS; NEURAL NETWORKS; PROBABILITY; PROBLEM SOLVING; STATISTICAL METHODS;

DISTRIBUTED REPRESENTATIONS;

LEARNING SYSTEMS;

EID: 0142166851     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/153244303322533223     Document Type: Conference Paper

References (33)

1. D. Baker and A. McCallum. Distributional clustering of words for text classification. In SIGIR'98, 1998.
2. J.R. Bellegarda. A latent semantic analysis framework for large-span language modeling. In Proceedings of Eurospeech 97, pages 1451-1454, Rhodes, Greece, 1997.
3. S. Bengio and Y. Bengio. Taking on the curse of dimensionality in joint distributions using neural networks. IEEE Transactions on Neural Networks, special issue on Data Mining and Knowledge Discovery, 11(3):550-557, 2000a.
4. Y. Bengio. New distributed probabilistic language models. Technical Report 1215, Dept. IRO, Université de Montréal, 2002.
5. Y. Bengio and S. Bengio. Modeling high-dimensional discrete data with multi-layer neural networks. In S. A. Solla, T. K. Leen, and K-R. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages 400-406. MIT Press, 2000b.
6. Y. Bengio and J-S. Senécal. Quick training of probabilistic neural nets by importance sampling. In AISTATS, 2003.
7. A. Berger, S. Della Pietra, and V. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22:39-71, 1996.
8. A. Brown and G.E. Hinton. Products of hidden markov models. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London, 2000.
9. P.F. Brown, V.J. Della Pietra, P.V. DeSouza, J.C. Lai, and R.L. Mercer. Class-based n-gram models of natural language. Computational Linguistics, 18:467-479, 1992.
10. S.F. Chen and J.T. Goodman. An empirical study of smoothing techniques for language modeling. Computer, Speech and Language, 13(4):359-393, 1999.
11. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391-407, 1990.
12. J. Dongarra, D. Walker, and The Message Passing Interface Forum. MPI: A message passing interface standard. Technical Report http://www-unix.mcs.anl.gov/mpi, University of Tenessee, 1995.
13. J.L. Elman. Finding structure intime. Cognitive Science, 14:179-211, 1990.
14. C. Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, 1998.
15. J. Goodman. A bit of progress in language modeling. Technical Report MSR-TR-2001-72, Microsoft Research, 2001.
16. G.E. Hinton. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pages 1-12, Amherst 1986, 1986. Lawrence Erlbaum, Hillsdale.
17. G.E. Hinton. Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London, 2000.
18. F. Jelinek and R. L. Mercer. Interpolated estimation of Markov source parameters from sparse data. In E. S. Gelsema and L. N. Kanal, editors, Pattern Recognition in Practice. North-Holland, Amsterdam, 1980.
19. K.J. Jensen and S. Riis. Self-organizing letter code-book for text-to-phoneme neural network model. In Proceedings ICSLP, 2000.
20. S.M. Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, ASSP-35 (3):400-401, March 1987.
21. R. Kneser and H. Ney. Improved backing-off for m-gram language modeling. In International Conference on Acoustics, Speech and Signal Processing, pages 181-184, 1995.
22. Y. LeCun, L. Bottou, G.B. Orr, and K.-R. Müller. Efficient backprop. In G.B. Orr and K.-R. Müller, editors, Neural Networks: Tricks of the Trade, pages 9-50. Springer, 1998.
23. R. Miikkulainen and M.G. Dyer. Natural language processing with modular neural networks and distributed lexicon. Cognitive Science, 15:343-399, 1991.
24. H. Ney and R. Kneser. Improved clustering techniques for class-based statistical language modelling. In European Conference on Speech Communication and Technology (Eurospeech), pages 973-976, Berlin, 1993.
25. T.R. Niesler, E.W.D. Whittaker, and P.C. Woodland. Comparison of part-of-speech and automatically derived category-based language models for speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pages 177-180, 1998.
26. A. Paccanaro and G.E. Hinton. Extracting distributed representations of concepts and relations from positive and negative propositions. In Proceedings of the International Joint Conference on Neural Network, IJCNN'2000, Como, Italy, 2000. IEEE, New York.
27. F. Pereira, N. Tishby, and L. Lee. Distributional clustering of english words. In 30th Annual Meeting of the Association for Computational Linguistics, pages 183-190, Columbus, Ohio, 1993.
28. S. Riis and A. Krogh. Improving protein secondary structure prediction using structured neural networks and multiple sequence profiles. Journal of Computational Biology, pages 163-183, 1996.
29. J. Schmidhuber. Sequential neural text compression. IEEE Transactions on Neural Networks, 7(1): 142-146, 1996.
30. H. Schutze. Word space. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages pp. 895-902, San Mateo CA, 1993. Morgan Kaufmann.
31. H. Schwenk and J-L. Gauvain. Connectionist language modeling for large vocabulary continuous speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pages 765-768, Orlando, Florida, 2002.
32. A. Stolcke. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Statistical Language Processing, Denver, Colorado, 2002.
33. W. Xu and A. Rudnicky. Can artificial neural network learn language models. In International Conference on Statistical Language Processing, pages M1-13, Beijing, China, 2000.