Adaptive importance sampling to accelerate training of a neural probabilistic language model

IEEE Transactions on Neural Networks

Volumn 19, Issue 4, 2008, Pages 713-722

  Bengio, Yoshua ^a   Senécal, Jean Sébastien ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Energy based models; Fast training; Importance sampling; Language modeling; Monte Carlo methods; Probabilistic neural networks

Indexed keywords

APPROXIMATION THEORY; COMPUTER SIMULATION; MAXIMUM LIKELIHOOD; PROBABILITY; STATISTICAL METHODS;

NEURAL PROBABILISTIC LANGUAGE MODEL; STATISTICAL LANGUAGE MODELING;

NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER LANGUAGE; COMPUTER SIMULATION; HUMAN; LANGUAGE; PROBABILITY; STATISTICAL MODEL;

COMPUTER SIMULATION; HUMANS; LANGUAGE; MARKOV CHAINS; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PROGRAMMING LANGUAGES;

EID: 42549142788     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2007.912312     Document Type: Article

References (34)

1. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, "A neural probabilistic language model," J. Mach. Learn. Res., vol. 3, pp. 1137-1155, 2003.
2. H. Schwenk and J.-L. Gauvain, "Connectionist language modeling for large vocabulary continuous speech recognition," in Proc. Int. Conf. Acoust. Speech Signal Process., Orlando, FL, 2002, pp. 765-768.
3. H. Schwenk, "Efficient training of large neural networks for language modeling," in Proc. IEEE Int. Joint Conf. Neural Netw., Jul. 2004, vol. 4, pp. 3059-3064.
4. P. Xu, A. Emami, and F. Jelinek, "Training connectionist models for the structured language model," in Proc. Conf. Empiric. Methods Natural Lang. Process., 2001, vol. 10, pp. 160-167.
5. C. D. Manning and H. Schütze, Foundations of Statisticat Natural Language Processing. Cambridge, MA: MIT Press, 1999.
6. F. Jelinek and R. L. Mercer" E. S. Gelsema and L. N. Kanal, Eds., "Interpolated estimation of Markov source parameters from sparse data," in Pattern Recognition in Practice. Amsterdam, The Netherlands: North-Holland, 1980.
7. S. M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer," IEEE Trans. Acoust. Speech Signal Process., vol. ASSP-35, no. 3, pp. 400-401, Mar. 1987.
8. R. Kneser and H. Ney, "Improved backing-off for m-gram language modeling," in Proc. Int. Conf. Acoust. Speech Signal Process., 1995, pp. 181-184.
9. Y. Bengio, R. Ducharme, and P. Vincent, "A neural probabilistic language model," in Advances in Neural Information Processing Systems 13, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds. Cambridge, MA: MIT Press, 2001, pp. 932-938.
10. G. Hinton, "Learning distributed representations of concerts," in Proc. 8th Annu. Conf. Cogn. Sci. Soc., Amherst, Hillsdale, 1986, pp. 1-12.
11. W. Xu and A. Rudnicky, "Can artificial neural network learn language models," in Proc. Int. Conf. Statist. Lang. Process., Beijing, China, 2000, pp. M1-13.
12. J. Bellegarda, "A latent semantic analysis framework for large-span language modeling," in Proc. Eurospeech, Rhodes, Greece, 1997, pp. 1451-1454.
13. S. Riis and A. Krogh, "Improving protein secondary structure prediction using structured neural networks and multiple sequence profiles," J. Comput. Biol., pp. 163-183, 1996.
14. K. Jensen and S. Riis, "Self organizing letter code-book for text-to-phoneme neural network model," in Proc. Int. Conf. Spoken Lang. Process., 2000, vol. 3, pp. 318-321.
15. K. J. Lang and G. E. Hinton, "The development of the time-delay neural network architecture for speech recognition," Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-88-152, 1988.
16. Y.-W. Teh, M. Welling, S. Osindero, and G. E. Hinton, "Energy-based models for sparse overcomplete representations," J. Mach. Learn. Res., vol. 4, pp. 1235-1260, 2003.
17. L. D. Brown, "Fundamentals of stafisfical exponential families," in Lecture Notes Monograph Series. Bethesda, MD: Institute of Mathematical Statistics, 1986, vol. 9.
18. G. E. Hinton and T. J. Sejnowski, "Learning and releasing in Boltzmann machines," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986.
19. R. Rosenfeld, S. F. Chen, and X. Zhu, "Whole-sentence exponential language models: A vehicle for linguistic-statistical integration," Comput. Speech Lang. vol. 15, no. 1, 2001 [Online]. Available: citeseer.nj.nec.com/448532.html
20. A. Berger, S. Della Pietra, and V. Della Pietra, "A maximum entropy approach to natural language processing," Comput. Linguist., vol. 22, pp. 39-71, 1996.
21. C. P. Robert and G. Casella, Monte Carlo Statistical Methods, New York: Springer-Verlag, 2000.
22. A. Kong, J. S. Liu, and W. H. Wong, "Sequential imputations and Bayesian missing data problems," J. Amer. Statist. Assoc., vol. 89, pp. 278-288, 1994.
23. J. S. Liu, Monte Carlo Strategies in Scientific Computing. New York: Springer-Verlag, 2001.
24. O. Luis and K. Leslie, "Adaptive importance sampling for estimation in structured domains," in Proc. 16th Annu. Conf. Uncertainty Artif. Intell., 2000, pp. 446-454.
25. Y. Bengio and J.-S. Senécal, "Quick training of probabilistic neural nets by sampling," in Proc. 9th Int. Workshop Artif. Intell. Statist., Key West, FL., 2003, vol. 9.
26. A. Kong, "A note on importance sampling using standardized weights," Dept. Statist., Univ. Chicago, Chicago, IL, Tech. Rep. 348, 1992.
27. J. Goodman, "A bit of progress in language modeling-extended version," Mach. Learn. Appl. Statist. Group, Microsoft Res., Redmond, WA, Tech. Rep. MSR-TR-2001-72, 2003.
28. J. Cheng and M. J. Druzdzel, "Ais-bn: An adaptive importance sampling algorithm for evidential reasoning in large Bayesian networks," J. Artif. Intell. Res., vol. 13, pp. 155-188, 2000.
29. R. Rosenfeld, "Two decades of statistical language modeling: Where do we go from here?," Proc. IEEE, vol. 88, no. 8, pp. 1270-1278, Aug. 2000.
30. J. Goodman, "A bit of progress in language modeling," Microsoft Res., Tech. Rep. MSR -TR-2001-72, 2001.
31. W. Wang and M. P. Harper, "The super ARV language model: Investigating th effectiveness of tightly integrating multiple knowledge sources," in Proc. ACL-02 Conf. Empirical Methods Natural Lang. Proress. Morristown, NJ, 2002, pp. 238-247.
32. H. Schwenk and J.-L. Gauvain, "Connectionist language modeling for large vocabulary continuous speech recognition," in Proc. Int. Conf. Acoust. Speach Signal Process., Orlando, FL, 2002, pp. 765-768.
33. G. Hinton, "Products of experts," in Proc. 9th Int. Conf. Artif. Neural Netw., Edinburgh, Scotland, 1999, pp. 1-6.
34. Y. Bengio, "New distributed probabilistic language models," Dept. IRO, Université de Montr_aveal, Montréal, QC, Canada, Tech Rep. 1215, 2002.