Strategies for training large scale neural network language models

2011 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2011, Proceedings

Volumn , Issue , 2011, Pages 196-201

  Mikolov, Tomáš ^a   Deoras, Anoop ^b   Povey, Daniel ^c   Burget, Lukáš ^a   Černocký, Jan ^a  

^a BRNO UNIVERSITY OF TECHNOLOGY   (Czech Republic)

^b JOHNS HOPKINS UNIVERSITY   (United States)

^c MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BROADCAST NEWS; FAST CONVERGENCE; LANGUAGE MODEL; LARGE DATASETS; MAXIMUM ENTROPY MODELS; NETWORK LANGUAGE; NEURAL NETWORK MODEL; RELATIVE REDUCTION; TRAINING DATA; WORD ERROR RATE;

COMPUTATIONAL LINGUISTICS; NEURAL NETWORKS;

SPEECH RECOGNITION;

EID: 84858966958     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ASRU.2011.6163930     Document Type: Conference Paper

References (21)

1. R. Rosenfeld, "A maximum entropy approach to adaptive statistical language modeling," Computer, Speech and Language, vol. 10, pp. 187-228, 1996.
2. Y. Bengio, R. Ducharme, P. Vincent et al., "A neural probabilistic language model," Journal of Machine Learning Research, vol. 3, pp. 1137-1155, 2003.
3. H. Soltau, G. Saon, and B. Kingsbury, "The IBM Attila speech recognition toolkit," in Proc. IEEE Workshop on Spoken Language Technology, 2010.
4. G. Zweig, P. Nguyen et al., "Speech recognition with segmental conditional random fields: A summary of the JHU CLSP summer workshop," in Proceedings of ICASSP, 2011.
5. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model," in Proceedings of ICASSP, 2011.
6. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Cernocky, "Empirical evaluation and combination of advanced language modeling techniques," in Proceedings of Interspeech, 2011.
7. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model," in Proceedings of Interspeech, 2010.
8. T. Alumae and M. Kurimo, "Efficient estimation of maximum entropy language models with N-gram features: an SRILM extension," in Proceedings of Interspeech, 2010.
9. W. Xu and A. Rudnicky, "Can artificial neural networks learn language models?" in International Conference on Statistical Language Processing, 2000.
10. H. Schwenk and J.-L. Gauvain, "Training neural network language models on very large corpora," in Proceedings of EMNLP, 2005.
11. P. Xu, A. Gunawardana, and S. Khudanpur, "Efficient subsampling for training complex language models," in Proceedings of EMNLP, 2011.
12. H.-S. Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language model," in Proceedings of ICASSP, 2011.
13. J. Goodman, "Classes for fast maximum entropy training," in Proceedings of ICASSP, 2001.
14. F. Morin and Y Bengio, "Hierarchical probabilistic neural network language model," in AISTATS, 2005, pp. 246-252.
15. Y Bengio, J. Louradour, R. Collobert, and J. Weston, "Curriculum learning," in Proceedings of ICML, 2009.
16. H. Schwenk, "Continuous space language models," Comput. Speech Lang., vol. 21, pp. 492-518, July 2007. (Pubitemid 46367510)
17. S. F Chen, L. Mangu, B. Ramabhadran, R. Sarikaya, and A. Sethy, "Scaling shrinkage-based language models," in Proceedings of ASRU, 2009.
18. J. L. Elman, "Learning and development in neural networks: The importance of starting small," Cognition, vol. 48, pp. 71-99, 1993.
19. A. Deoras, T. Mikolov, and K. Church, "A fast re-scoring strategy to capture long-distance dependencies," in Proceedings of EMNLP, 2011.
20. A. Deoras, D. Filimonov, M. Harper, and F. Jelinek, "Model combination for speech recognition using Empirical Bayes risk minimization," in Proc. of IEEE Workshop on Spoken Language Technology (SLT), 2010.
21. S. F. Chen, "Shrinking exponential language models," in Proc. NAACL HLT, 2009.