Large scale hierarchical neural network language models

13th Annual Conference of the International Speech Communication Association 2012, INTERSPEECH 2012

Volumn 2, Issue , 2012, Pages 1670-1673

  Kuo, Hong Kwang Jeff ^a   Arsoy, Ebru ^a   Emami, Ahmad ^a   Vozila, Paul ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b NUANCE COMMUNICATIONS   (United States)

Author keywords

Neural network language models

Indexed keywords

BROADCAST NEWS; HIERARCHICAL NEURAL NETWORKS; N-GRAM LANGUAGE MODELS; PARALLELIZING; TRAINING DATA; TRAINING TIME; VOCABULARY SIZE; WORD ERROR RATE;

RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION;

COMPUTATIONAL LINGUISTICS;

EID: 84878422162     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling, " Computer Speech and Language, vol. 13, no. 4, 1999.
2. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, "A neural probabilistic language model, " Journal of Machine Learning Research, vol. 3, pp. 1137-1155, 2003.
3. H. Schwenk and J.-L. Gauvain, "Training neural network language models on very large corpora, " in Proc. HLT EMNLP 2005, Vancouver, British Columbia, Canada, 2005, pp. 201-208.
4. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model, " in Proc. Interpseeh 2010, Makuhari, Chiba, JP, 2010, pp. 1045-1048.
5. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model, " in Proc. ICASSP 2011, Praha, CZ, 2011, pp. 5528-5531.
6. H. Schwenk, "Continuous space language models, " Comput. Speech Lang., vol. 21, no. 3, pp. 492-518, Jul. 2007.
7. A. Emami, "A neural syntactic language model, " Ph.D. dissertation, Johns Hopkins University, Baltimore, MD, USA, 2006.
8. F. Morin and Y. Bengio, "Hierarchical probabilistic neural network language model, " in Proc. AISTATS, 2005, pp. 246-252.
9. A. Mnih and G. Hinton, "A scalable hierarchical distributed language model, " in Proc. NIPS, 2008.
10. H. Son Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language model, " in Proc. ICASSP 2011, Prague, Czech Republic, 2011, pp. 5524-5527.
11. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, "Strategies for training large scale neural network language models, " in Proc. ASRU 2011, Hawaii, HI, USA, 2011, pp. 196-201.
12. S. F. Chen, B. Kingsbury, L.Mangu, D. Povey, G. Saon, H. Soltau, and G. Zweig, "Advances in speech transcription at IBM under the DARPA EARS program, " IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 5, pp. 1596-1608, 2006.
13. S. F. Chen, "Shrinking exponential language models, " in HLTNAACL. The Association for Computational Linguistics, 2009.
14. S. F. Chen, L. Mangu, B. Ramabhadran, R. Sarikaya, and A. Sethy, "Scaling shrinkage-based language models, " in Proc. ASRU 2009, Merano, Italy, Dec. 2009, pp. 299-304.
15. A. Emami and L. Mangu, "Empirical study of neural network language models for Arabic speech recognition, " in Proc. ASRU 2007, Kyoto, Japan, Dec. 2007, pp. 147-152.
16. P. F. Brown, V. J. D. Pietra, P. V. deSouza, J. C. Lai, and R. L. Mercer, "Class-based n-gram models of natural language, " Computational Linguistics, vol. 18, no. 4, 1990.
17. I. Oparin, M. Sundermeyer, H. Ney, and J.-L. Gauvain, "Performance analysis of neural networks in combination with n-gram language models, " in Proc. ICASSP 2012, Kyoto, JP, 2012.