CUED-RNNLM - An open-source toolkit for efficient training and evaluation of recurrent neural network language models

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2016-May, Issue , 2016, Pages 6000-6004

  Chen, X ^a   Liu, X ^a   Qian, Y ^a   Gales, M J F ^a   Woodland, P C ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

GPU; language model; open source toolkit; recurrent neural network; speech recognition

Indexed keywords

EID: 84973392043     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2016.7472829     Document Type: Conference Paper

References (34)

1. R. Kneser and H. Ney, "Improved backing-off for n-gram language modeling", Proc. ICASSP, 1995
2. S. F. Chen and J. T. Goodman. "An empirical study of smoothing techniques for language modeling", Computer Speech and Language, Vol. 13, Issue 4, pp. 359-394, 1999
3. T. Mikolov, S. Kombrink, L. Burget, J. H. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model", Proc. ICASSP, 2011
4. J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz, and J. Makhoul, "Fast and robust neural network joint models for statistical machine translation, " Association for Computational Linguistics, 2014
5. M. Sundermeyer, R. Schlüter, and H. Ney, "LSTM neural networks for language modeling", Proc. Interspeech, 2012
6. W. D. Mulder, S. Bethard, and M. F. Moens, "A survey on the application of recurrent neural networks to statistical language modeling", Computer Speech & Language, vol. 30, no. 1, pp. 61-98, 2015
7. S. Kombrink, T. Mikolov, M. Karafiat, and L. Burget, "Recurrent neural network based language modeling in meeting recognition., " Proc. Interspeech, 2011
8. T. Mikolov, S. Kombrink, A. Deoras, L. Burget, and J. Cernocky, "Recurrent neural network language modeling toolkit", Proc. ASRU Workshop, 2011
9. X. Chen, Y. Wang, X. Liu, M. J. F. Gales, and P. C. Woodland, "Efficient training of recurrent neural network language models using spliced sentence bunch", Proc. Interspeech, 2014
10. Geoffrey Zweig and Konstantin Makarychev, "Speed regularization and optimality in word classing", Proc. ICASSP, 2013
11. X. Chen, X. Liu, M. J. F. Gales, and P. C. Woodland, "Improving the training and evaluation efficiency of recurrent neural network language models", Proc. ICASSP, 2015
12. X. Chen, X. Liu, M. J. F. Gales, and P. C. Woodland, "Recurrent neural network language model training with noise contrastive estimation for speech recognition", Proc. ICASSP, 2015
13. T. Mikolov, M. Karafíat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model", Proc. ISCA Interspeech, 2010
14. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning representations by back-propagating errors, MIT Press, Cambridge, MA, USA, 1988
15. H. Schwenk, "Continuous space language models, " Computer Speech & Language, vol. 21, no. 3, pp. 492-518, 2007
16. A. Emami and L. Mangu, "Empirical study of neural network language models for Arabic speech recognition", Proc. ASRU Workshop, 2007
17. J. Park, X. Liu, M. J. F. Gales, and P. C. Woodland, "Improved neural network based language modelling and adaptation", Proc. Interspeech, 2010
18. H. Le, I. Oparin, A. Allauzen, J. Gauvain, and F. Yvon, "Structured output layer neural network language models for speech recognition", IEEE Trans Audio, Speech, and Language Processing, vol. 21, no. 1, pp. 197-206, 2013
19. X. Liu, Y. Wang, X. Chen, M. J. F. Gales, and P. C. Woodland, "Efficient lattice rescoring using recurrent neural network language models", Proc. ICASSP, 2014
20. D. Yu, A. Eversole, M. Seltzer, K. Yao, Z. Huang, B. Guenter, O. Kuchaiev, Y. Zhang, F. Seide, H. Wang, et al., "An introduction to computational networks and the computational network toolkit", Tech. Rep. MSR, http: //codebox/cntk, 2014
21. H. Schwenk, "CSLM-A modular open-source continuous space language modeling toolkit., " Proc. Interspeech, 2013
22. M. Sundermeyer, R. Schlüter, and H. Ney, "rwthlm the RWTH Aachen university neural network language modeling toolkit", Proc. Interspeech, 2014
23. Y. Shi, M.-Y. Hwang, K. Yao, and M. Larson. "Speed up of recurrent neural network language models with sentence independent subsampling stochastic gradient descent", Proc. Interspeech, 2013
24. A. Mnih and Y. W. Teh, "A fast and simple algorithm for training neural probabilistic language models, " Proc. International Conference on Machine Learning, 2012
25. S. Young, G. Evermann, M. J. F. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, A. Ragni, V. Valtchev, P. C. Woodland and C. Zhang, "The HTK book (for HTK 3. 5)", Cambridge University Engineering Department, 2015
26. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsk, G. Stemmer and K. Vesel "The Kaldi speech recognition toolkit, " Proc. ASRU Workshop, 2011
27. A. Deoras, T. Mikolov, S. Kombrink, and K. Church, "Approximate inference: A sampling based modeling technique to capture complex dependencies in a language model, " Speech Communication, vol. 55, no. 1, pp. 162-177, 2013
28. A. Stolcke, "SRILM-an extensible language modeling toolkit, " Proc. Interspeech, 2002
29. E. Arisoy, S. F. Chen, B. Ramabhadran, and A. Sethy, "Converting neural network language models into back-off language models for efficient decoding in automatic speech recognition, " IEEE/ACM Trans. Audio, Speech, and Language Processing, vol. 22, no. 1, pp. 184-192, 2014
30. X. Chen, T. Tan, X. Liu, P. Lanchantin, M. Wan, M. J. F. Gales and P. C. Woodland, "Recurrent Neural Network Language Model Adaptation for Multi-Genre Broadcast Speech Recognition, " Proc. Interspeech, 2015
31. Y. Shi, M. Larson, and C. M. Jonker, "K-component recurrent neural network language models using curriculum learning, " Proc. ASRU Workshop, 2013
32. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, D. Reidsma, and P. Wellner, "The AMI meeting corpus: A preannouncement, " Machine learning for multimodal interaction, pp. 28-39. Springer, 2006
33. K. Vesely, A. Ghoshal, L. Burget, and D. Povey, "Sequencediscriminative training of deep neural networks., " Proc. Interspeech, 2013
34. L. Mangu, E. Brill, and A. Stolcke. "Finding consensus in speech recognition: word error minimization and other applications of confusion networks, " Computer Speech and Language, 14(4): 373-400, 2000