Improving the training and evaluation efficiency of recurrent neural network language models

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 5401-5405

  Chen, X ^a   Liu, X ^a   Gales, M J F ^a   Woodland, P C ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

GPU; language models; recurrent neural network; speech recognition

Indexed keywords

AUDIO SIGNAL PROCESSING; C (PROGRAMMING LANGUAGE); COMPUTATIONAL LINGUISTICS; GRAPHICS PROCESSING UNIT; PROGRAM PROCESSORS; RECURRENT NEURAL NETWORKS; SPEECH COMMUNICATION;

CLASS-BASED; LANGUAGE MODEL; NORMALISATION; OUTPUT LAYER; REGULARISATION; TEST TIME; TRAINING TIME;

SPEECH RECOGNITION;

EID: 84946051719     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7179003     Document Type: Conference Paper

References (30)

1. Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, " Recurrent neural network based language model", Proc. Interspeech 2010, pp. 1045-1048.
2. Tomas Mikolov, Stefan Kombrink, Lukas Burget, J.H. Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network language model", Proc. ICASSP 2011, pp. 5528-5531.
3. Anoop Deoras, Tomas Mikolov, Stefan Kombrink, Martin Karafiat, and Sanjeev Khudanpur, " Variational approximation of long-span language models for LVCSR", Proc. ICASSP 2011,pp.5532-5535.
4. Gwenole Lecorve and Petr Motlicek, "Conversion of recurrent neural network language models to weighted finite state transducers for automatic speech recognition," Tech. Report, IDIAP, 2012.
5. Martin Sundermeyer, Ilya Oparin, Jean-Luc Gauvain, Ben Freiberg, Ralf Schluter, and Hermann Ney, "Comparison of feedforward and recurrent neural network language models", Proc. ICASSP 2013, pp. 8430-8434.
6. Gregoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio, " Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding", Proc. lnterspeech, 2013, pp. 3771-3775.
7. Kaisheng Yao, Geoffrey Zweig, Mei-Yuh Hwang, Yangyang Shi, and Dong Yu, " Recurrent neural networks for language understanding", Proc. lnterspeech 2013, pp. 2524-2528.
8. Gregoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio, " Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding", Proc. Interspeech,2013.
9. Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig, "Joint language and translation modeling with recurrent neural networks", Proc. EMNLP 2013, pp. 1044-1054.
10. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul, " Fast and robust neural network joint models for statistical machine translation", Proc. 52nd Annual Meeting o/the ACL, 2014.
11. Xie Chen, Yongqiang Wang, Xunying Liu, Mark Gales, and P. C. Woodland, "Efficient training of recurrent neural network language models using spliced sentence bunch", Proc. lnterspeech 2014.
12. Yongzhe Shi, Wei-Qiang Zhang, Meng Cai, and Jia Liu, "Efficient one-pass decoding with NNLM for speech recognition", Signal Processing Letters, vol. 21, no. 4, pp. 377-381, 2014.
13. Yongzhe Shi, Wei-Qiang Zhang, Meng Cai, and Jia Liu, " Variance regularization of RNNLM for speech recognition", Proc. ICASSP 2014.
14. Xunying Liu, Yongqiang Wang, Xie Chen, Mark Gales, and P. C. Woodland, "Efficient lattice rescoring using recurrent neural network language models", Proc. ICASSP 2014.
15. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams, Learning representations by back-propagating errors, MIT Press, 1988.
16. Holger Schwenk, "Continuous space language models", Computer Speech &Language, vol. 21, no. 3, pp. 492-518, 2007.
17. Ahmad Emami and Lidia Mangu, "Empirical study of neural network language models for arabic speech recognition", Proc. IEEE Workshop on ASRU 2007, pp. 147-152.
18. Junho Park, Xunying Liu, Mark J. F. Gales, and P. C. Woodland, " Improved neural network based language modelling and adaptation", Proc. lnterspeech 2010, pp. 1041-1044.
19. Hai-Son Le, Ilya Oparin, Alexandre Allauzen, J Gauvain, and Franois Y von, " Structured output layer neural network language models for speech recognition", IEEE Trans. Audio, Speech, and Language Processing, vol. 21, no. 1, pp. 197-206, 2013.
20. Geoffrey Zweig and Konstantin Makarychev, " Speed regularization and optimality in word classing", Proc. ICASSP 2013, pp.8237-8241.
21. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and 1enifer C Lai, "Class-based n-gram models of natural language", Computational linguistics, vol. 18, no. 4, pp. 467-479, 1992.
22. Diamanino Caeiro and Andrej Ljolje, "Multiple parallel hidden layers and other improvements to recurrent neural network language modeling", Proc. ICASSP, 2013, pp. 8426-8429.
23. Hong-Kwang Kuo, Ebru Arisoy, Ahmad Emami, and Paul Vozila, "Large scale hierarchical neural network language models", Proc. Interspeech, 2012.
24. Reinhard Kneser and Hermann Ney, " Improved clustering techniques for class-based statistical language modelling", Proc. Eurospeech 1993.
25. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig, " Linguistic regularities in continuous space word representations", Proc. NAACL-HLT 2013, pp. 746-751.
26. Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu, " On parallelizability of stochastic gradient descent for speech DNNs", Proc. ICASSP, 2014.
27. Xie Chen, Adam Eversole, Gang Li, Dong Yu, and Frank Seide, " Pipelined back-propagation for context-dependent deep neural networks", Proc. lnterspeech, 2012.
28. G. Evermann, H. Y. Chan, M. J. F. Gales, B. Jia, D. Mrva, P. C. Woodland, and K. Yu, " Training LVCSR systems on thousands of hours of data", Proc. ICASSP 2005, pp. 209-212.
29. Ivan Bulyko, Mari Ostendorf, and Andreas Stolcke, "Getting more mileage from web text sources for conversational speech language modeling using class-dependent mixtures", Proc. HLT, ACL, 2003, pp. 7-9.
30. Yongzhe Shi, Wei-Qiang Zhang, Meng Cai, and Jia Liu, " Temporal kernel neural network language modeling", Proc. ICASSP 2013.