Recurrent neural network language model training with noise contrastive estimation for speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 5411-5415

  Chen, X ^a   Liu, X ^a   Gales, M J F ^a   Woodland, P C ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

GPU; language model; noise contrastive estimation; recurrent neural network; speech recognition

Indexed keywords

ACOUSTIC NOISE; AUDIO SIGNAL PROCESSING; COMPUTATIONAL LINGUISTICS; GRAPHICS PROCESSING UNIT; RECURRENT NEURAL NETWORKS; SPEECH; SPEECH COMMUNICATION;

APPLICATION AREA; COMPUTATIONAL COSTS; CONVERSATIONAL TELEPHONE SPEECH RECOGNITION; LANGUAGE MODEL; LARGE VOCABULARY; NETWORK LANGUAGE; TRAINING AND TESTING; TRAINING SPEED;

SPEECH RECOGNITION;

EID: 84946014155     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7179005     Document Type: Conference Paper

References (30)

1. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, "Recurrent neural network based language model," in Proc. ISCA Interspeech, Makuhari, Japan, 2010, pp. 1045-1048.
2. Tomas Mikolov, Stefan Kombrink, Lukas Burget, J.H. Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network language model," in Proc. IEEE ICASSP, Prague, Czech Republic, 2011, pp. 5528-5531.
3. Anoop Deoras, Tomas Mikolov, Stefan Kombrink, Martin Karafiát, and Sanjeev Khudanpur, "Variational approximation of long-span language models for LVCSR," in Proc. IEEE ICASSP, Prague, Czech Republic, 2011, pp. 5532-5535.
4. Gwénolé Lecorvé and Petr Motlicek, "Conversion of recurrent neural network language models to weighted finite state transducers for automatic speech recognition," in Proc. ISCA Interspeech, Portland, OR, 2012.
5. Martin Sundermeyer, Ilya Oparin, Jean-Luc Gauvain, Ben Freiberg, Ralf Schluter, and Hermann Ney, "Comparison of feedforward and recurrent neural network language models," in Proc. IEEE ICASSP, Vancouver, Canada, May 2013, pp. 8430-8434.
6. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio, "Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding," in Proc. ISCA Interspeech, Lyon, France, 2013, pp. 3771-3775.
7. Kaisheng Yao, Geoffrey Zweig, Mei-Yuh Hwang, Yangyang Shi, and Dong Yu, "Recurrent neural networks for language understanding," in Proc. ISCA Interspeech, Lyon, France, 2013, pp. 2524-2528.
8. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio, "Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding," in Proc. ISCA Interspeech, Lyon, France, 2013, pp. 3771-3775.
9. Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig, "Joint language and translation modeling with recurrent neural networks," in Proc. ACL EMNLP, Seattle, Washington, 2013, pp. 1044-1054.
10. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul, "Fast and robust neural network joint models for statistical machine translation," in Proc. ACL, Association for Computational Linguistics, Baltimore, Maryland, USA, June 2014, pp. 1370-1380.
11. Yangyang Shi, Mei-Yuh Hwang, Kaisheng Yao, and Martha Larson, "Speed up of recurrent neural network language models with sentence independent subsampling stochastic gradient descent," in Proc. ISCA Interspeech, Lyon, France, 2013, pp. 1203-1206.
12. Zhiheng Huang, Geoffrey Zweig, Michael Levit, Benoit Dumoulin, Barlas Oguz, and Shawn Chang, "Accelerating recurrent neural network training via two stage classes and parallelization," in Proc. IEEE ASRU, Olomouc, Czech Republic, December, 2013, pp. 326-331.
13. Yongzhe Shi, Wei-Qiang Zhang, Meng Cai, and Jia Liu, "Ef-ficient one-pass decoding with nnlm for speech recognition," IEEE Signal Processing Letters, vol. 21, no. 4, pp. 377-381, 2014.
14. Yongzhe Shi,Wei-Qiang Zhang, Meng Cai, and Jia Liu, "Variance regularization of rnnlm for speech recognition," in Proc. IEEE ICASSP, Florence, Italy, 2014, pp. 4893-4897.
15. Xie Chen, Yongqiang Wang, Xunying Liu, Mark Gales, and Phil Woodland, "Efficient training of recurrent neural network language models using spliced sentence bunch," in Proc. ISCA Interspeech, Singapore, 2014, pp. 641-645.
16. Xie Chen, Xunying Liu, Mark Gales, and Phil Woodland, "Improving the training and evaluation efficiency of recurrent neural network language models," to appear in Proc. IEEE ICASSP, Brisbane, Australia, 2015.
17. Michael U Gutmann and Aapo Hyvärinen, "Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics," Journal of Machine Learning Research, vol. 13, no. 1, pp. 307-361, 2012.
18. Andriy Mnih and YeeWhye Teh, "A fast and simple algorithm for training neural probabilistic language models," in Proc. ICML, Edinburgh, UK, 2012, pp. 1751-1758.
19. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang, "Decoding with large-scale neural language models improves translation," in Proc. ACL EMNLP, Seattle, Washington, 2013, pp. 1387-1392.
20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean, "Distributed representations of words and phrases and their compositionality," in Proc. NIPS, Lake Tahoe, NV, 2013, pp. 3111-3119.
21. Xunying Liu, Yongqiang Wang, Xie Chen, Mark Gales, and PhilWoodland, "Efficent lattice rescoring using recurrent neural network language models," in Proc. IEEE ICASSP, Florence, Italy, 2014, pp. 4941-4945.
22. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, Learning representations by back-propagating errors, MIT Press, Cambridge, MA, USA, 1988.
23. Holger Schwenk, "Continuous space language models," Computer Speech and Language, vol. 21, no. 3, pp. 492-518, 2007.
24. Ahmad Emami and Lidia Mangu, "Empirical study of neural network language models for arabic speech recognition," Proc. ASRU, Kyoto, Japan, 2007, pp. 147-152.
25. Junho Park, Xunying Liu, Mark J. F. Gales, and Phil Woodland, "Improved neural network based language modelling and adaptation," in Proc. ISCA Interspeech, Makuhari, Japan, 2010, pp. 1041-1044.
26. Hai-Son Le, Ilya Oparin, Alexandre Allauzen, J. Gauvain, and François Yvon, "Structured output layer neural network language models for speech recognition," IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 1, pp. 197-206, 2013.
27. G. Evermann, H. Y. Chan, M. J. F. Gales, B. Jia, D. Mrva, P. C. Woodland, and K. Yu, "Training LVCSR systems on thousands of hours of data," in Proc. IEEE ICASSP, Philadelphia, PA, 2005, vol. 1, pp. 209-212.
28. Ivan Bulyko, Mari Ostendorf, and Andreas Stolcke, "Getting more mileage from web text sources for conversational speech language modeling using class-dependent mixtures," in Proc. HLT-NAACL, Stroudsburg, PA, USA, vol. 2 pp. 7-9.
29. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson, "One billion word benchmark for measuring progress in statistical language modeling," Tech. Rep, Google, 2013.
30. Andreas Stolcke, "SRILM-an extensible language modeling toolkit," in Proc. ISCA ICSLP, Denver, CO, 2002, pp. 901-904.