Scaling recurrent neural network language models

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 5391-5395

  Williams, Will ^a   Prasad, Niranjani ^a   Mrva, David ^a   Ash, Tom ^a   Robinson, Tony ^a  

^a Cantab Research Ltd   (United Kingdom)

Author keywords

GPU; language modelling; recurrent neural network; RNNLM; speech recognition

Indexed keywords

AUDIO SIGNAL PROCESSING; COMPUTATIONAL LINGUISTICS; GRAPHICS PROCESSING UNIT; PROGRAM PROCESSORS; RECURRENT NEURAL NETWORKS; SPEECH COMMUNICATION; SPEECH RECOGNITION;

COMPUTATIONAL COSTS; LANGUAGE MODELLING; N-GRAM MODELS; RNNLM; SCALING PROPERTIES; WORD ERROR RATE; WORD LANGUAGES; WORD PREDICTION;

MODELING LANGUAGES;

EID: 84946091011     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7179001     Document Type: Conference Paper

References (22)

1. Anthony J. Robinson, "An application of recurrent nets to phone probability estimation," Neural Networks, IEEE Transactions on, vol. 5, no. 2, pp. 298-305, 1994.
2. Ronald J. Williams, "Complexity of exact gradient computation algorithms for recurrent neural networks," Tech. Rep., 1989.
3. Mike Schuster and Kuldip K. Paliwal, "Bidirectional recurrent neural networks," Signal Processing, IEEE Transactions on, vol. 45, no. 11, pp. 2673-2681, 1997.
4. Tomáš Mikolov, Statistical language models based on neural networks, Ph.D. Thesis, Brno University of Technology, 2012.
5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson, "One Billion Word Benchmark for measuring progress in statistical language modeling,".
6. Dietrich Klakow, "Selecting articles from the language model training corpus," in Acoustics, Speech, and Signal Processing, 2000. ICASSP'00. Proceedings. 2000 IEEE International Conference on. IEEE, 2000, vol. 3, pp. 1695-1698.
7. Robert C. Moore and William Lewis, "Intelligent selection of language model training data," in Proceedings of the ACL 2010 Conference Short Papers. Association for Computational Linguistics, 2010, pp. 220-224.
8. Christian Buck, Kenneth Heafield, and Bas van Ooyen, "N-gram counts and language models from the common crawl," LREC, 2014.
9. Kenneth Heafield, "KenLM: Faster and smaller language model queries," in Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, 2011, pp. 187-197.
10. Quoc V Le, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, and Andrew Y. Ng, "Building high-level features using large scale unsupervised learning," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8595-8598.
11. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Cěrnocky, and Sanjeev Khudanpur, "Recurrent neural network based language model," in Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH), 2010, pp. 1045-1048.
12. Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Razvan Pascanu, "Advances in optimizing recurrent networks," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8624-8628.
13. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and J Cernocky, "Rnnlm-recurrent neural network language modeling toolkit,".
14. Ilya Sutskever, James Martens, and Geoffrey E Hinton, "Generating text with recurrent neural networks," in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 1017-1024.
15. Zhiheng Huang, Geoffrey Zweig, Michael Levit, Benoit Dumoulin, Barlas Oguz, and Shawn Chang, "Accelerating recurrent neural network training via two stage classes and parallelization," in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 326-331.
16. Xie Chen, YongqiangWang, Xunying Liu, Mark JF Gales, and Philip C Woodland, "Efficient GPU-based training of recurrent neural network language models using spliced sentence bunch," Proc. ISCA Interspeech, 2014.
17. Boxun Li, Erjin Zhou, Bo Huang, Jiayi Duan, Yu Wang, Ningyi Xu, Jiaxing Zhang, and Huazhong Yang, "Large scale recurrent neural network on gpu," in Neural Networks (IJCNN), 2014 International Joint Conference on. IEEE, 2014, pp. 4062-4069.
18. T. Tieleman and G. Hinton, "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude," COURSERA: Neural Networks for Machine Learning, vol. 4, 2012.
19. Andriy Mnih and Koray Kavukcuoglu, "Learning word embeddings efficiently with noise-contrastive estimation," in Advances in Neural Information Processing Systems, 2013, pp. 2265-2273.
20. Anthony Rousseau, Paul Deléglise, and Yannick Estève, "TED-LIUM: an Automatic Speech Recognition dedicated corpus," in LREC, 2012, pp. 125-129.
21. J. Pino, A. Waite, T. Xiao, A. de Gispert, F. Flego, and W. Byrne, "The University of Cambridge Russian-English system at WMT13," in Proceedings of the Eighth Workshop on Statistical Machine Translation, 2013, pp. 198-203.
22. BNC Consortium et al., "The British National Corpus, version 3 (BNC XML Edition), distributed by Oxford University Computing Services on behalf of the BNC Consortium," 2007.