Accelerating recurrent neural network training via two stage classes and parallelization

2013 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2013 - Proceedings

Volumn , Issue , 2013, Pages 326-331

  Huang, Zhiheng ^a   Zweig, Geoffrey ^b   Levit, Michael ^a   Dumoulin, Benoit ^a   Oguz, Barlas ^a   Chang, Shawn ^a  

^a MICROSOFT   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

hierarchical classes; language modeling; parallelization; recurrent neural network (RNN); speed up

Indexed keywords

HIERARCHICAL CLASSES; LANGUAGE MODEL; PARALLELIZATIONS; RECURRENT NEURAL NETWORK (RNN); SPEED UP;

COMPUTATIONAL LINGUISTICS; SPEECH RECOGNITION;

RECURRENT NEURAL NETWORKS;

EID: 84893657400     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ASRU.2013.6707751     Document Type: Conference Paper

References (21)

1. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling, " in ACL, 1996, pp. 310-318.
2. Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin, T. Hoffman, T. Poggio, and J. Shawe-taylor, "A neural probabilistic language model, " Journal of Machine Learning Research,, no. 3, 2003.
3. H. K. J. Kuo, E. Arisoy, A. Emami, and P. Vozila, "Large scale hierarchical neural network language models, " in INTERSPEECH, 2012.
4. R. Rosenfeld, Adaptive Statistical Language Modeling: A Maximum Entropy Approach, Ph.D. thesis, Carnegie Mellon University, 1994.
5. S. F. Chen, "Performance prediction for exponential lanuage models, " in HLT-NAACL, 2009, pp. 450-458.
6. T. Alumae and M. Kurimo, "Efficient estimation of maximum entropy language models with n-gram features: An srilm extension, " in INTERSPEECH, 2010.
7. P. Xu, S. Khudanpur, and A. Gunawardana, "Randomized maximum entropy language models, " in IEEE ASRU, 2011.
8. G. Zweig and S. Chang, "Personalizing model m for voice-search, " in INTERSPEECH, 2011.
9. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model, " in INTERSPEECH, 2010.
10. T. Mikolov, S. Kombrink, L. Burget, J. H. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model, " in ICASSP, 2011.
11. S. Kombrink, T. Mikolov, M. Karafiat, and L. Burget, "Recurrent neural network based language modeling in meeting recognition, " in INTERSPEECH, 2011.
12. Y. Shi, P. Wiggers, and C. M. Jonker, "Towards recurrent neural networks language models with linguistic and contextual features, " in INTERSPEECH, 2012.
13. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. H. Cernocky, "Strategies for training large scale neural network languge models, " in ASRU, 2011.
14. F. Morin and Y. Bengio, "Hierarchical probabilistic neural network language model, " in AISTATS, 2005.
15. A. Mnih and G. E. Hinton, "A scalable hierarchical distributed languaged model, " in NIPS, 2009.
16. H. S. Le and I. Oparin and A. Allauzen and J. L. Gauvain and F. Yvon, "Structured output layer neural network language model, " ICCASP, 2011.
17. G. Zweig and K. Makarychev, "Speed regularization and optimality in word classing, " in ICASSP, 2013.
18. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng, "Large scale distributed deep networks, " in NIPS, 2012.
19. J. Goodman, "Classes for fast maximum entropy training, " in ICASSP, 2001.
20. T. Mikolov, S. Kombrink, A. Deoras, L. Burget, and J. Cernocky, "Rnnlm - recurrent neural network language modeling toolkit, " in ASRU Demo Session, 2011.
21. D. E. Rumelhart, G. E. Hinton, and R. J. Williams., "Learning internal representations by back-propagating errors, " in Nature, 1986, pp. 533- 536. (Pubitemid 16025374)