Rapid bayesian learning for recurrent neural network language model

Proceedings of the 9th International Symposium on Chinese Spoken Language Processing, ISCSLP 2014

Volumn , Issue , 2014, Pages 34-38

  Chien, Jen Tzung ^a   Ku, Yuan Chu ^a   Huang, Mou Yue ^a  

^a NATIONAL CHIAO TUNG UNIVERSITY   (Taiwan)

Author keywords

Bayesian learning; Hessian matrix; Recurrent neural network language model; Speech recognition

Indexed keywords

COMPUTATIONAL LINGUISTICS; LEARNING SYSTEMS; MATRIX ALGEBRA; MAXIMUM LIKELIHOOD; PARAMETER ESTIMATION; SPEECH RECOGNITION;

BAYESIAN LEARNING; CURSE OF DIMENSIONALITY; GRADIENT VECTORS; HESSIAN MATRICES; HIGH-DIMENSIONAL; MAXIMUM A POSTERIORI CRITERIA; SYNAPTIC WEIGHT; WORD ERROR RATE;

RECURRENT NEURAL NETWORKS;

EID: 84912113889     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ISCSLP.2014.6936640     Document Type: Conference Paper

References (18)

1. G. Saon and J.-T. Chien, "Large-vocabulary continuous speech recognition systems-a look at some recent advances , " IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 18-33, 2012.
2. J.-T. Chien, "Association pattern language modeling , " IEEE Transactions on Audio, Speech and Language Processing, vol. 14, no. 5, pp. 1719-1728, 2006.
3. R. Kneser and H. Ney, "Improved backing-off for mgram language modeling , " Proceedings of International Conference on Acoustic, Speech, and Signal Processing (ICASSP), pp. 181-184, 1995.
4. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling , " Computer Speech and Language, vol. 13, no. 4, pp. 359-394, 1999.
5. R. Kuhn and R. De Mori, "A cache-based natural language model for speech recognition , " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 6, pp. 570-583, 1990.
6. J.-T. Chien and C.-H. Chueh, "Dirichlet class language models for speech recognition , " IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 3, pp. 482-495, 2011.
7. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model , " Proceedings of Annual Conference of International Speech Communication Association (INTERSPEECH), pp. 1045-1048, 2010.
8. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, "A neural probabilistic language model , " Journal of Machine Learning Research, vol. 3, pp. 1137-1155, 2003.
9. R. M. Neal, "Bayesian learning for neural networks , " Ph.D. dissertation, University of Toronto, 1995.
10. D. J. C. MacKay, "A practical Bayesian framework for backpropagation networks , " Neural Computation, vol. 4, no. 2, pp. 448-472, 1992.
11. C. M. Bishop, Pattern Recognition and Machine Learning. Springer Science, 2006.
12. Y. Bengio, P. Simard, and P. Frasconi, "Learning longterm dependencies with gradient descent is difficult , " IEEE Transactions on Neural Networks, vol. 5, pp. 157-166, 1994.
13. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, "One billion word benchmark for measuring progress in statistical language modelling , " 2014. [Online]. Available: http://arxiv.org/abs/1312.3005
14. A. Stolcke, "SRILM-an extensible language modeling toolkit , " Proceedings of International Conference on Spoken Language Processing, pp. 901-904, 2002.
15. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and G. Hanza, "RNNLM-recurrent neural network language modeling toolkit , " Proceeding of IEEE ASRU Workshop, 2011.
16. T. Mikolov, S. Konbrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model , " Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5528-5531, 2011.
17. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, , and J. H. Cernocky, "Empirical evaluation and combination of advanced language modeling techniques , " Proceedings of Annual Conference of International Speech Communication Association (INTERSPEECH), pp. 605-608, 2011.
18. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, "The Kaldi speech recognition toolkit , " Proceeding of IEEE ASRU Workshop, 2011.