Prefix tree based N-best list re-scoring for recurrent neural network language model used in speech recognition system

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2013, Pages 3419-3423

  Si, Yujing ^a   Zhang, Qingqing ^a   Li, Ta ^a   Pan, Jielin ^a   Yan, Yonghong ^a  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

Author keywords

Bunch mode; N best list; PTNR; RNNLM; Speedup

Indexed keywords

FORESTRY; RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION;

BUNCH MODE; N-BEST LIST; PTNR; RNNLM; SPEEDUP;

COMPUTATIONAL LINGUISTICS;

FORESTRY; LANGUAGES; MODELS; NEURAL NETWORKS;

EID: 84906240855     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. A. Deoras, T. Mikolov, and K. Church, "A fast re-scoring strategy to capture long-distance dependencies, " in Proceedings of EMNLP, 2011, pp. 1116-1127.
2. S. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling, " Computer Speech & Language, vol. 13, no. 4, pp. 359-393, 1999.
3. R. Rosenfeld, "Two decades of statistical language modeling: Where do we go from here?" Proceedings of the IEEE, vol. 88, no. 8, pp. 1270-1278, 2000.
4. I. Oparin, M. Sundermeyer, H. Ney, and J. Gauvain, "Performance analysis of neural networks in combination with n-gram language models, " Proc. of ICASSP12, pp. 5005-5008, 2012.
5. C. Chelba, J. Schalkwyk, T. Brants, V. Ha, B. Harb, W. Neveitt, C. Parada, and P. Xu, "Query language modeling for voice search, " in Spoken Language Technology Workshop (SLT), 2010 IEEE. IEEE, 2010, pp. 127-132.
6. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model, " in Eleventh Annual Conference of the International Speech Communication Association, 2010, pp. 1045-1048.
7. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Cernocky, "Empirical evaluation and combination of advanced language modeling techniques, " in Twelfth Annual Conference of the International Speech Communication Association, 2011, pp. 605-608.
8. S. Kombrink, T. Mikolov, M. Karafiat, and L. Burget, "Recurrent neural network based language modeling in meeting recognition, " Twelfth Annual Conference of the International Speech Communication Association, 2011.
9. H. Schwenk, "Continuous space language models, " Computer Speech & Language, vol. 21, no. 3, pp. 492-518, 2007.
10. L. Hai Son, A. Allauzen, and F. Yvon, "Measuring the influence of long range dependencies with neural network language models, " in Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT. Association for Computational Linguistics, 2012, pp. 1-10.
11. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model, " in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 5528-5531.
12. T. Mikolov, A. Deoras, D. Povey, and L. Burget, "Strategies for training large scale neural network language models, " in In: ASRU 2011, 2011.
13. H. Schwenk and J. Gauvain, "Training neural network language models on very large corpora, " in Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2005, pp. 201-208.
14. H.-S. Le, I. Oparin, A. Allauzen, J. Gauvain, and F. Yvon, "Structured output layer neural network language model, " in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 5524-5527.
15. H.-K. J. Kuo, E. Arsoy, A. Emami, and P. Vozila, "Large scale hierarchical neural network language models, " in Eleventh Annual Conference of the International Speech Communication Association, 2012.
16. K. Chen, W. Bao, and H. Chi, "Speed up training of the recurrent neural network based on constrained optimization techniques, " Journal of Computer Science and Technology, vol. 11, no. 6, pp. 581-588, 1996.
17. G. Lecorve and P. Motlicek, "Conversion of recurrent neural network language models to weighted finite state transducers for automatic speech recognition, " in 13th Annual Conference of the International Speech Communication Association, 2012.
18. H. Schwenk and J.-L. Gauvain, "Connectionist language modeling for large vocabulary continuous speech recognition, " in Acoustics, Speech, and Signal Processing (ICASSP), 2002 IEEE International Conference on, vol. 1. IEEE, 2002, pp. 762-765.
19. Y. Si, T. Li, S. Cai, J. Pan, and Y. Yan, "Recurrent neural network language model in mandarin voice input system, " in Natural Computation (ICNC), 2012 Eighth International Conference on. IEEE, 2012, pp. 270-274.
20. J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel, "Using phipac to speed error back-propagation learning, " in Acoustics, Speech, and Signal Processing, 1997. ICASSP-97., 1997 IEEE International Conference on, vol. 5. IEEE, 1997, pp. 4153-4156.
21. R. D. Yoshua Bengio and P. Vincent, "A neural probabilistic language model, " Journal of Machine Learning Research, pp. 1137- 1155, 2003.
22. A. Rastrow, M. Dreyer, A. Sethy, S. Khudanpur, B. Ramabhadran, and M. Dredze, "Hill climbing on speech lattices: A new rescoring framework, " in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 5032-5035.
23. Y. Si, T. Li, J. Pan, and Y. Yan, "A prefix tree based n-best list re-scoring strategy for recurrent neural network language model, " Chinese Journal of Electronics, 2013.
24. K. Heafield, "Kenlm: Faster and smaller language model queries, " in Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, UK, 2011.
25. J. Shao, T. Li, Q. Zhang, Q. Zhao, and Y. Yan, "A one-pass real-time decoder using memory-efficient state network, " IEICE TRANSACTIONS on Information and Systems, vol. 91, no. 3, pp. 529-537, 2008.
26. A. Stolcke, "Srilm-an extensible language modeling toolkit, " in Seventh International Conference on Spoken Language Processing, 2002.
27. M. Boden, "A guide to recurrent neural networks and backpropagation, " The DALLAS project. Report from the NUTEK-supported project AIS-8, SICS. Holst: Application of data analysis with learning systems, 2001.
28. T. Mikolov, S. Kombrink, A. Deoras, L. Burget, and J. Cernocky, "Rnnlm - recurrent neural network language modeling toolkit, " in In: ASRU 2011 Demo Session Brief description of the RNN LM toolkit that is available on this website, 2011.