From feedforward to recurrent LSTM neural networks for language modeling

IEEE Transactions on Audio, Speech and Language Processing

Volumn 23, Issue 3, 2015, Pages 517-529

  Sundermeyer, Martin ^a   Ney, Hermann ^a,b   Schluter, Ralf ^a  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b CNRS   (France)

Author keywords

Feedforward neural network; Kneser Ney smoothing; language modeling; long short term memory (LSTM); recurrent neural network (RNN)

Indexed keywords

BRAIN; COMPLEX NETWORKS; COMPUTATIONAL LINGUISTICS; FEEDFORWARD NEURAL NETWORKS; MODELING LANGUAGES; PROBABILITY DISTRIBUTIONS; SPEECH RECOGNITION; RECURRENT NEURAL NETWORKS;

ARCHITECTURAL STRUCTURE; KNESER-NEY SMOOTHING; LANGUAGE MODEL; LARGE VOCABULARY SPEECH RECOGNITION; LONG SHORT TERM MEMORY; RECURRENT NEURAL NETWORK (RNN); RELATIVE FREQUENCIES; STRONG CORRELATION;

RECURRENT NEURAL NETWORKS; FEEDFORWARD NEURAL NETWORKS;

EID: 84924036578     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2015.2400218     Document Type: Article

References (53)

1. R. Rosenfeld, "Two decades of statistical language modeling: Where do we go from here?," Proc. IEEE, vol. 88, no. 8, pp. 359-394, Aug. 2000.
2. S. M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer," IEEE Trans. Acoust., Speech, Signal Process., vol. 35, no. 3, pp. 400-401, Mar. 1987.
3. R. Kneser and H. Ney, "Improved backing-off for M-Gram language modeling," in Proc. ICASSP, 1995, pp. 181-184.
4. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling," Comput. Speech Lang., vol. 13, no. 4, pp. 359-393, 1999.
5. H. Ney, S. Martin, F. Wessel, S. Young, and G. Bloothooft, "Statistical language modeling using leaving-one-out," in Corpus-Based Methods in Language And Speech Processing. Norwell, MA, USA: Kluwer, 1997, ch. 6, pp. 174-207.
6. Y. Bengio and R. Ducharme, "A neural probabilistic language model,"in Proc. NIPS, 2000, vol. 13, pp. 933-938.
7. H. Schwenk, "Continuous space language models," Comput. Speech Lang., vol. 21, pp. 492-518, 2007.
8. H.-S. Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language models for speech recognition,"IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 1, pp. 197-206, Jan. 2013.
9. T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur, "Recurrent neural network based language model," in Proc. Interspeech, 2010, pp. 1045-1048.
10. M. Sundermeyer, R. Schlüter, and H. Ney, "LSTM neural networks for language modeling," in Proc. Interspeech, 2012.
11. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition- the shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
12. H. Ney and S. Ortmanns, "Dynamic programming search for continuous speech recognition," IEEE Signal Process. Mag., vol. 16, no. 5, pp. 64-83, Sep. 1999.
13. M. Sundermeyer, T. Alkhouli, J. Wuebker, and H. Ney, "Translation modeling with bidirectional recurrent neural networks," in Proc. EMNLP, 2014, pp. 14-25.
14. T. Mikolov, S. Kombrink, L. Burget, J. Černocký, and S. Khudanpur, "Extensions of recurrent neural network language model," in Proc. ICASSP, 2011, pp. 5528-5531.
15. E. Ar soy, T. N. Sainath, B. Kingsbury, and B. Ramabhadran, "Deep neural network language models," in Proc. NAACL-HLT Workshop, 2012, pp. 20-28.
16. I. Oparin, M. Sundermeyer, H. Ney, and J.-L. Gauvain, "Performance analysis of neural networks in combination with -gram language models," in Proc. ICASSP, 2012, pp. 5005-5008.
17. M. Sundermeyer, I. Oparin, J.-L. Gauvain, B. Freiberg, R. Schlüter, and H. Ney, "Comparison of feedforward and recurrent neural network language models," in Proc. ICASSP, 2013, pp. 8430-8434.
18. H. S. Le, A. Allauzen, and F. Yvon, "Measuring the influence of long range dependencies with neural network language models," in Proc. NAACL-HLT Workshop, 2012, pp. 1-10.
19. S. Kombrink, T. Mikolov, M. Karafiát, and L. Burget, "Recurrent neural network based language modeling in meeting recognition," in Proc. Interspeech, 2011, pp. 2877-2880.
20. Y. Si, Q. Zhang, T. Li, J. Pan, and A. Yan, "Prefix tree based N-best list re-scoring for recurrent neural network language model used in speech recognition system," in Proc. Interspeech, 2013, pp. 3419-3423.
21. C. Chelba and F. Jelinek, "Recognition performance of a structured language model," in Proc. Eurospeech, 1999, vol. 4, pp. 1567-1570.
22. A. Deoras, T. Mikolov, and K. Church, "A fast re-scoring strategy to capture long-distance dependencies," in Proc. EMLNP, 2011, pp. 1116-1127.
23. M. Auli, M. Galley, C. Quirk, and G. Zweig, "Joint language and translation modeling with recurrent neural networks," in Proc. EMNLP, 2013, pp. 1044-1054.
24. X. Liu, Y. Wang, X. Chen, M. J. F. Gales, and P. C. Woodland, "Efficient lattice rescoring using recurrent neural network language models," in Proc. ICASSP, 2014, pp. 4941-4945.
25. M. Sundermeyer, Z. Tüske, R. Schlüter, and H. Ney, "Lattice decoding and rescoring with long-span neural network language models," in Proc. Interspeech, 2014, pp. 661-665.
26. X. Liu, M. J. F. Gales, and P. C. Woodland, "Use of contexts in language model interpolation and adaptation," Comput. Speech Lang., vol. 27, no. 1, pp. 301-321, 2013.
27. L. Mangu, E. Brill, and A. Stolcke, "Finding consensus in speech recognition: Word error minimization and other applications of confusion networks,"Comput. Speech Lang., vol. 14, no. 4, pp. 373-400, 2000.
28. Z. Huang, G. Zweig, and B. Dumoulin, "Cache based recurrent neural network language model inference for first pass speech recognition,"in Proc. ICASSP, 2015, pp. 6404-6408.
29. T. Hori, Y. Kubo, and A. Nakamura, "Real-time one-pass decoding with recurrent neural network language model for speech recognition,"in Proc. ICASSP, 2015, pp. 6414-6418.
30. C. M. Bishop, "Single-layer networks," in Neural Networks for Pattern Recognition . Oxford, U.K.: Oxford Univ. Press, 1995, ch. 3, pp. 77-115.
31. J. Goodman, "Classes for fast maximum entropy training," in Proc. ICASSP, 2001, pp. 561-564.
32. th Int. Workshop Artif. Intell. Statist., 2005, pp. 246-252.
33. R. Kneser and H. Ney, "Forming word classes by statistical clustering for statistical language modelling," in Proc. QUALICO, 1991, pp. 221-226.
34. P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. Della Pietra, and J. C. Lai, "Class-based n-gram models of natural language," Comput. Linguist., vol. 18, no. 4, pp. 467-479, 1992.
35. G. Zweig and K. Makarychev, "Speed regularization and optimality in word classing," in Proc. ICASSP, 2013, pp. 8237-8241.
36. S. Martin, J. Liermann, and H. Ney, "Algorithms for bigram and trigram word clustering," Speech Commun., vol. 24, no. 1, pp. 19-37, 1998.
37. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994.
38. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
39. F. Gers, "Learning to forget: Continual prediction with LSTM," in Proc. Int. Conf. Artif. Neural Netw., 1999, pp. 850-855.
40. F. A. Gers, N. N. Schraudolph, and J. Schmidhuber, "Learning precise timing with LSTM recurrent networks," J. Mach. Learn. Res., vol. 3, pp. 115-143, 2002.
41. M. Sundermeyer, R. Schlüter, and H. Ney, "RWTHLM-The RWTH Aachen University neural network language modeling toolkit," in Proc. Interspeech, 2014, pp. 2093-2097.
42. nd Ed. ed. New York, NY, USA: Springer, 2012, ch. 18, pp. 421-436.
43. D. E. Rumelhart, G. E. Hinton, R. J. Williams, J. L. McClelland, and D. E. Rumelhart, "Learning internal representations by error propagation,"in the PDP Research Group, Parallel Distributed Processing. Cambridge, MA, USA: MIT Press, 1986, pp. 318-362.
44. R. J. Williams, D. Zipser, Y. Chauvain, and D. E. Rumelhart, "Gra-dient- based learning algorithms for recurrent networks and their computational complexity," in Backpropagation: Theory, Architectures, and Applications. Hove, U.K.: Psychology Press, 1995, pp. 433-486.
45. T. Mikolov, S. Kombrink, A. Deoras, L. Burget, and J. Černocký, "RNNLM-recurrent neural network language modeling toolkit," in Proc. ASRU, 2011, pp. 196-201.
46. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, "Strategies for training large scale neural network language models," in Proc. ASRU, 2011, pp. 196-201.
47. F. Weng, A. Stolcke, and A. Sankar, "Efficient lattice representation and generation," in Proc. ICSP, 1998.
48. H. Ney and S. Ortmanns, "Progress in dynamic programming search for LVCSR," Proc. IEEE, vol. 88, no. 8, pp. 1224-1240, Aug. 2000.
49. M. Sundermeyer, M. Nuß baum-Thom, S. Wiesler, C. Plahl, A. El-Desoky Mousa, S. Hahn, D. Nolden, R. Schlüter, and H. Ney, "The RWTH 2010 QUAERO ASR evaluation system for English, French, and German," in Proc. ICASSP, 2011, pp. 2212-2215.
50. Z. Tüske, R. Schlüter, and H. Ney, "Multilingual hierarchical MRASTA features for ASR," in Proc. Interspeech, 2013, pp. 2222-2226.
51. H. Hermansky, D. P. Ellis, and S. Sharma, "Tandem connectionist feature extraction for conventional HMM systems," in Proc. ICASSP, 2000, pp. 1635-1638.
52. D. Rybach, S. Hahn, P. Lehnen, D. Nolden, M. Sundermeyer, Z. Tüske, S. Wiesler, R. Schlüter, and H. Ney, "RASR-the RWTH Aachen University open source speech recognition toolkit," in Proc. ASRU, 2011.
53. D. Klakow and J. Peters, "Testing the correlation of word error rate and perplexity," in Speech Commun., 2002, vol. 38, no. 1, pp. 19-28.