Converting neural network language models into back-off language models for efficient decoding in automatic speech recognition

IEEE Transactions on Audio, Speech and Language Processing

Volumn 22, Issue 1, 2014, Pages 184-192

  Arisoy, Ebru ^a   Chen, Stanley F ^a   Ramabhadran, Bhuvana ^a   Sethy, Abhinav ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Back off language models; Language modeling; Neural network language models

Indexed keywords

CONTINUOUS SPEECH RECOGNITION; DECODING; NATURAL LANGUAGE PROCESSING SYSTEMS; NEURAL NETWORKS; MODELING LANGUAGES; SPEECH PROCESSING; SPEECH RECOGNITION;

ACCURACY IMPROVEMENT; APPROXIMATE METHODS; AUTOMATIC SPEECH RECOGNITION; HIGHER-ORDER MODELS; LANGUAGE MODEL; LARGE VOCABULARY CONTINUOUS SPEECH RECOGNITION; N-GRAM LANGUAGE MODELS; NETWORK LANGUAGE;

COMPUTATIONAL LINGUISTICS;

EID: 84897930752     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2013.2286919     Document Type: Article

References (31)

1. E. Arisoy, S. F. Chen, B. Ramabhadran, and A. Sethy, "Converting neural network language models into back-off language models for efficient decoding in automatic speech recognition," in Proc. ICASSP, 2013, pp. 8242-8246.
2. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, "A neural probabilistic language model," J. Mach. Learn. Res., vol. 3, pp. 1137-1155, 2003.
3. H. Schwenk and J.-L. Gauvain, "Training neural network language models on very large corpora," in Proc. HLT-EMNLP, 2005, pp. 201-208.
4. H. Schwenk, "Continuous space language models," Comput. Speech Lang., vol. 21, no. 3, pp. 492-518, Jul. 2007.
5. H.-K. J. Kuo, E. Arisoy, A. Emami, and P. Vozila, "Large scale hierarchical neural network language models," in Proc. Interspeech, Portland, OR, USA, 2012.
6. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model," in Proc. Interspeech, 2010, pp. 1045-1048.
7. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model," in Proc. ICASSP, 2011, pp. 5528-5531.
8. H. Schwenk and J.-L. Gauvain, "Connectionist language modeling for large vocabulary continuous speech recognition," in Proc. ICASSP, Orlando, FL, USA, 2002, pp. 765-768.
9. A. Stolcke, "Entropy-based pruning of backoff language models," in Proc. DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, VA, USA, 1998, pp. 270-274.
10. M. Siu and M. Ostendorf, "Variable n-grams and extensions for conversational speech language modeling," IEEE Trans. Acoust., Speech, Signal Process., vol. 8, no. 1, pp. 63-75, Jan. 2000. (Pubitemid 30540740)
11. V. Siivola and B. Pellom, "Growing an n-gram model," in Proc. Interspeech, 2005, pp. 1309-1312.
12. V. Siivola, T. Hirsimaki, and S. Virpioja, "On growing and pruning Kneser-Ney smoothed n-gram models," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 5, pp. 1617-1624, Jul. 2007.
13. S. Virpioja and M. Kurimo, "Compact n-gram models by incremental growing and clustering of histories," in Proc. Interspeech - ICSLP, Pittsburgh, PA, USA, 2006, pp. 1037-1040.
14. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling," Comput. Speech Lang., vol. 13, no. 4, pp. 359-394, 1999. (Pubitemid 30518216)
15. P. F. Brown, V. J. Della Pietra, P. V. deSouza, J. C. Lai, and R. L. Mercer, "Class-based n-gram models of natural language," Comput. Linguist., vol. 18, no. 4, pp. 467-479, 1992.
16. E. Arisoy, T. N. Sainath, B. Kingsbury, and B. Ramabhadran, "Deep neural network language models," in Proc. NAACL-HLT Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Lang. Model. for HLT, Montreal, QC, Canada, Jun. 2012, pp. 20-28.
17. J. Goodman, "Classes for fast maximum entropy training," in Proc. ICASSP, 2001, pp. 561-564.
18. A. Emami, "A neural syntactic language model," Ph.D. dissertation, Johns Hopkins Univ., Baltimore, MD, USA, 2006.
19. G. Zweig and K. Makarychev, "Speed regularization and optimality in word classing," in Proc. ICASSP, Vancouver, BC, Canada, 2013, pp. 8237-8241.
20. F. Morin and Y. Bengio, "Hierarchical probabilistic neural network language model," in Proc. AISTATS, 2005, pp. 246-252.
21. A. Mnih and G. Hinton, "A scalable hierarchical distributed language model," in Proc. NIPS, 2008, pp. 1081-1088.
22. H.-S. Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language models for speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 1, pp. 197-206, Jan. 2013.
23. Y. Bengio and J.-S. Senecal, "Quick training of probabilistic neural nets by importance sampling," in Proc. AISTATS, 2003.
24. A. Mnih and Y. W. Teh, "A fast and simple algorithm for training neural probabilistic language models," in Proc. ICML, Edinburgh, U.K., 2012.
25. H.-K. J. Kuo, L. Mangu, A. Emami, I. Zitouni, and Y.-S. Lee, "Syntactic features for arabic speech recognition," in Proc. ASRU, Merano, Italy, 2009, pp. 327-332.
26. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, "Strategies for training large scale neural network language models," in Proc. ASRU, 2011, pp. 196-201.
27. A. Deoras, T. Mikolov, S. Kombrink, M. Karafiat, and S. Khudanpur, "Variational approximation of long-span language models for LVCSR," in Proc. ICASSP, Prague, Czech Republic, 2012, pp. 5532-5535.
28. A. Deoras, T. Mikolov, S. Kombrink, and K. Church, "Approximate inference: A sampling based modeling technique to capture complex dependencies in a language model," Speech Commun., vol. 55, no. 1, pp. 162-177, Jan. 2013.
29. G. Lecorvé and P. Motlicek, "Conversion of recurrent neural network language models to weighted finite state transducers for automatic speech recognition," in Proc. Interspeech, Portland, OR, USA, 2012.
30. W. Wang, A. Stolcke, and M. P. Harper, "The use of a linguistically motivated language model in conversational speech recognition," in Proc. ICASSP, 2004, pp. 261-264.
31. S. F. Chen, B. Kingsbury, L. Mangu, D. Povey, G. Saon, H. Soltau, and G. Zweig, "Advances in speech transcription at IBM under the DARPA EARS program," IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 5, pp. 1596-1608, Sep. 2006.