Approximate inference: A sampling based modeling technique to capture complex dependencies in a language model

Speech Communication

Volumn 55, Issue 1, 2013, Pages 162-177

  Deoras, Anoop ^a   Mikolov, Tomáš ^b   Kombrink, Stefan ^b   Church, Kenneth ^c  

^a MICROSOFT   (United States)

^b BRNO UNIVERSITY OF TECHNOLOGY   (Czech Republic)

^c IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Decoding; Long span language models; Recurrent neural networks; Speech recognition

Indexed keywords

APPROXIMATE INFERENCE; CONTEXT INFORMATION; LANGUAGE MODEL; LARGE VOCABULARY CONTINUOUS SPEECH RECOGNITION; LONG SPAN; MODELING TECHNIQUE; N-BEST LIST; N-GRAM LANGUAGE MODELS; N-GRAM MODELS; SAMPLING-BASED; SEARCH SPACES; SEMANTIC INFORMATION; SPEECH RECOGNITION SYSTEMS; SUB-OPTIMAL PERFORMANCE; VARIATIONAL APPROXIMATION;

CONTINUOUS SPEECH RECOGNITION; DECODING; RECURRENT NEURAL NETWORKS; SEARCH ENGINES; SEMANTICS; SPEECH RECOGNITION;

COMPUTATIONAL LINGUISTICS;

EID: 84870293590     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2012.08.004     Document Type: Article

References (35)

1. Allauzen, C.; Mohri, M.; Roark, B.; 2003. Generalized Algorithms for constructing statistical language models. In: Proc. of Association of Computational Linguistics (ACL).
2. Bengio, Y.; LeCun, Y.; 2007. Scaling Learning Algorithms Towards AI. MIT Press, Large-Scale Kernel Machines.
3. P.J. Bickel, and K.A. Doksum Mathematical Statistics: Basic Ideas and Selected Topics 1977 Holden-Day Inc. Oakland, CA
4. Bishop, C.M.; 2006. Pattern Recognition and Machine Learning. Springer-Verlag New York, Inc.; Secaucus, NJ, USA.
5. Boden, M.; 2002. A Guide to Recurrent Neural Networks and Back Propagation. Dallas Project.
6. C. Chelba, and F. Jelinek Structured language modeling Comput. Speech Language 14 2000 283 332
7. Chen, S.F.; Mangu, L.; Ramabhadran, B.; Sarikaya, R.; Sethy, A.; 2009. Scaling shrinkage-based language models. In: Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 299-304.
8. T. Cover, and J. Thomas Elements of Information Theory 1991 John Wiley and Sons Inc. NY
9. Deng, L.; 2011. An overview of deep-structured learning for information processing. In: Proc. Asian-Pacific Signal and Information Processing - Annual Summit and Conference (APSIPA-ASC).
10. Deoras, A.; 2011. Search and decoding strategies for complex lexical modeling in LVCSR. PhD thesis, The Johns Hopkins University.
11. Deoras, A.; Jelinek, F.; 2009. Iterative decoding: A novel re-scoring framework for confusion networks. In: Proc. IEEE Workshop on Automatic Speech Recognition Understanding (ASRU), pp. 282-286.
12. Deoras, A.; Filimonov, D.; Harper, M.; Jelinek, F.; 2010. Model combination for speech recognition using empirical bayes risk minimization. In: Proc. IEEE-Spoken Language Technologies.
13. Deoras, A.; Mikolov, T.; Church, K.; 2011a. A fast re-scoring technique to capture long distance dependencies. In: Proc. 2011 Conf. on Empirical Methods in Natural Language Processing (EMNLP).
14. Deoras, A.; Mikolov, T.; Kombrink, S.; Karafiát, M.; Khudanpur, S.; 2011b. Variational Approximation of Long-Span Language Models for LVCSR, in: Proc. IEEE Internat. Conf. on Acoustics, Speech, and Signal Processing (ICASSP).
15. J. Elman Finding structure in time Cognit. Sci. 14 2 1990 179 211
16. Filimonov, D.; Harper, M.; 2009. A joint language model with fine-grain syntactic tags. In: Proc. 2009 Conf. on Empirical Methods in Natural Language Processing (EMNLP).
17. S. Geman, and D. Geman Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images IEEE Trans. Pattern Anal. Machine Intell. 6 1984 721 741 (Pubitemid 15453722)
18. Glass, J.; Hazen, T.; Cyphers, S.; Malioutov, I.; Huynh, D.; Barzilay, R.; 2007. Recent progress in MIT spoken lecture processing project. In: Proc. ICSLP-Interspeech.
19. Hain, T.; 2005. The 2005 AMI system for the transcription of speech in meetings. In: Proc. Rich Transcription 2005 Spring Meeting Recognition Evaluation Workshop, UK.
20. Jordon, M.I.; 1998. Learning in Graphical Models. The MIT Press, Cambridge, MA.
21. D. Lin, K. Church, H. Ji, S. Sekine, D. Yarowsky, S. Bergsma, K. Patil, E. Pitler, R. Lathbury, V. Rao, K. Dalwani, and S. Narsale New tools for web-scale n-grams N.C.C. Chair, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, D. Tapias, Proc. Seventh Conf. on International Language Resources and Evaluation (LREC'10) 2010 European Language Resources Association (ELRA) Valletta, Malta
22. Mikolov, T.; Deoras, A.; Kombrink, S.; Burget, L.; Černocký , J.H.; 2011a. Empirical Evaluation and Combination of Advanced Language Modeling Techniques, in: Proc. of ICSLP Interspeech.
23. Mikolov, T.; Deoras, A.; Povey, D.; Burget, L.; Černocký, J.H.; 2011b. Strategies for training large scale neural network language models. In: Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).
24. Mikolov, T.; Kombrink, S.; Burget, L.; Černocký, J.H.; Khudanpur, S.; 2011c. Extensions of recurrent neural network language model. In: Proc. IEEE Internat. Conf. on Acoustics, Speech, and Signal Processing (ICASSP).
25. Mikolov, T.; Karafiát, M.; Burget, L.; Černocký, J.H.; Khudanpur, S.; 2010. Recurrent neural network based language model. In: Proc. ICSLP-Interspeech.
26. Momtazi, S.; Faubel, F.; Klakow, D.; 2010. Within and across sentence boundary language model. In: Proc. ICSLP-Interspeech.
27. Nederhof, M.-J.; 2005. A general technique to train language models on language models. In: Proc. Association of Computational Linguistics (ACL).
28. Nederhof, M.-J.; Satta, G.; 2004. Kullback-Leibler distance between probabilistic context-free grammars and probabilistic finite automata. In: Proc. Association of Computational Linguistics (ACL).
29. B. Roark Probabilistic top-down parsing and language modeling Comput. Linguist. 27 2001 249 276 (Pubitemid 33597448)
30. Rumelhart, D.E.; Hinton, G.E.; Williams, R.J.; 1986. Learning internal representations by error propagation. In: Computational Models of Cognition And Perception Series, MIT Press, pp. 318-362.
31. C. Shannon Prediction and entropy of printed English Bell Syst. Tech. J. 30 1951 50 64
32. Soltau, H.; Saon, G.; Kingsbury, B.; 2010. The IBM Attila speech recognition toolkit. In: Proc. IEEE Workshop on Spoken Language Technology.
33. Stolcke, A.; 1998. Entropy-based pruning of backoff language models. In: Proc. DARPA Broadcast News Transcription and Understanding Workshop, pp. 8-11.
34. Stolcke, A.; Segal, J.;1994. Precise n-gram probabilities from stochastic context-free grammars. In:Proc. Association of Computational Linguistics (ACL).
35. Xu, P.; 2005. Random forests and the data sparseness problem in language modeling. PhD thesis, Johns Hopkins University.