Context dependent recurrent neural network language model

2012 IEEE Workshop on Spoken Language Technology, SLT 2012 - Proceedings

Volumn , Issue , 2012, Pages 234-239

  Mikolov, Tomas ^a   Zweig, Geoffrey ^b  

^a BRNO UNIVERSITY OF TECHNOLOGY   (Czech Republic)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Language Modeling; Latent Dirichlet Allocation; Recurrent Neural Network; Topic Models

Indexed keywords

CONTEXT DEPENDENT; CONTEXTUAL INFORMATION; DATA FRAGMENTATION; DATA SUBSETS; INPUT VECTOR; LANGUAGE MODELING; LATENT DIRICHLET ALLOCATION; STATE-OF-THE-ART PERFORMANCE; TOPIC MODEL; TREEBANKS; WALL STREET JOURNAL;

RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION; STATISTICS;

COMPUTATIONAL LINGUISTICS;

EID: 84874235486     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SLT.2012.6424228     Document Type: Conference Paper

References (37)

1. Tomas Mikolov, Martin Karafiat, Jan Cernocky, and Sanjeev Khudanpur, "Recurrent neural network based language model," in Interspeech, 2010.
2. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network based language model," in ICASSP, 2011.
3. Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukas Burget, and Jan Cernocky, "Empirical evaluation and combination of advanced language modeling techniques," in Interspeech, 2011.
4. Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget, and Jan Cernocky, "Strategies for Training Large Scale Neural Network Language Models," in ASRU, 2011.
5. Y. Bengio, R. Ducharme, Vincent, P., and C. Jauvin, "A neural probabilistic language model," Journal of Machine Learning Reseach, vol. 3, no. 6, 2003.
6. Holger Schwenk, "Continuous space language models," Computer Speech and Language, vol. 21, no. 3, pp. 492-518, 2007.
7. AndriyMnih and Geoffrey Hinton, "Three new graphical models for statistical language modelling," in Proceedings of the 24th International Conference on Machine Learning, 2007, pp. 641-648.
8. Hai-Son Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language model," in ICASSP, 2011.
9. Le Hai Son, Alexandre Allauzen, and Francois Yvon, "Measuring the influence of long range dependencies with neural network language models," in Proceedings of the Workshop on the Future of Language Modeling for HLT (NAACL/HLT 2012), 2012.
10. Yoshua Bengio, Patrice Simard, and Paolo Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157-166, 1994.
11. R. Kuhn and R. de Mori, "A Cache-based Natural Language Model for Speech Recognition," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 12, no. 6, 1990.
12. J. Bellegarda, "Exploiting latent semantic information in statistical language modeling," Proceedings of the IEEE, vol. 88, no. 8, 2000.
13. N. Coccaro and D. Jurafsky, "Towards better integration of semantic predictors in statistical language modeling," in Proceedings, ICSLP, 1998.
14. R. Kneser and V. Steinbiss, "On the Dynamic Adaptation of Stochastic LM," in ICASSP, 1993.
15. R.M. Iyer and M. Ostendorf, "Modeling long distance dependence in language: topic mixtures versus dynamic cache models," Speech and Audio Processing, IEEE Transactions on, vol. 7, no. 1, pp. 30-39, 1999.
16. Sanjeev Khudanpur and Jun Wu, "Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling," Computer Speech and Language, vol. 14, no. 4, pp. 355-372, 2000.
17. G. Zweig and S. Chang, "Personalizing Model M for Voicesearch," in ICSLP 2011, 2011.
18. S. Chen, "Performance prediction for exponential language models," in NAACL-HLT, 2009.
19. S. Chen, "Shrinking exponential language models," in NAACLHLT, 2009.
20. R. Lau, R. Rosenfeld, and S. Roukos, "Trigger-based Language Models: A Maximum Entropy Approach," in Proc. of ICASSP, 1993.
21. R. Rosenfeld, "A whole sentence maximum entropy language model," in Automatic Speech Recognition and Understanding, 1997. Proceedings., 1997 IEEE Workshop on, 1997.
22. J. Martens and I. Sutskever, "Learning Recurrent Neural Networks with Hessian-Free Optimization," in ICML, 2011.
23. S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," Neural Computation, no. 9, 1997.
24. David M. Blei, Andrew Y. Ng, and Michael I. Jordan, "Latent dirichlet allocation," J. Mach. Learn. Res., vol. 3, pp. 993-1022, 2003.
25. F. Zamora-Martinez, S. Espana-Boquera, M.J. Castro-Bleda, and R. De-Mori, "Cache neural network language models based on long-distance dependencies for a spoken dialog system," in Proceeedings of ICASSP, 2012.
26. E. Arisoy, M. Saraclar, B. Roark, and I. Shafran, "Discriminative language modeling with linguistic and statistically derived features," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 2, pp. 540-550, feb. 2012.
27. S. Chu and L. Mangu, "Improving arabic broadcast transcription using automatic topic clustering," in Proceeedings of ICASSP, 2012.
28. Tomas Mikolov, "Statistical language models based on neural networks," in PhD Thesis, Brno University of Technology, 2012.
29. Ahmad Emami and Frederick Jelinek, "Exact training of a neural syntactic language model," in ICASSP, 2004.
30. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini, "Building a large annotated corpus of english: the penn treebank," Comput. Linguist., vol. 19, no. 2, pp. 313-330, 1993.
31. Peng Xu and Frederick Jelinek, "Random forests and the data sparseness problem in language modeling," Computer Speech and Language, vol. 21, no. 1, pp. 105-152, 2007.
32. Denis Filimonov and Mary Harper, "A joint language model with fine-grain syntactic tags," in EMNLP, 2009.
33. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman, "Indexing by latent semantic analysis," Journal of the American Society for Information Science, vol. 41, no. 96, 1990.
34. D. Povey and A. Ghoshal et al., "The Kaldi Speech Recognition Toolkit," in ASRU, 2011.
35. W. Reichl andW. Chou, "Robust Decision Tree State Tying for Continuous Speech Recognition," in IEEE Trans. Speech and Audio Processing, 2000.
36. D. Povey and L. Burget et al., "The Subspace Gaussian Mixture Model-a Structured Model for Speech Recognition," in Computer Speech and Language, 2011.
37. Puyang Xu, D. Karakos, and S. Khudanpur, "Self-Supervised Discriminative Training of Statistical Language Models," in ASRU, 2009.