Deep neural network language models

WLM 2012 - Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012

Volumn , Issue , 2012, Pages 20-28

  Arisoy, Ebru ^a   Sainath, Tara N ^a   Kingsbury, Brian ^a   Ramabhadran, Bhuvana ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACOUSTIC MODELING; COMPUTATIONAL LINGUISTICS; MODELING LANGUAGES; MULTILAYER NEURAL NETWORKS; SPEECH RECOGNITION;

ACOUSTICS MODEL; HIDDEN LAYERS; INPUT FEATURES; LANGUAGE MODEL; N-GRAM LANGUAGE MODELS; NETWORK LANGUAGE; NEURAL-NETWORKS; SINGLE-HIDDEN-LAYER NEURAL NETWORKS; WALL STREET JOURNAL; WORD ERROR RATE;

DEEP NEURAL NETWORKS;

EID: 85075436378     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155.
2. Yoshua Bengio. 2007. Learning Deep Architectures for AI. Technical report, Universit e de Montreal.
3. S. F. Chen and J. Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech and Language, 13(4).
4. Stanley F. Chen, Lidia Mangu, Bhuvana Ramabhadran, Ruhi Sarikaya, and Abhinav Sethy. 2009. Scaling shrinkage-based language models. In Proc. ASRU 2009, pages 299-304, Merano, Italy, December.
5. Stanley F. Chen. 2008. Performance prediction for exponential language models. Technical Report RC 24671, IBM Research Division.
6. George E. Dahl, Marc'Aurelio Ranzato, Abdel rahman Mohamed, and Geoffrey E. Hinton. 2010. Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine. In Proc. NIPS.
7. Ahmad Emami. 2006. A neural syntactic language model. Ph.D. thesis, Johns Hopkins University, Baltimore, MD, USA.
8. Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18:1527-1554.
9. H-K. J. Kuo, L. Mangu, A. Emami, I. Zitouni, and YS. Lee. 2009. Syntactic features for Arabic speech recognition. In Proc. ASRU 2009, pages 327-332, Merano, Italy.
10. Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proc. INTERSPEECH 2010, pages 1045-1048.
11. Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget, and Jan Cernocky. 2011a. Strategies for training large scale neural network language models. In Proc. ASRU 2011, pages 196-201.
12. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2011b. Extensions of recurrent neural network language model. In Proc. ICASSP 2011, pages 5528-5531.
13. Andriy Mnih and Geoffrey Hinton. 2008. A scalable hierarchical distributed language model. In Proc. NIPS.
14. Abdel-rahman Mohamed, George E. Dahl, and Geoffrey Hinton. 2009. Deep belief networks for phone recognition. In Proc. NIPS Workshop on Deep Learning for Speech Recognition and Related Applications.
15. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proc. AISTATS05, pages 246-252.
16. Douglas B. Paul and Janet M. Baker. 1992. The design for the wall street journal-based csr corpus. In Proc. DARPA Speech and Natural Language Workshop, page 357362.
17. Tara N. Sainath, Brian Kingsbury, and Bhuvana Ramabhadran. 2012. Improvements in Using Deep Belief Networks for Large Vocabulary Continuous Speech Recognition. Technical report, IBM, Speech and Language Algorithms Group.
18. Ruhi Sarikaya, Mohamed Afify, and Brian Kingsbury. 2009. Tied-mixture language modeling in continuous space. In HLT-NAACL, pages 459-467.
19. Holger Schwenk and Jean-Luc Gauvain. 2005. Training neural network languagemodels on very large corpora. In Proc. HLT-EMNLP 2005, pages 201-208.
20. Holger Schwenk. 2007. Continuous space language models. Comput. Speech Lang., 21(3):492-518, July.
21. Frank Seide, Gang Li, Xie Chen, and Dong Yu. 2011. Feature Engineering in Context-DependentDeep Neural Networks for Conversational Speech Transcription. In Proc. ASRU.
22. Hagen Soltau, George. Saon, and Brian Kingsbury. 2010. The IBM Attila speech recognition toolkit. In Proc. IEEE Workshop on Spoken Language Technology, pages 97-102.
23. Hai Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and Francois Yvon. 2011. Structured output layer neural network language model. In Proceedings of IEEE International Conference on Acoustic, Speech and Signal Processing, pages 5524-5527, Prague, Czech Republic.
24. Andreas Stolcke. 1998. Entropy-based pruning of backoff language models. In Proceedings of DARPA Broadcast News Transcription and Understanding Workshop, pages 270-274, Lansdowne, VA, USA.