Multiple parallel hidden layers and other improvements to recurrent neural network language modeling

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2013, Pages 8426-8429

  Caseiro, Diamantino ^a   Ljolje, Andrej ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

Automatic Speech Recognition; Language Modeling; Multiple Parallel Hidden Layers; Recurrent Neural Network Language Model

Indexed keywords

AUTOMATIC SPEECH RECOGNITION; ERROR RATE REDUCTION; HIDDEN LAYERS; LANGUAGE MODEL; MULTI-PASS; NETWORK LANGUAGE; SINGLE MODELS; STATE OF THE ART;

ALGORITHMS; RECURRENT NEURAL NETWORKS; SIGNAL PROCESSING;

COMPUTATIONAL LINGUISTICS;

EID: 84890445091     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2013.6639309     Document Type: Conference Paper

References (18)

1. J. Goodman, "Classes for fast maximum entropy training transform," in Proceedings of ICASSP, May 2001.
2. Stanley F. Chen, "Shrinking exponential language models," in Proceedings of HLT-NAACL, 2009.
3. Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin, "A neural probabilistic language model," J. Mach. Learn. Res., vol. 3, pp. 1137-1155, Mar. 2003.
4. H.-S. Le, I. Oparin, A.Allauzen, J.-L. Gauvain, and F.Yvon, "Structured output layer neural network language models for speech recognition," IEEE Transactions on Audio, Speech and Language Processing, vol. 21, no. 1, pp. 197-206, 2012.
5. Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, "Recurrent neural network based language model," in Proceedings of INTERSPEECH, 2010, vol. 2010, pp. 1045-1048.
6. Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukas Burget, and Jan Cernocky, "Empirical evaluation and combination of advanced language modeling techniques," in Proceedings of Interspeech, 2011, vol. 2011, pp. 605-608.
7. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network language model," in Proceedings of ICASSP, 2011, pp. 5528-5531.
8. Frederic Morin and Yoshua Bengio, "Hierarchical probabilistic neural network language model," in AISTATS05, 2005, pp. 246-252.
9. H.-S. Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language model," in Proceedings of ICASSP, 2011, pp. 5524-5527.
10. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai, "Class-based n-gram models of natural language," Comput. Linguist., vol. 18, no. 4, pp. 467-479, Dec. 1992.
11. Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget, and Jan Cernocky, "Strategies for training large scale neural network language models," in Proceedings of ASRU, 2011, pp. 196-201.
12. D.J. Miller and Lian Yan, "Critic-driven ensemble classification," Trans. Sig. Proc., vol. 47, no. 10, pp. 2833-2844, Oct. 1999.
13. Wei Xu, "Towards optimal one pass large scale learning with averaged stochastic gradient descent," CoRR, vol. abs/1107.2490, 2011.
14. A. Ljolje, V. Goffin, D. Caseiro, T. Mishra, and M. Gilbert, "Visual voice mail to text on the iphone/ipad," in INTERSPEECH. 2011, pp. 3337-3338, ISCA.
15. Anoop Deoras, Tom Mikolov, and Kenneth Church, "A fast re-scoring strategy to capture long-distance dependencies," in Proceedings of EMNLP, 2011, pp. 1116-1127.
16. Andrei Alexandrescu and Katrin Kirchhoff, "Factored neural language models," in Proceedings of NAACL, Stroudsburg, PA, USA, 2006, NAACL-Short 06, pp. 1-4.
17. Dietrich Klakow, "Log-linear interpolation of language models," in Proceedings of ICSLP, 1998.
18. Gideon Mann, Ryan McDonald, Mehryar Mohri, Nathan Silberman, and Dan Walker, "Efficient large-scale distributed training of conditional maximum entropy models," in Advances in Neural Information Processing Systems 22, pp. 1231-1239. 2009.