Variance regularization of RNNLM for speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2014, Pages 4893-4897

  Shi, Yongzhe ^a   Zhang, Wei Qiang ^a   Cai, Meng ^a   Liu, Jia ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

recurrent neural network language model; speech recognition; Variance regularization

Indexed keywords

COMPUTATIONAL COMPLEXITY; COMPUTATIONAL LINGUISTICS; REAL TIME SYSTEMS; RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION;

CLASS-BASED; LANGUAGE MODEL; OUTPUT LAYER; REGULARIZATION ALGORITHMS; TRAINING PHASE; VARIANCE REGULARIZATION; WORD ERROR RATE; WORD PREDICTION;

SIGNAL PROCESSING;

EID: 84905252011     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2014.6854532     Document Type: Conference Paper

References (17)

1. Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukas Burget, and Jan Honza Cemocky, "Empirical evaluation and combination of advanced language modeling techniques, " in Proc. ofInterSpeech, 2011.
2. Tomas Mikolov, Statistical Language Models Based on Neural Netowrks, Ph.D. Thesis, Brno University of Technology (BUT), 2012, [Online] http://www.fit.vutbr.cz/imikolov/rnnlm/thesis.pdf.
3. Holger Schwenk and Jean-Luc Gauvain, "Connectionist language modeling for large vocabulary continuous speech recognition, " in Proc. ofICASSP, 2002, pp. 765768.
4. Frederic Morin and Yoshua Bengio, "Hierarchical probabilistic neural network language model, " in Proc. of AISTATS, 2005, pp.246-252.
5. Andriy Mnih and Geoffrey Hinton, "A scalable hierarchical distributed language model, " Advances in Neural Information Processing Systems, vol. 21, 2008.
6. Hai Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and Francois Yvon, "Structured output layer neural network language model, " in Proc. of ICASSP, 2011, pp.5524-5527.
7. Yongzhe Shi, Wei-Qiang Zhang, Jia Liu, and Michael T. Johnson, "RNN language model with word clustering and class-based output layer, " EURASIP Journal on Audio, Speech, and Music Processing, vol. 22, 2013.
8. F. Zamora-Martinez, M. J. Castro-BIeda, and S. EspanaBoquera, "Fast evaluation of connectionist language models, " in Proc. ofIWANN '09, 2009, pp. 33-40.
9. Andriy Mnih and Yee Whye Teh, "A fast and simple algorithm for training neural probabilistic language models'" in Proc. ofICML, 2012.
10. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets, " in Proc. ofICASSP, 2013.
11. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Honza Cemocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network language model, " in Proc. ofICASSP, 2011.
12. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai, "Classbased n-gram models for natural language, " Comput. Linguist., vol. 18, no. 4, pp. 467-479, 1992. 4897
13. Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukas Burget, and Jan Honza Cemocky, "RNNLM-recurrent neural network language modeling toolkit, " in Proc. of ASRU, 2011, [Available] http://www.fit.vutbr.cz/imikolov/mnlm/.
14. J. Godfrey and E. Holliman, "Switchboard-1 release 2, " Linguistic Data Consortium, Philadelphia, 1997.
15. Meng Cai, Yongzhe Shi, and Jia Liu, "Deep maxout neural networks for speech recognition, " in Proc. of ASRU, 2013.
16. Frank Seide, Gang Li, Xie Chen, and Dong Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. ofASRU, 2011.
17. "The CMU Pronouncing Dictionary Release 0.7a, " 2007, [Available] http://www.speech.cs.cmu.edu/cgibinicmudict