Training deep bidirectional LSTM acoustic model for LVCSR by a context-sensitive-chunk BPTT approach

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2015-January, Issue , 2015, Pages 3600-3604

  Chen, Kai ^a,b   Yan, Zhi Jie ^b   Huo, Qiang ^b  

^a UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

BPTT; Context sensitive chunk; DBLSTM; DNN; Long short term memory; LVCSR

Indexed keywords

BRAIN; CONTINUOUS SPEECH RECOGNITION; SPEECH COMMUNICATION;

BPTT; CONTEXT SENSITIVE; DBLSTM; LONG SHORT TERM MEMORY; LVCSR;

SPEECH RECOGNITION;

EID: 84959076031     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. A. J. Robinson, "An application of recurrent nets to phone proba-bility estimation, " IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 298-305, 1994.
2. T. Robinson, M. Hochberg, and S. Renals, "The use of recurrentneural networks in continuous speech recognition, " in C.-H. Lee, F. Soong, and K. Paliwal (Eds.), Automatic Speech and SpeakerRecognition, pp. 233-258, 1996.
3. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural net-works, " IEEE Transactions on Signal Processing, vol. 9, no. 11, pp. 2673-2681, 1997.
4. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recogni-tion: The shared views of four research groups, " IEEE Signal Pro-cessing Magazine, vol. 29, no. 16, pp. 82-97, 2012.
5. D. Yu and L. Deng, Automatic Speech Recognition: A Deep Learn-ing Approach, Springer, 2015.
6. S. Hochreiter, and J. Schmidhuber, "Long short-term memory, "Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
7. F. A. Gers, J. Schmidhuber, and F. Cummins, "Learning to forget: continual prediction with LSTM, " Neural Computation, vol. 12, no. 10, pp. 2451-2471, 2000.
8. F. A. Gers, N. N. Schraudolph, and J. Schmidhuber, "Learning pre-cise timing with LSTM recurrent networks, " Journal of MachineLearning Research, vol. 3, pp. 115-143, 2003.
9. A. Graves, and J. Schmidhuber, "Framewise phoneme classifica-tion with bidirectional LSTM and other neural network architec-tures, " Neural Networks vol. 18, no. 5, pp. 602-610, 2005.
10. A. Graves, S. Fernánd ez and J. Schmidhuber, "BidirectionalLSTM networks for improved phoneme classification and recog-nition, " Proc. ICANN-2005, Springer LNCS 3697, pp. 799-804.
11. A. Graves, M. Liwicki, S. Fernánd ez, R. Bertolami, H. Bunke, and J. Schmidhuber, "A novel connectionist system for unconstrainedhand writing recognition, " IEEE Transactions on PAMI, vol. 31, no. 5, pp. 855-868, 2009.
12. A. Graves, S. Fernánd ez, F. Gomez, and J. Schmidhuber "Con-nectionist temporal classification: labelling unsegmented sequencedata with recurrent neural networks, " Proc. ICML-2006, pp. 369-376.
13. A. Graves, A. R. Mohamed, and G. Hinton, "Speech recogni-tion with deep recurrent neural networks, " Proc. ICASSP-2013, pp. 6645-6649.
14. A. Graves, and N. Jaitly, "Towards end-to-end speech recognitionwith recurrent neural networks, " Proc. ICML-2014, pp. 1764-1772.
15. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach, Kluwer, Norwell, MA, 1993.
16. A. Graves, N. Jaitly, and A. R. Mohamed, "Hybrid speech recog-nition with deep bidirectional LSTM, " Proc. ASRU-2013, pp. 273-278.
17. H. Sak, A. Senior, and F. Beaufays, "Long short-term memory re-current neural network architectures for large scale acoustic mod-eling, " Proc. INTERSPEECH-2014, pp. 338-342.
18. H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao, "Sequence discriminative distributed train-ing of long short-term memory recurrent neural networks, " Proc. INTERSPEECH-2014, pp. 1209-1213.
19. R. J. Williams, and J. Peng, "An efficient gradient-based algo-rithm for on-line training of recurrent network trajectories, " NeuralComputation, vol. 2, no. 4, pp. 490-501, 1990.
20. F. Eyben, J. Bergmann, and F. Weninger, CURRENNT: CUDA-enabled Machine Learning Library For Recurrent Neural Net-works, http: //sourceforge. net/projects/currennt/.
21. K. Chen, Z.-J. Yan, and Q. Huo, "A context-sensitive-chunkBPTT approach to training deep LSTM/BLSTM recurrent neuralnetworks for offline hand writing recognition, " Proc. ICDAR-2015.
22. P. Doetsch, M. Kozielski, and H. Ney, "Fast and robust trainingof recurrent neural networks for offline hand writing recognition, "Proc. ICFHR-2014, pp. 279-284.
23. G. Saon, H. Soltau, A. Emami, and M. Picheny, "Un-folded recurrent neural network for speech recognition, " Proc. INTERSPEECH-2014, pp. 343-347.
24. A. Senior, G. Heigold, M. A. Ranzato, and K. Yang, "An em-pirical study of learning rates in deep neural networks for speechrecognition, " Proc. ICASSP-2013, pp. 6724-6728.
25. S. Wiesler, A. Richard, R. Schluter, and H. Ney, "Mean-normalized stochastic gradient for large-scale deep learning, " Proc. ICASSP-2014, pp. 180-184.
26. J. J. Godfrey, Edward. C. Holliman, and J. McDaniel, "Switch-board: telephone speech corpus for research and development, "Proc. ICASSP-1992, pp. I-517-520.
27. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors, " http: //arxiv. org/abs/1207. 0580, 2012.
28. F. Seide, G. Li, and D. Yu, "Conversational speech tran-scription using context-dependent deep neural networks, " Proc. INTERSPEECH-2011, pp. 437-440.
29. Z.-J. Yan, Q. Huo, and J. Xu, "A scalable approach to usingDNN-derived features in GMM-HMM based acoustic modeling forLVCSR, " Proc. INTERSPEECH-2013, pp. 104-108.
30. Q. Huo and C. Chan, "A study on the use of bi-directional contex-tual dependence in Markov rand om field-based acoustic modellingfor speech recognition, " Computer Speech and Language, vol. 10, pp. 95-105, 1996.