Sequence discriminative distributed training of long short-term memory recurrent neural networks

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1209-1213

  Sak, Haşim ^a   Vinyals, Oriol ^a   Heigold, Georg ^a   Senior, Andrew ^a   McDermott, Erik ^a   Monga, Rajat ^a   Mao, Mark ^a  

^a GOOGLE INC   (United States)

Author keywords

Acoustic modeling; Long short term memory; Recurrent neural network; Sequence discriminative training

Indexed keywords

BRAIN; SPEECH COMMUNICATION; STEREOPHONIC BROADCASTING; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

ACOUSTIC MODEL; DEEP NEURAL NETWORKS; DISCRIMINATIVE TRAINING; LONG SHORT-TERM MEMORY; MAXIMUM MUTUAL INFORMATION; OBJECTIVE FUNCTIONS; RECURRENT NEURAL NETWORK (RNNS); STOCHASTIC GRADIENT DESCENT;

RECURRENT NEURAL NETWORKS;

EID: 84910072094     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. G. Dahl, D. Yu, and L. Deng, "Context-dependent pre-trained deep neural networks for large vocabulary speech recognition, " in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2011.
2. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in INTERSPEECH, 2011, pp. 437-440.
3. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, " IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 1, pp. 30-42, Jan. 2012. [Online]. Available: Http://dx.doi.org/10.1109/TASL.2011.2134090.
4. G. Dahl, D. Yu, and L. Deng, "Context-dependent pre-trained deep neural networks for large vocabulary speech recognition, " in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2011.
5. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in INTERSPEECH, 2011, pp. 437-440.
6. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, " IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 1, pp. 30-42, Jan. 2012. [Online]. Availablehttp://dx.doi.org/10.1109/TASL.2011.2134090.
7. A. Rahman Mohamed, G. E. Dahl, and G. E. Hinton, "Acoustic modeling using deep belief networks, " IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 1, pp. 14-22, 2012.
8. N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition, " in INTERSPEECH, 2012.
9. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, 2012.
10. A. Graves, N. Jaitly, and A. Mohamed, "Hybrid speech recognition with deep bidirectional LSTM, " in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 273-278.
11. H. Sak, A. Senior, and F. Beaufays, "Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition, " ArXiv e-prints, Feb. 2014.
12. S. Hochreiter and J. Schmidhuber, "Long short-term memory, " Neural Computation, vol. 9, no. 8, pp. 1735-1780, Nov. 1997.
13. F. A. Gers, J. Schmidhuber, and F. Cummins, "Learning to forget: Continual prediction with LSTM, " Neural Computation, vol. 12, no. 10, pp. 2451-2471, 2000.
14. F. A. Gers, N. N. Schraudolph, and J. Schmidhuber, "Learning precise timing with LSTM recurrent networks, " Journal of Machine Learning Research, vol. 3, pp. 115-143, Mar. 2003.
15. H. Bourlard and N. Morgan, Connectionist speech recognition. Kluwer Academic Publishers, 1994.
16. Y. Normandin, "Hidden Markov models, maximum mutual information, and the speech recognition problem, " Ph.D. dissertation, McGill University, Montreal, Canada, 1991.
17. D. Povey, "Discriminative training for large vocabulary speech recognition, " Ph.D. dissertation, Cambridge, England, 2004.
18. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, Taiwan, Apr. 2009, pp. 3761-3764.
19. B. Kingsbury, T. N. Sainath, and H. Soltau, "Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization, " in INTERSPEECH, 2012.
20. H. Su, G. Li, D. Yu, and F. Seide, "Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription, " in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2013, pp. 6664-6668.
21. K. Veselý, A. Ghoshal, L. Burget, and D. Povey, "Sequencediscriminative training of deep neural networks, " in INTERSPEECH, 2013.
22. L. Bottou, "Stochastic gradient learning in neural networks, " in Neuro-Nîmes, 1991.
23. G. Heigold, E. McDermott, V. Vanhoucke, A. Senior, and M. Bacchiani, "Asynchronous stochastic optimization for sequence training of deep neural networks, " in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2014.
24. R. J.Williams and J. Peng, "An efficient gradient-based algorithm for online training of recurrent network trajectories, " Neural Computation, vol. 2, pp. 490-501, 1990.
25. G. Heigold, "A log-linear discriminative modeling framework for speech recognition, " Ph.D. dissertation, RWTH Aachen University, Aachen, Germany, Jun. 2010.
26. H. Sak, A. Senior, and F. Beaufays, "Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling, " in INTERSPEECH 2014, 2014.