Sequence-discriminative training of deep neural networks

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2013, Pages 2345-2349

  Veselý, Karel ^a   Ghoshal, Arnab ^b   Burget, Lukáš ^a   Povey, Daniel ^c  

^a BRNO UNIVERSITY OF TECHNOLOGY   (Czech Republic)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c Johns Hopkins University   (United States)

Author keywords

Deep learning; Neural networks; Reproducible research; Sequence criterion training; Speech recognition

Indexed keywords

NEURAL NETWORKS; STEREOPHONIC BROADCASTING;

CONVERSATIONAL TELEPHONE SPEECH; DEEP LEARNING; DEEP NEURAL NETWORKS; GRADIENT COMPUTATION; MINIMUM BAYES RISK; MINIMUM PHONE ERROR; MUTUAL INFORMATIONS; REPRODUCIBLE RESEARCH;

SPEECH RECOGNITION;

EID: 84906274730     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. H. Bourlard and N. Morgan, Connectionist Speech Recognition- A Hybrid Approach. Kluwer Academic, 1994.
2. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco, "Connectionist probability estimators in HMM speech recognition, " IEEE Trans. Speech Audio Process., vol. 2, no. 1, pp. 161- 174, 1994.
3. A. J. Robinson, "An application of recurrent nets to phone probability estimation, " IEEE Trans. Neural Networks, vol. 5, no. 2, pp. 298-305, 1994.
4. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, " IEEE Trans. Audio Speech Lang. Process., vol. 20, no. 1, pp. 30-42, 2012.
5. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. IEEE ASRU, December 2011, pp. 24-29.
6. N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition, " in Proc. INTERSPEECH, September 2012.
7. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition, " in Proc. IEEE ASRU, December 2011, pp. 30-35.
8. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets, " Neural Computation, vol. 18, pp. 1527- 1554, 2006.
9. L. R. Bahl, P. F. Brown, P. V. de Souza, and R. L. Mercer, "Maximum mutual information estimation of hidden Markov model parameters for speech recognition, " in Proc. IEEE ICASSP, vol. 1, April 1986, pp. 49-52.
10. V. Valtchev, J. Odell, P. Woodland, and S. Young, "MMIE training of large vocabulary recognition systems, " Speech Communication, vol. 22, no. 4, pp. 303-314, September 1997.
11. D. Povey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah, "Boosted MMI for model and feature-space discriminative training, " in Proc. IEEE ICASSP, 2008, pp. 4057- 4060.
12. D. Povey, "Discriminative training for large vocabulary speech recognition, " Ph.D. dissertation, University of Cambridge, Cambridge, UK, 2003.
13. J. Kaiser, B. Horvat, and Z. Kacic, "A novel loss function for the overall risk criterion based discriminative training of HMM models, " in Proc. ICSLP, vol. 2, October 2000, pp. 887-890.
14. M. Gibson and T. Hain, "Hypothesis spaces for minimum Bayes risk training in large vocabulary speech recognition, " in Proc. INTERSPEECH, September 2006, pp. 2406-2409.
15. D. Povey and B. Kingsbury, "Evaluation of proposed modifications to MPE for large scale discriminative training, " in Proc. IEEE ICASSP, vol. 4, April 2007, pp. 321-324.
16. J. S. Bridle and L. Dodd, "An Alphanet approach to optimising input transformations for continuous speech recognition, " in Proc. IEEE ICASSP, vol. 1, April 1991, pp. 277-280.
17. A. Krogh and S. K. Riis, "Hidden neural networks, " Neural Computation, vol. 11, no. 2, pp. 541-563, February 1999.
18. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " in Proc. IEEE ICASSP, April 2009, pp. 3761-3764.
19. G.Wang and K. C. Sim, "Sequential classification criteria for NNs in automatic speech recognition, " in Proc. INTERSPEECH, August 2011, pp. 441-444.
20. B. Kingsbury, T. N. Sainath, and H. Soltau, "Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization, " in Proc. INTERSPEECH, September 2012.
21. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, "The Kaldi speech recognition toolkit, " in IEEE ASRU, December 2011.
22. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors, " Nature, vol. 323, pp. 533-536, October 1986.
23. G. Heigold, T. Deselaers, R. Schluter, and H. Ney, "Modified MMI/MPE: A direct evaluation of the margin in speech recognition, " in Proc. ICML, 2008, pp. 384-391.
24. M. J. F. Gales, "Semi-tied covariance matrices for hidden Markov models, " IEEE Transactions on Speech and Audio Processing, vol. 7, no. 3, pp. 272-281, May 1999.
25. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng, "Large scale distributed deep networks, " in Advances in Neural Information Processing Systems. MIT Press, 2012, pp. 1232-1240.