Beyond cross-entropy: Towards better frame-level objective functions for deep neural network training in automatic speech recognition

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1214-1218

  Huang, Zhen ^a   Li, Jinyu ^b   Weng, Chao ^a   Lee, Chin Hui ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b MICROSOFT   (United States)

Author keywords

Boosting difficult samples; Cross entropy; Deep neural network; Log posterior ratio

Indexed keywords

CHEMICAL ACTIVATION; CONTINUOUS SPEECH RECOGNITION; NEURAL NETWORKS; SPEECH COMMUNICATION;

AUTOMATIC SPEECH RECOGNITION; CROSS ENTROPY; DEEP NEURAL NETWORKS; LARGE VOCABULARY CONTINUOUS SPEECH RECOGNITION; LOG POSTERIOR RATIO; OBJECTIVE FUNCTIONS; POSTERIOR PROBABILITY; TARGET PREDICTION;

ENTROPY;

EID: 84910061470     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
2. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in Proc. Interspeech, 2011, pp. 437-440.
3. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition, " in Proc. ASRU, 2011, pp. 30-35.
4. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for largevocabulary speech recognition, " IEEE Trans. Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 30- 42, 2012.
5. A. Mohamed, G. E. Dahl, and G. E. Hinton, "Acoustic modeling using deep belief networks, " IEEE Trans. Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 14- 22, 2012.
6. B. Kingsbury, T. N. Sainath, and H. Soltau, "Scalable minimum bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization." in Proc. Interspeech, 2012.
7. K. Vesely, A. Ghoshal, L. Burget, and D. Povey, "Sequence-discriminative training of deep neural networks, " in Proc. Interspeech, 2013, pp. 2345-2349.
8. Z. Yan, Q. Huo, and J. Xu, "A scalable approach to using DNN-derived features in GMM-HMM based acoustic modeling for LVCSR, " in Proc. Interspeech, 2013.
9. R. P. Lippmann, "Review of neural networks for speech recognition, " Neural computation, vol. 1, no. 1, pp. 1-38, 1989.
10. H. A. Bourlard and N. Morgan, Connectionist speech recognition: A hybrid approach. Springer, 1994, vol. 247.
11. M.-Y. Hwang and X. Huang, "Shared-distribution hidden Markov models for speech recognition, " IEEE Trans. Speech and Audio Processing, vol. 1, no. 4, pp. 414-420, 1993.
12. S. Wiesler, J. Li, and J. Xue, "Investigations on Hessianfree optimization for cross-entropy training of deep neural networks, " in Proc. Interspeech, 2013.
13. A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models, " in Proc. ICML, 2013.
14. M. Cai, Y. Shi, and J. Liu, "Deep maxout neural networks for speech recognition, " in Proc. ASRU, 2013, pp. 291- 296.
15. P. Swietojanski, J. Li, and J. T. Huang, "Investigation of maxout networks for speech recognition, " in Proc. ICASSP, 2014.
16. D. Yu, K. Yao, H. Su, G. Li, and F. Seide, "KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition, " in Proc. ICASSP, 2013, pp. 7893-7897.
17. D. E. Rumelhart, G. E. Hinton, and R. J.Williams, Learning representations by back-propagating errors. MIT Press, Cambridge, MA, USA, 1988.
18. T. Zhang, "Solving large scale linear prediction problems using stochastic gradient descent algorithms, " in Proc. ICML, 2004, pp. 919-926.
19. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao, "Optimal distributed online prediction, " in Proc. ICML, 2011, pp. 713-720.
20. D. Erhan, P.-A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent, "The difficulty of training deep architectures and the effect of unsupervised pre-training, " in Proc. AISTATS, 2009, pp. 153-160.
21. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks, " Science, vol. 313, no. 5786, pp. 504-507, 2006.
22. B. H. Juang, W. Chou, and C.-H. Lee, "Minimum classification error rate methods for speech recognition, " IEEE Trans. Speech and Audio Processing, vol. 5, no. 3, pp. 257-265, 1997.
23. J. Li, M. Yuan, and C.-H. Lee, "Soft margin estimation of hidden Markov model parameters." in Proc. Interspeech, 2006.
24. J. Li, M. Yuan, and C.-H. Lee, "Approximate test risk bound minimization through soft margin estimation, " IEEE Trans. Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2393-2404, 2007.
25. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, "The Kaldi speech recognition toolkit, " in Proc. ASRU, 2011.
26. J. J. Godfrey and E. Holliman, "Switchboard-1 release 2, " Linguistic Data Consortium, Philadelphia, 1997.
27. J. Fiscus, W. M. Fisher, A. F. Martin, M. A. Przybocki, and D. S. Pallett, "2000 NIST evaluation of conversational speech recognition over the telephone: English and mandarin performance results, " in Proc. Speech Transcription Workshop, 2000.