KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2013, Pages 7893-7897

  Yu, Dong ^a   Yao, Kaisheng ^b   Su, Hang ^c,d   Li, Gang ^c   Seide, Frank ^c  

^a MICROSOFT RESEARCH   (United States)

^b MICROSOFT   (United States)

^c MICROSOFT RESEARCH ASIA   (China)

^d TSINGHUA UNIVERSITY   (China)

Author keywords

CD DNN HMM; deep neural network; Kullback Leibler divergence regularization; speaker adaptation

Indexed keywords

ADAPTATION TECHNIQUES; CD-DNN-HMM; DEEP NEURAL NETWORKS; KULLBACK LEIBLER DIVERGENCE; LARGE VOCABULARY SPEECH RECOGNITION; SPEAKER ADAPTATION; SPEECH TRANSCRIPTIONS; UNSUPERVISED ADAPTATION;

HIDDEN MARKOV MODELS; SIGNAL PROCESSING; SPEECH RECOGNITION; SPEECH TRANSMISSION;

NEURAL NETWORKS;

EID: 84890542079     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2013.6639201     Document Type: Conference Paper

References (30)

1. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for large vocabulary speech recognition," IEEE Trans. on Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 30-42, 2012.
2. D. Yu, L. Deng, and G. Dahl, "Roles of pretraining and finetuning in context-dependent DNN-HMMs for real-world speech recognition," in Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Dec. 2010.
3. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks," in Proc. Interspeech'11, pp. 437-440, 2011.
4. D. Yu, Y.-C. Ju, Y.-Y. Wang, G. Zweig, and A. Acero, "Automated Directory Assistance System-from Theory to Practice", in Proc. Interspeech'07, pp. 2709-2712, 2007.
5. A. Mohamed, G. Dahl, and G. Hinton, "Acoustic modeling using deep belief networks," IEEE Trans. on Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 14-22, 2012.
6. N. Jaitly, P. Nguyen, and V. Vanhoucke, "application of pretrained deep neural networks to large vocabulary speech recognition", in Proc. Interspeech'12, 2012.
7. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A.-r. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition", in Proc. ASRU'11, pp. 30-35, 2011.
8. B. Kingsbury, T. N. Sainath, and H. Soltau, "Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization," in Proc. Interspeech'12, 2012.
9. Hang Su, Gang Li, Dong Yu, Frank Seide, "Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription", in Proc. ICASSP 2013.
10. Michael Seltzer, Dong Yu, Yongqiang Wang, "An investigation of deep neural networks for noise robust speech recognition", in Proc. ICASSP 2013.
11. G. Hinton, L. Deng, D. Yu, G. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition," IEEE Signal Processing Magazine, 2012.
12. V. Abrash, H. Franco, A. Sankar, and M. Cohen, "Connectionist speaker normalization and adaptation," in Proc. EUROSPEECH'95, pp. 2183-2186, 1995.
13. J. Neto, L. Almeida, M. Hochberg, C. Martins, L. Nunes, and S. Renals, T. Robinson, "Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system," in Proc. EUROSPEECH'95, pp. 2171-2174, 1995.
14. D. Albesano, R. Gemello, and F. Mana, "Hybrid HMM-NN modeling of stationary-transitional units for continuous speech recognition", in Proc. NIPS'97, pp. 1112-1115, 1997.
15. B. Li and K. C. Sim, "Comparison of discriminative input and output transformations for speaker adaptation in the Hybrid NN/HMM systems", in Proc. Interspeech'10, pp. 526-529, 2010.
16. R. Gemello, F. Mana, S. Scanzio, P. Laface, and R. De Mori, "Linear hidden transformations for adaptation of hybrid ANN/HMM models", Speech Communication 49, no. 10, pp. 827-83, 2007.
17. X. Liu, M. J. F. Gales, and P. C. Woodland. "Improving LVCSR system combination using neural network language model cross adaptation," in Proc. Interspeech'11, Pp. 2857-2860, 2011.
18. Y. Xiao, Z. Zhang, S. Cai, J. Pan, and Y. Yan, "A initial attempt on task-specific adaptation for deep neural networkbased large vocabulary continuous speech recognition", in Proc. Interspeech'12, 2012.
19. J. Trmal, J. Zelinka, and L. Müller. "Adaptation of a feedforward artificial neural network using a linear transform," Text, Speech and Dialogue. Springer Berlin/Heidelberg, pp. 423-430, 2010.
20. K. Yao, D. Yu, F. Seide, H. Su, L.i Deng, and Y. Gong, "Adaptation of context-dependent deep neural networks for automatic speech recognition", in Proc. SLT'12, 2012.
21. J. Stadermann and G. Rigoll, "Two-stage speaker adaptation of hybrid tied-posterior acoustic models," in Proc. ICASSP'05, vol. I, pp. 997-1000, 2005.
22. D. Albesano, R. Gemello, P. Laface, F. Mana, and S. Scanzio, "Adaptation of artificial neural networks avoiding catastrophic forgetting," in Proc. Int. Jnt. Conference on Neural Networks 2006, pp. 2863-2870, 2006.
23. X. Li and J. Bilmes, "Regularized adaptation of discriminative classifiers," in Proc. ICASSP'06, 2006.
24. S. Dupont and L. Cheboub, "Fast speaker adaptation of artificial neural networks for automatic speech recognition", in Proc. ICASSP'00, vol.3, pp. 1795-1798, 2000.
25. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription," in Proc. ASRU'11, 2011.
26. D. Yu, X. Chen, and L. Deng, "Factorized deep neural networks for adaptive speech recognition", International workshop on statistical machine learning for speech processing, 2012.
27. D. Yu, L. Deng, and F. Seide, "The deep tensor neural network with applications to large vocabulary speech recognition", IEEE Trans. on Audio, Speech, and Language Processing, 2013.
28. J. Snoek, H. Larochelle, and RP. Adams, "Practical bayesian optimization of machine learning algorithms," arXiv:0912.4896, 2012.
29. D. Yu, L. Deng, Y. Gong, and A. Acero, "A novel framework and training algorithm for variable-parameter hidden markov models", IEEE Trans. on Audio, Speech, and Language Processing, vol 17, no. 7, pp. 1348-1360, 2009.
30. J. Trmal, J. Zelinka, and L. Müller, "On speaker adaptive training of artificial neural networks", in Proc. Interspeech'10, pp. 554-557, 2010.