Learning small-size DNN with output-distribution-based criteria

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1910-1914

  Li, Jinyu ^a   Zhao, Rui ^b   Huang, Jui Ting ^a   Gong, Yifan ^a  

^a MICROSOFT   (United States)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

Device; DNN; Learning; Model compression; Output distribution

Indexed keywords

CLUSTERING ALGORITHMS; DIGITAL STORAGE; SPEECH COMMUNICATION;

DEVICE; DNN; LEARNING; MODEL COMPRESSION; OUTPUT DISTRIBUTION;

SPEECH RECOGNITION;

EID: 84910035297     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. D. Yu, L. Deng, and G. Dahl, "Roles of pretraining and finetuning in context-dependent DBN-HMMs for real-world speech recognition, " in Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010.
2. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition, " in Proc. Workshop on Automatic Speech Recognition and Understanding, pp. 30-35, 2011.
3. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, " IEEE Trans. on Audio, Speech and Language Processing, vol. 20, no. 1, pp. 30-42, 2012.
4. N. Jaitly, P. Nguyen, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition", in Proc. Interspeech, 2012.
5. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
6. L. Deng, J. Li, J. -T. Huang et al. "Recent advances in deep learning for speech research at Microsoft, " in Proc. ICASSP, 2013.
7. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets, " in Proc. ICASSP, pp. 6655-6659, 2013.
8. J. Xue, J. Li, and Y. Gong, "Restructuring of deep neural network acoustic models with singular value decomposition, " in Proc. Interspeech, pp. 2365-2369, 2013.
9. V. Vanhoucke, A. Senior, and M. Z. Mao, "Improving the speed of neural networks on CPUs, " in Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
10. V. Vanhoucke, M. Devin, and G. Heigold. "Multiframe deep neural networks for acoustic modeling." In Proc. IEEE Acoustics, Speech and Signal Processing, 2013.
11. X. Lei, A. Senior, A., A. Gruenstein, and J. Sorensen, "Accurate and compact large vocabulary speech recognition on mobile devices, " in Proc. Interspeech, 2013.
12. C. Bucilu, R. Caruana, and A. Niculescu-Mizil, "Model compression, " In Proc. of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535- 541, 2006.
13. L. Ba and R. Caurana, "Do deep nets really need to be deep?" arXiv preprint arXiv:1312.6184, 2013.
14. S. J. Young, J. J. Odell, and P. C. Woodland, "Tree-based state tying for high accuracy acoustic modelling, " in HLT, pp. 307- 312, 1994.
15. G. Heigold, H. Ney, P. Lehnen, T. Gass, and R. Schluter, "Equivalence of generative and log-linear models, " IEEE Transactions on Audio, Speech, and Language Processing, vol.19, no.5, pp.1138-1148, 2011.
16. H. Su, G. Li, D. Yu, and F. Seide, "Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription, " In Proc. ICASSP, 2013.
17. R. Zhao, J. Li, and Y. Gong, "Variable-component deep neural network for robust speech recognition, " In Proc. Interspeech, 2014.