Improving deep neural network acoustic models using unlabeled data

2013 IEEE China Summit and International Conference on Signal and Information Processing, ChinaSIP 2013 - Proceedings

Volumn , Issue , 2013, Pages 137-141

  Cai, Meng ^a   Zhang, Wei Qiang ^a   Liu, Jia ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

acoustic modeling; deep neural network; speech recognition; unlabeled data

Indexed keywords

A-PRIORI PROBABILITIES; ACOUSTIC MODEL; CONTEXT DEPENDENT; DEEP NEURAL NETWORKS; SMOOTHING ALGORITHMS; TRAINING PROCEDURES; TRAINING PROCESS; UNLABELED DATA;

BENCHMARKING; DATA PROCESSING; SPEECH RECOGNITION;

NEURAL NETWORKS;

EID: 84889596187     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ChinaSIP.2013.6625314     Document Type: Conference Paper

References (24)

1. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for large vocabulary speech recognition, " IEEE Transactions on Audio, Speech and Language Processing, vol. 20, pp. 30-42, January 2012.
2. G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Processing Magazine, vol. 29, pp. 82-97, November 2012.
3. A. Mohamed, G. E. Dahl, and G. E. Hinton, "Deep belief networks for phone recognition, " in Proc. NIPS Workshop Deep Learning for Speech Recognition and Related Aplications. Neural Information Processing Systems Foundation, 2009.
4. A. Mohamed, G. E. Dahl, and G. E. Hinton, "Acoustic modeling using deep belief networks, " IEEE Transactions on Audio, Speech and Language Processing, vol. 20, pp. 14-22, January 2012.
5. L. Deng, M. Seltzer, D. Yu, A. Acero, A. Mohamed, and G. Hinton, "Binary coding of speech spectrograms using a deep auto-encoder, " in Proc. INTERSPEECH. ISCA, 2010.
6. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Large vocabulary continuous speech recognition with contextdependent DBN-HMMs, " in Proc. ICASSP. IEEE, 2011, pp. 4688-4691.
7. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in Proc. Interspeech. ISCA, 2011, pp. 437-440.
8. L. Deng, J. Li, J. Huang, K. Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X. He, J. Williams, Y. Gong, and A. Acero, "Recent advances in deep learning for speech research at Microsoft, " in Proc. ICASSP. IEEE, 2013.
9. L. Deng and D. Yu, "Deep convex network: A scalable architecture for speech pattern classification, " in Proc. INTERSPEECH. ISCA, 2011.
10. B. Hutchinson, L. Deng, and D. Yu, "A deep architecture with bilinear modeling of hidden representations: Applications to phonetic recognition, " in Proc. ICASSP. IEEE, 2012.
11. D. Yu, L. Deng, and F. Seide, "The deep tensor neural network with applications to large vocabulary speech recognition, " IEEE Transactions on Audio, Speech and Language Processing, vol. 21, no. 2, pp. 388-396, Feburary 2013.
12. L. Deng, B. Hutchinson, and D. Yu, "Parallel training of deep stacking networks, " in Proc. INTERSPEECH. ISCA, 2012.
13. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach, Kluwer Academic Publishers, 1993.
14. G. E. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks, " Science, vol. 313, pp. 504-507, July 2006.
15. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets, " Neural Computation, vol. 18, no. 7, pp. 1527-1554, 2006.
16. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagation errors, " Nature, vol. 323, no. 6088, pp. 533-536, 1986.
17. G. E. Hinton, "A practical guide to training restricted boltzmann machines, " Tech. Rep. UTML TR 2010-003, Department of Computer Science, University of Toronto, 2010.
18. X. Chen, A. Eversole, G. Li, D. Yu, and F. Seide, "Pipelined back-propagation for context-dependent deep neural networks, " in Proc. Interspeech. ISCA, 2012.
19. J. Godfrey, E. Holliman, and J. McDaniel, "Switchboard: Telephone speech corpus for research and development, " in Proc. ICASSP. IEEE, 1992, pp. 517-520.
20. S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, Povey D., Valtchev V., and Woodland P., The HTK book, version 3. 4. 1, Cambridge University Engineering Department, 2009.
21. V. Mnih, "CUDAMat: a CUDA-based matrix class for python, " Tech. Rep. UTML TR 2009-004, Department of Computer Science, University of Toronto, 2009.
22. Y. Qian, J. Xu, D. Povey, and J. Liu, "Strategies for using MLP based features with limited target-language training data, " in Proc. ASRU. IEEE, 2011, pp. 354-358.
23. D. Yu, L. Deng, and G. E. Dahl, "Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition, " in Proc. NIPSWorkshop on Deep Learning and Unsupervised Feature Learning. Neural Information Processing Systems Foundation, 2010.
24. Q. V. Le, M. A. Ranzato, M. Monga, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng, "Building high-level features using large-scale unsupervised learning, " in Proc. ICML. International Machine Learning Society, 2012.