Improving deep neural networks for LVCSR using dropout and shrinking structure

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2014, Pages 6849-6853

  Zhang, Shiliang ^a   Bao, Yebo ^a   Zhou, Pan ^b   Jiang, Hui ^a   Dai, Lirong ^a  

^a National Engineering Laboratory for Speech and Language Information Processing   (China)

^b YORK UNIVERSITY   (Canada)

Author keywords

deep neural networks; DNN HMM; dropout; dropout as pre conditioner (DAP); LVCSR; shrinking hidden layer

Indexed keywords

CONTINUOUS SPEECH RECOGNITION; HIDDEN MARKOV MODELS; SHRINKAGE;

DEEP NEURAL NETWORKS; DNN-HMM; DROPOUT; DROPOUT AS PRE-CONDITIONER (DAP); HIDDEN LAYERS; LVCSR;

SIGNAL PROCESSING;

EID: 84905252086     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2014.6854927     Document Type: Conference Paper

References (21)

1. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks.," in INTERSPEECH, 2011, pp. 437-440.
2. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 1, pp. 30-42, 2012.
3. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription," in ASRU. IEEE, 2011, pp. 24-29.
4. J. Pan, C. Liu, Z. G. Wang, Y. Hu, and H. Jiang, "Investigation of deep neural networks (DNN) for large vocabulary continuous speech recognition: Why DNN surpasses GMMs in acoustic modeling," in 8th International Symposium on Chinese Spoken Language Processing (ISCSLP). IEEE, 2012, pp. 301-305.
5. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A-R Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition," in ASRU. IEEE, 2011, pp. 30-35.
6. Pan Zhou, Cong Liu, Qingfeng Liu, Lirong Dai, and Hui Jiang, "A cluster-based multiple deep neural networks method for large vocabulary continuous speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 6650-6654.
7. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural computation, vol. 18, no. 7, pp. 1527-1554, 2006.
8. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks," Advances in neural information processing systems, vol. 19, pp. 153, 2007.
9. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, "Why does unsupervised pre-training help deep learning?," The Journal of Machine Learning Research, vol. 11, pp. 625-660, 2010.
10. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors," arXiv preprint arXiv:1207. 0580, 2012.
11. Michael L Seltzer, Dong Yu, and YongqiangWang, "An investigation of deep neural networks for noise robust speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7398-7402.
12. Yajie Miao and Florian Metze, "Improving low-resource cddnnhmm using dropout and multilingual dnn training," in Proc. Interspeech, 2013, pp. 2237-2241.
13. G. E. Dahl, T. N. Sainath, and G. E. Hinton, "Improving deep neural networks for LVCSR using rectified linear units and dropout," in ICASSP, 2013.
14. J. Li, X. Wang, and B. Xu, "Understanding the dropout strategy and analyzing its effectiveness on LVCSR," in ICASSP, 2013.
15. D. Yu, F. Seide, G. Li, and L. Deng, "Exploiting sparseness in deep neural networks for large vocabulary speech recognition," in ICASSP. IEEE, 2012, pp. 4409-4412.
16. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets," in ICASSP. IEEE, 2013.
17. J. Xue, J. Y. Li, and Y. F. Gong, "Restructuring of deep neural network acoustic models with singular value decomposition," in INTERSPEECH, 2013.
18. P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol, "Extracting and composing robust features with denoising autoencoders," in Proceedings of the 25th international conference on Machine learning. ACM, 2008, pp. 1096-1103.
19. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," The Journal of Machine Learning Research, vol. 9999, pp. 3371-3408, 2010.
20. Y. B. Bao, H. Jiang, C. Liu, Y. Hu, and L. R. Dai, "Investigation on dimensionality reduction of concatenated features with deep neural network for LVCSR systems," in 11th International Conference on Signal Processing (ICSP), 2012, vol. 1, pp. 562-566.
21. Pan Zhou, Lirong Dai, and Hui Jiang, "Sequence training of multiple deep neural networks for better performance and faster training speed," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014.