Distributed learning of multilingual DNN feature extractors using GPUs

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 830-834

  Miao, Yajie ^a   Zhang, Hao ^a   Metze, Florian ^a  

^a Language Technologies Institute School of Computer Science Carnegie Mellon University ^*  (United States)

Author keywords

Automatic speech recognition; Deep neural networks; Distributed learning

Indexed keywords

PROGRAM PROCESSORS; SPEECH COMMUNICATION; STOCHASTIC SYSTEMS;

AUTOMATIC SPEECH RECOGNITION; CROSS LANGUAGES; DEEP NEURAL NETWORKS; DISTRIBUTED LEARNING; FEATURE EXTRACTOR; LEARNING PROCESS; MULTILINGUAL TRAININGS; STOCHASTIC GRADIENT DESCENT;

SPEECH RECOGNITION;

EID: 84910068044     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. G. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for large vocabulary speech recognition, " IEEE Transactions on Audio, Speech and Language Processing, vol. 20(1), pp. 30-42, 2012.
2. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. ASRU, pp. 24-29, 2011.
3. Y. Miao, H. Zhang, and F. Metze, "Towards speaker adaptive training of deep neural network acoustic models, " to appear in Proc. Interspeech, 2014.
4. Y. Miao, and F. Metze, "Improving low-resource CDDNN- HMM using dropout and multilingual DNN training, " in Proc. Interspeech, pp. 2237-2241, 2013.
5. Y. Miao, F. Metze, and S. Rawat, "Deep maxout networks for low-resource speech recognition, " in Proc. ASRU, 2013.
6. P. Swietojanski, A. Ghoshal, and S. Renals, "Unsupervised cross-lingual knowledge transfer in DNNbased LVCSR, " in Proc. SLT, pp. 246-251, 2012.
7. N. T. Vu, W. Breiter, F. Metze, and T. Schultz, "An investigation on initialization schemes for multilayer perceptron training using multilingual data and their effect on ASR performance, " in Proc. Interspeech, 2012.
8. D. Yu, M. L. Seltzer, J. Li, J. Huang, and F. Seide, "Feature learning in deep neural networks - studies on speech recognition tasks, " in International Conference on Learning Representations 2013.
9. J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, "Crosslanguage knowledge transfer using multilingual deep neural network with shared hidden layers, " in Proc. ICASSP, pp. 7304-7308, 2013.
10. X. Zhang, J. Trmal, D. Povey, and S. Khudanpur, "Improving deep neural network acoustic models using generalized maxout networks, " in Proc. ICASSP, 2014.
11. T. N. Sainath, B. Kingsbury, H. Soltau, and B. Ramabhadran, "Optimization techniques to improve training speed of deep neural networks for large speech tasks, " IEEE Transactions on Audio, Speech and Language Processing, vol. 21, no. 11, pp. 2267-2276, 2013.
12. X. Chen, A. Eversole, G. Li, D. Yu, and F. Seide, "Pipelined back-propagation for context-dependent deep neural networks, " in Proc. Interspeech, 2012.
13. S. Zhang, C. Zhang, Z. You, R. Zheng, and B. Xu, "Asynchronous stochastic gradient descent for DNN training, " in Proc. ICASSP, pp. 6660-6663, 2013.
14. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, "Multilingual acoustic models using distributed deep neural networks, " in Proc. ICASSP, pp. 8619-8623, 2013.
15. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng, "Large scale distributed deep networks, " in Proc. NIPS, 2012.
16. Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng, "Building high-level features using large scale unsupervised learning, " in Proc. International Conference on Machine Learning, pp. 81- 88, 2012.
17. Y. Miao, and F. Metze, "Improving language-universal feature extraction with deep maxout and convolutional neural networks, " to appear in Proc. Interspeech, 2014.
18. Y. Miao, F. Metze, and A. Waibel, "Subspace mixture model for low-resource speech recognition in crosslingual settings, " in Proc. ICASSP, pp. 7339-7342, 2013.
19. G. Mann, R. Mcdonald, M. Mohri, N. Silberman, and D. D. Walker, "Efficient large-scale distributed training of conditional maximum entropy models, " in Proc. NIPS, 2009.
20. A. Asuncion, P. Smyth, and M. Welling, "Asynchronous distributed learning of topic models, " in Proc. NIPS, 2012.
21. M. A. Zinkevich, A. Smola, M. Weimer, L. Li, "Parallelized stochastic gradient descent, " in Proc. NIPS, 2010.
22. J. Park, F. Diehl, M.J.F. Gales, M. Tomalin, and P.C. Woodland, "The efficient incorporation of MLP features into automatic speech recognition systems, " Computer Speech and Language, volume 25, issue 3, pp. 519-534, 2011.
23. F. Valente, M. M. Doss, C. Plahl, S. Ravuri, and W. Wang, "Transcribing Mandarin broadcast speech using multi-layer perceptron acoustic features, " IEEE Transactions on Audio, Speech and Language Processing, vol. 19, no. 8, pp. 2439-2450, 2011.
24. Y. Qian, and J. Liu, "Cross-lingual and ensemble MLPs strategies for low-resource speech recognition, " in Proc. Interspeech, 2012.
25. J. Gehring, Y. Miao, F. Metze, and A. Waibel, "Extracting deep bottleneck features using stacked autoencoders, " in Proc. ICASSP, pp. 3377-3381, 2013.
26. J. Gehring, W. Lee, K. Kilgour, I. Lane, Y. Miao, and A. Waibel, "Modular Combination of Deep Neural Networks for Acoustic Modeling, " in Proc. Interspeech, pp. 94-98, 2013.
27. J. Chiu, and A. Rudnicky, "Using conversational word bursts in spoken term detection, " in Proc. Interspeech, 2013.
28. J. Chiu, Y. Wang, J. Trmal, D. Povey, G. Chen, and A. Rudnicky, "Combination of FST and CN search in spoken term detection, " to appear in Proc. Interspeech, 2014.
29. M. Gales, "Maximum likelihood linear transformations for HMM-based speech recognition, " Computer Speech and Language, vol. 12, pp. 75-98, 1998.
30. Y. Miao, "Kaldi+PDNN: Building DNN-based ASR systems with Kaldi and PDNN, " arXiv:1401.6984, 2014.
31. O. Abdel-Hamid, A. Mohamed, H. Jiang, and G. Penn, "Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition, " in Proc. ICASSP, pp. 4277-4280, 2012.
32. T. N. Sainath, A. Mohamed, B. Kingsbury, and B. Ramabhadran, "Deep convolutional neural networks for LVCSR, " in Proc. ICASSP, pp. 8614-8618, 2013.