1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1058-1062

  Seide, Frank ^a   Fu, Hao ^a,b   Droppo, Jasha ^c   Li, Gang ^a   Yu, Dong ^c  

^a MICROSOFT RESEARCH ASIA   (China)

^b TSINGHUA UNIVERSITY   (China)

^c MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ION BEAMS; PROGRAM PROCESSORS; STOCHASTIC SYSTEMS;

COMPUTATION SPEED; DEEP NEURAL NETWORKS; DOUBLE BUFFERING; FRAMES PER SECONDS; ITS APPLICATIONS; LOSS OF ACCURACY; QUANTIZATION ERRORS; STOCHASTIC GRADIENT DESCENT;

SPEECH COMMUNICATION;

EID: 84910069984     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. D. Yu, L. Deng, and G. Dahl, "Roles of Pretraining and Fine-Tuning in Context-Dependent DNN-HMMs for Real-World Speech Recognition, " NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Dec. 2010.
2. F. Seide, G. Li, and D. Yu, "Conversational Speech Transcription Using Context-Dependent Deep Neural Networks, " Interspeech, 2011.
3. Q.-V. Le, M.-A. Ranzato, R. Monga, M. Devin, K. Chen, G.-S. Corrado, J. Dean, and A.-Y. Ng, "Building High- Level Features Using Large Scale Unsupervised Learning, " ICML, 2012.
4. F. Niu, B. Recht, C. Re, and S. J. Wright, "Hogwild!: A lock-free approach to parallelizing stochastic gradient descent, " arXiv preprint arXiv:1106.5730, 2011.
5. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M.-A. Ranzato, A. Senior, P. Tucker, K. Yang, A. Y. Ng, "Large Scale Distributed Deep Networks, " NIPS, 2012.
6. A. Coates, B. Huval, T. Wang, D.-J. Wu, and A.-Y. Ng, "Deep Learning with COTS HPC Systems, " ICML, 2013.
7. S. Zhang, C. Zhang, Z. You, R. Zheng, and B. Xu, "Asynchronous Stochastic Gradient Descent for DNN Training, " ICASSP, 2013.
8. F. Seide, H. Fu, J. Droppo, G. Li, D. Yu, "On Parallelizability of Stochastic Gradient Descent for Speech DNNs, " ICASSP 2014.
9. "Delta-Sigma Modulation, " Wikipedia, http://en.wikipedia.org/wiki/Delta-sigmamodulation.
10. H. Franco et al., "Context-Dependent Connectionist Probabilty Estimatation in a Hybrid Hidden Markov Model- Neural Net Speech Recognition System, " Computer Speech and Language, vol. 8, pp. 211-222, 1994.
11. P. Zhou, C. Liu, Q. Liu, L. Dai, and H. Jiang, "A Cluster- Based Multiple Deep Neural Networks Method for Large Vocabulary Continuous Speech Recognition, " ICASSP, 2013.
12. T.-N. Sainath, B. Kingsbury, H. Soltau, and B. Ramabhadran, "Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech Tasks, " IEEE Trans. on Audio, Speech, and Language Processing, Vol. 21, No. 11, Nov. 2013.
13. J. Xue, J. Li, and Y. Gong, "Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition, " Interspeech 2013.
14. S. Wiesler, J. Li, and J. Xue, "Investigations on Hessian- Free Optimization for Cross-Entropy Training of Deep Neural Networks, " Interspeech, 2013.
15. B. Kingsbury, T. Sainath, and H. Soltau, "Scalable Minimum Bayes Risk Training of Deep Neural Network Acoustic Models Using Distributed Hessian-free Optimization, " Interspeech, 2012.
16. X. Zhang, J. Trmal, D. Povey, S. Khudanpur, "Improving Deep Neural Network Acoustic Models Using Generalized Maxout Networks, " ICASSP 2014.
17. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, " in Foundations and Trends in Machine Learning, Vol. 3, No. 1 (2010) 1-122.
18. Q. Huo, Z. Yan, K. Chen, "Deep Learning Using Alternating Direction Method of Multipliers, " U.S. Patent Application, filed on 4/8/2014.
19. X. Chen, A. Eversole, G. Li, D. Yu, and F. Seide, "Pipelined Back-Propagation for Context-Dependent Deep Neural Networks, " Interspeech, 2012.
20. F. Rosenblatt, "Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms", Spartan Books, Wash. DC, 1961.
21. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context- Dependent Pre-Trained Deep Neural Networks for Large Vocabulary Speech Recognition, " IEEE Trans. Speech and Audio Proc., Special Issue on Deep Learning for Speech and Language Processing, 2011.
22. G. Hinton, S. Osindero, and Y. Teh, "A Fast Learning Algorithm for Deep Belief Nets", Neural Computation, vol. 18, pp. 1527-1554, 2006.
23. F. Seide, G. Li, X. Chen, and D. Yu, "Feature Engineering in Context-Dependent Deep Neural Networks for Conversational Speech Transcription, " Proc. ASRU, Waikoloa Village, 2011.
24. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " ICASSP, 2009.
25. D. Rumelhart, G. Hinton, and R. Williams, "Learning Representations By Back-Propagating Errors, " Nature, vol. 323, Oct. 1986.
26. J. Martens, "Deep learning via Hessian-free optimization, " ICML, 2010.
27. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The Rprop algorithm, " International Conference on Neural Networks, 1993.
28. J. Godfrey and E. Holliman, "Switchboard-1 Release 2, " Linguistic Data Consortium, Philadelphia, 1997.
29. "Evaluation Campaign, " IWSLT 2013, http://www.iwslt2013.org/59.php.
30. J. Duchi, E. Hazan, and Y. Singer, "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, " http://www.cs.berkeley.edu/-jduchi/projects/DuchiHaSi10.pdf, 2010.
31. A. Senior, G. Heigold, M.-A. Ranzato, K. Yang, "An Empirical Study of Learning Rates in Deep Neural Networks for Speech Recognition, " ICASSP, 2013.