Scalable distributed DNN training using commodity GPU cloud computing

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2015-January, Issue , 2015, Pages 1488-1492

  Strom, Nikko ^a  

^a AMAZON   (United States)

Author keywords

Deep neural networks; Distributed stochastic gradient descent; Speech recognition

Indexed keywords

SPEECH RECOGNITION; STOCHASTIC SYSTEMS; TIME VARYING NETWORKS;

ACOUSTIC MODELLING; BOTTLENECK PROBLEM; COMMODITY CLOUDS; COMMUNICATION BANDWIDTH; CONVERGENCE RATES; DEEP NEURAL NETWORKS; STOCHASTIC GRADIENT DESCENT; THREE ORDERS OF MAGNITUDE;

SPEECH COMMUNICATION;

EID: 84959142008     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition, " IEEE Signal Processing Magazine, 2012.
2. Y. LeCun, F.-J. Huang and L. Bottou, "Learning methods for generic object recognition with invariance to pose and lighting, " in Proc. of the Computer Vision and Pattern Recognition Conference, Los Alamitos, CA, 2004.
3. J. Martens, "Deep learning via Hessian-free optimization, " in Proc. of ICML, 2010.
4. T. N. Sainath, I.-h. Chung, B. Ramabhadran, M. Picheny, J. Gunnels, B. Kingsbury, G. Saon, V. Austel and U. Chaudhari, "Parallel deep neural network training for LVCSR tasks using Blue Gene/Q, " in Proc. of INTERSPEECH, Singapore, 2014.
5. D. Povey, X. Zhang and S. Khudanpur, "Parallel training of DNNs with natural gradient and parameter averaging, " in Proc. of ICLR 2015, San Diego, 2015.
6. J. Duchi, E. Hazan and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization, " Journal of Machine Learning Research, vol. 12, p. 2121-2159, 2011.
7. M. D. Zeiler, "ADADELTA: An adaptive learning rate method, " arXiv preprint arXiv: 1212. 5701, 2012.
8. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. A. Ranzato, A. Senior, P. Tucker, K. Yang and A. Y. Ng., "Large scale distributed deep networks, " in Proc. of Advances in Neural Information Processing Systems, Lake Tahoe, NV, 2012.
9. F. Seide, F. Hao, J. Droppo, G. Li and D. Yu, "On parallelizability of stochastic gradient descent for speech DNNs, " in Proc. of ICASSP, Florence, Italy, 2014.
10. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro and N. A, "Deep learning with COTS HPC systems, " in Proc. of the 30th International Conference on Machine Learning, 2013.
11. Z. Shanshan, C. Zhang, Z. You, Z. Rong and B. Xu, "Asynchronous stochastic gradient descent for DNN training, " in Proc. of ICASSP, Vancouver, Canada, 2013.
12. C. Xie, A. Eversole, G. Li, D. Yu and F. Seide, "Pipelined Back-Propagation for Context-Dependent Deep Neural Networks, " in Proc. of INTERSPEECH, Portland, OR, 2012.
13. F. Seide, H. Fu, J. Droppo, G. Li and D. Yu, "1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs, " in Proc. of INTERSPEECH, Singapore, 2014.
14. A. Agarwal and J. J. Duchi, "Distributed delayed stochastic optimization, " in Proc. of Advances in Neural Information Processing Systems, Granada, Spain, 2011.
15. Y. Nesterov, "Gradient methods for minimizing composite objective function, " in CORE Discussion Paper 2007/76, Catholic University of Louvain, 2007.
16. B. Recht, C. Re, S. Wright and F. Niu, "Hogwild: A lock-free approach to parallelizing stochastic gradient descent, " in Proc. of Advances in Neural Information Processing Systems, Granada, Spain, 2011.
17. N. Ström, "A tonotopic artificial neural network architechture for phoneme probability estimation, " in Proc. of IEEE Workshop on Speech Recognition and Understanding, Santa Barbara, CA, 1997.
18. N. Ström, "Sparse connection and pruning in large dynamic artificial neural networks, " in Proc. of EUROSPEECH, Rhodes, Greece, 1997.
19. N. Wilt, The cuda handbook: A comprehensive guide to gpu programming, Pearson Education, 2013.
20. R. F. Rice and R. Plaunt, "Adaptive Variable-Length Coding for Efficient Compression of Spacecraft Television Data, " IEEE Transactions on Communication Technology, vol. 16, no. 9, 1971.
21. K. Veselý., A. Ghoshal, L. Burget and D. Povey, "Sequencediscriminative training of deep neural networks, " in Proc. of INTERSPEECH, Lyon, France, 2013.
22. F. Seide, G. Li, X. Chen and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. of IEEE ASRU, Waikoloa, HI, 2011.