Model accuracy and runtime tradeoff in distributed deep learning: A systematic study

Proceedings - IEEE International Conference on Data Mining, ICDM

Volumn 0, Issue , 2016, Pages 171-180

  Gupta, Suyog ^a   Zhang, Wei ^a   Wang, Fei ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b WEILL CORNELL MEDICAL COLLEGE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DISTRIBUTED COMPUTER SYSTEMS; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

DISTRIBUTED COMPUTING FRAMEWORKS; LARGE-SCALES; MODEL TRAINING; MODEL@RUNTIME; MODELING ACCURACY; SERVER-BASED; STOCHASTIC GRADIENT DESCENT ALGORITHM; SYNCHRONIZATION PROTOCOLS; SYSTEMATIC STUDY; TRAINING PROCESS;

DEEP NEURAL NETWORKS;

EID: 85014543286     PISSN: 15504786     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDM.2016.122     Document Type: Conference Paper

References (26)

1. Mpi 3.0 standard. www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf, 2012.
2. Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437-478. Springer, 2012.
3. T. M. Breuel. The effects of hyperparameters on SGD training of neural networks. arXiv:1508.02788, 2015.
4. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project Adam: Building an efficient and scalable deep learning training system. OSDI'14, pages 571-582, 2014.
5. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew. Deep learning with cots hpc systems. In Proceedings of the 30th ICML, pages 1337-1345, 2013.
6. H. e. a. Cui. Exploiting bounded staleness to speed up big data analytics. In USENIX ATC'14, pages 37-48, 2014.
7. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS, 2012.
8. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121-2159, 2011.
9. O. R. et al. ImageNet Large Scale Visual Recognition Challenge. IJCV, pages 1-42, 2015.
10. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 249-256, 2010.
11. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv:1502.01852, 2015.
12. Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing. More effective distributed ML via a stale synchronous parallel parameter server. In NIPS 26, pages 1223-1231. 2013.
13. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
14. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
15. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 1(4):7, 2009.
16. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
17. L. Lamport. Time, clocks, and the ordering of events in a distributed system. Commun. ACM, 21(7):558-565, 1978.
18. Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436-444, 2015.
19. X. Lian, Y. Huang, Y. Li, and J. Liu. Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization. ArXiv eprints, June 2015.
20. M. Raynal and M. Singhal. Logical time: Capturing causality in distributed systems. Computer, 29(2):49-56, 1996.
21. A. Senior, G. Heigold, M. Ranzato, and K. Yang. An empirical study of learning rates in deep neural networks for speech recognition. In ICASSP, pages 6724-6728. IEEE, 2013.
22. A. Smola and S. Narayanamurthy. An architecture for parallel topic models. Proc. VLDB Endow., 3(1-2):703-710, 2010.
23. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In Proceedinge of the 30th ICML, pages 1139-1147, 2013.
24. M. D. Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
25. S. Zhang, A. Choromanska, and Y. LeCun. Deep learning with elastic averaging SGD. CoRR, abs/1412.6651, 2014.
26. Y. Zou, X. Jin, Y. Li, Z. Guo, E. Wang, and B. Xiao. Mariana: Tencent deep learning platform and its applications. Proc. VLDB Endow., 7(13):1772-1777, 2014.