Practical efficiency of asynchronous stochastic gradient descent

Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis

Volumn , Issue , 2017, Pages 56-62

  Bhardwaj, Onkar ^a   Cong, Guojing ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING SYSTEMS; SPEED; STOCHASTIC SYSTEMS;

COMMUNICATION COST; COMPUTING RESOURCE; CONVERGENCE RATES; CONVERGENCE SPEED; FINITE-TIME CONVERGENCE; LARGE-SCALE MACHINE LEARNING; MATHEMATICAL ANALYSIS; STOCHASTIC GRADIENT DESCENT;

COST BENEFIT ANALYSIS;

EID: 85015209547     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/MLHPC.2016.10     Document Type: Conference Paper

References (28)

1. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al., "Large scale distributed deep networks," in Advances in neural information processing systems, 2012, pp. 1223-1231.
2. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, "Scaling distributed machine learning with the parameter server," in 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014, pp. 583-598.
3. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman, "Project adam: Building an efficient and scalable deep learning training system," in 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014, pp. 571-582.
4. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, "On parallelizability of stochastic gradient descent for speech dnns," in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 235-239.
5. W. Chan and I. Lane, "Distributed asynchronous optimization of convolutional neural networks." in INTERSPEECH, 2014, pp. 1073-1077.
6. X. Lian, Y. Huang, Y. Li, and J. Liu, "Asynchronous parallel stochastic gradient for nonconvex optimization," in Advances in Neural Information Processing Systems, 2015, pp. 2737-2745.
7. S. Ghadimi and G. Lan, "Stochastic first-and zeroth-order methods for nonconvex stochastic programming," SIAM Journal on Optimization, vol. 23, no. 4, pp. 2341-2368, 2013.
8. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao, "Optimal distributed online prediction using mini-batches," Journal of Machine Learning Research, vol. 13, no. Jan, pp. 165-202, 2012.
9. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, "Robust stochastic approximation approach to stochastic programming," SIAM Journal on optimization, vol. 19, no. 4, pp. 1574-1609, 2009.
10. R. Johnson and T. Zhang, "Accelerating stochastic gradient descent using predictive variance reduction," in Advances in Neural Information Processing Systems, 2013, pp. 315-323.
11. L. Xiao and T. Zhang, "A proximal stochastic gradient method with progressive variance reduction," SIAM Journal on Optimization, vol. 24, no. 4, pp. 2057-2075, 2014.
12. C.-J. Hsieh, K.-W. Chang, C.-J. Lin, S. S. Keerthi, and S. Sundararajan, "A dual coordinate descent method for large-scale linear svm," in Proceedings of the 25th international conference on Machine learning. ACM, 2008, pp. 408-415.
13. P. Richtárik and M. Takác, "Distributed coordinate descent method for learning with big data," 2013.
14. O. Fercoq, Z. Qu, P. Richtárik, and M. Takáč, "Fast distributed coordinate descent for non-strongly convex losses," in 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2014, pp. 1-6.
15. J. Konečnỳ, Z. Qu, and P. Richtárik, "Semi-stochastic coordinate descent," arXiv preprint arXiv:1412.6293, 2014.
16. Z. Qu and P. Richtárik, "Coordinate descent with arbitrary sampling i: Algorithms and complexity," arXiv preprint arXiv:1412.8060, 2014.
17. J. Liu, S. J. Wright, C. Re, V. Bittorf, and S. Sridhar, "An asynchronous parallel stochastic coordinate descent algorithm," Journal of Machine Learning Research, vol. 16, no. 285-322, pp. 1-5, 2015.
18. M. Jaggi, V. Smith, M. Takac, J. Terhorst, S. Krishnan, T. Hofmann, and M. I. Jordan, "Communication-efficient distributed dual coordinate ascent," in Advances in Neural Information Processing Systems, 2014, pp. 3068-3076.
19. Y. Nesterov, "Efficiency of coordinate descent methods on huge-scale optimization problems," SIAM Journal on Optimization, vol. 22, no. 2, pp. 341-362, 2012.
20. J. Liu and S. J. Wright, "Asynchronous stochastic coordinate descent: Parallelism and convergence properties," SIAM Journal on Optimization, vol. 25, no. 1, pp. 351-376, 2015.
21. Q. Meng, W. Chen, J. Yu, T. Wang, Z.-M. Ma, and T.-Y. Liu, "Asynchronous accelerated stochastic gradient descent," in Proceedings of the 25th international joint conference on Artificial Intelligence, 2016.
22. P. Richtárik and M. Takáč, "Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function," Mathematical Programming, vol. 144, no. 1-2, pp. 1-38, 2014.
23. Y. Nesterov, Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2013, vol. 87.
24. "Cifar10 dataset," https://www.cs.toronto.edu/~kriz/cifar.html, accessed January 11, 2016.
25. "Cifar10 model," https://github.com/eladhoffer/ConvNet-torch/blob/master/Models/Model.lua, accessed January 11, 2016.
26. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting." Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
27. "Torch: A scientific computing framework for luajit," http://torch.ch.
28. Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, "Efficient backprop," in Neural networks: Tricks of the trade. Springer, 2012, pp. 9-48.