Communication-efficient distributed optimization using an approximate Newton-type method

31st International Conference on Machine Learning, ICML 2014

Volumn 3, Issue , 2014, Pages 2665-2681

  Shamir, Ohad ^a   Srebro, Nathan ^b   Zhang, Tong ^c,d  

^a WEIZMANN INSTITUTE OF SCIENCE   (Israel)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^c RUTGERS UNIVERSITY   (United States)

^d BAIDU INC   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; OPTIMIZATION;

DATA SIZE; DISTRIBUTED OPTIMIZATION; LEARNING PROBLEM; LINEAR RATE OF CONVERGENCES; NEWTON-TYPE METHODS; NUMBER OF ITERATIONS; STOCHASTIC OPTIMIZATIONS;

LEARNING SYSTEMS;

EID: 84919935839     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Agarwal, A., Chapelle, O., Dudfk, M., and Langford, J. A reliable effective terascale linear learning system. CoRR, abs/1110.4198, 2011.
2. Beck, A. and Teboulle, M. Mirror descent and nonlinear projected sub gradient methods for convex optimization. Oper. Res. Lett., 31(3): 167-175, 2003.
3. Bekkerman, R., Bilenko, M., and Langford, J. Scaling up machine learning: Parallel and distributed approaches. Cambridge University Press, 2011.
4. Boyd, S.P., Parikh, N., Chu, E., Peleato, B. and Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1): 1-122, 2011.
5. Cotter, A., Shamir, O., Srebro, N., and Sridharan, K. Better mini-batch algorithms via accelerated gradient methods. In NIPS, 2011.
6. Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction using mini- batches. Journal of Machine Learning Research, 13: 165-202, 2012.
7. Deng, W. and Yin, W. On the global and linear convergence of the generalized alternating direction method of multipliers. Technical report, Rice University Technical Report TR12-14, 2012.
8. Duchi, J., Agarwal, A., and Wainwright, M. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Trans. Automat. Contr., 57(3): 592-606, 2012.
9. Hong, M. and Luo, Z.-Q. On the linear convergence of the alternating direction method of multipliers. CoRR, abs/1208.3922, 2012.
10. Mahajan, D., Keerthy, S., Sundararajan, S., and Bottou, L. A parallel sgd method with strong convergence. CoRR, abs/1311.0636, 2013.
11. Nemirovski, A. and Yudin, D. On cesaro's convergence of the gradient descent method for finding saddle points of convex-concave functions. Doklady Akademii Nauk SSSR, 239(4), 1978.
12. Nemirovsky, A. and Yudin, D. Problem Complexity and Method Efficiency in Optimization. Wiley-Inter science, 1983.
13. Rakhlin, A., Shamir, O., and Sridharan, K. Making gradient descent optimal for strongly convex stochastic optimization. In ICML, 2012.
14. Recht, B., Re, C., Wright, S. and Niu, F. Hog wild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, 2011.
15. Richtarik, P. and Takac, M. Distributed coordinate descent method for learning with big data. CoRR, abs/1310.2059, 2013.
16. Shalev-Shwartz, S. and Zhang, T. Stochastic dual coordinate ascent methods for regularized loss. Journal of Machine Learning Research, 14(1):567-599, 2013.
17. Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. Stochastic convex optimization. In COLT, 2009.
18. Sridharan, K., Shalev-Shwartz, S., and Srebro, N. Fast rates for regularized objectives. In Advances in Neural Information Processing Systems, pp. 1545-1552, 2008.
19. Tropp, J. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12(4):389-434, 2012.
20. Yang, T. Trading computation for communication: Distributed stochastic dual coordinate ascent. In NIPS, 2013.
21. Zhang, Y., Duchi, J., and Wainwright, M. Communication- efficient algorithms for statistical optimization. Journal of Machine Learning Research, 14:3321-3363, 2013.
22. Zinkevich, M., Weimer, M., Smola, A., and Li, L. Parallelized stochastic gradient descent. In NIPS, pp. 2595- 2603, 2010.