An asynchronous parallel stochastic coordinate descent algorithm

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 285-322

  Liu, Ji ^a   Wright, Stephen J ^a   Ré, Christopher ^b   Bittorf, Victor ^a   Sridhar, Srikrishna ^a  

^a University of Wisconsin   (United States)

^b Stanford University   (United States)

Author keywords

Asynchronous parallel optimization; Stochastic coordinate descent

Indexed keywords

ALGORITHMS; CONSTRAINED OPTIMIZATION; FUNCTIONS; NONLINEAR EQUATIONS; OPTIMIZATION; PARALLEL PROCESSING SYSTEMS;

ASYNCHRONOUS PARALLEL; CONSTRAINED FUNCTION; CONVEX FUNCTIONS; COORDINATE DESCENT; LINEAR CONVERGENCE RATE; MULTI-CORE SYSTEMS; STRONG CONVEXITIES; UNCONSTRAINED OPTIMIZATION;

STOCHASTIC SYSTEMS;

EID: 84930680874     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (32)

1. A. Agarwal and J. C. Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information Processing Systems 24, pages 873-881. 2011. URL http://papers. nips.cc/paper/4247-distributed-delayed-stochastic-optimization.pdf.
2. H. Avron, A. Druinsky, and A. Gupta. Revisiting asynchronous linear solvers: Provable convergence rate through randomization. IPDPS, 2014.
3. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sciences, 2(1):183-202, 2009.
4. A. Beck and L. Tetruashvili. On the convergence of block coordinate descent type methods. SIAM Journal on Optimization, 23(4):2037-2060, 2013.
5. D. P. Bertsekas and J. N. Tsitsiklis. Parallel and Distributed Computation: Numerical Methods. Pentice Hall, 1989.
6. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1):1-122, 2011.
7. C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines, 2011. URL http://www.csie.ntu.edu.tw/-cjlin/libsvm/.
8. C. Cortes and V. Vapnik. Support vector networks. Machine Learning, pages 273-297, 1995.
9. A. Cotter, O. Shamir, N. Srebro, and K. Sridharan. Better mini-batch algorithms via accelerated gradient methods. In Advances in Neural Information Processing Systems 24, pages 1647-1655. 2011. URL http://papers.nips.cc/paper/ 4432-better-mini-batch-algorithms-via-accelerated-gradient-methods.pdf.
10. J. C. Duchi, A. Agarwal, and M. J.Wainwright. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic Control, 57 (3):592-606, 2012.
11. M. C. Ferris and O. L. Mangasarian. Parallel variable distribution. SIAM Journal on Optimization, 4(4):815-832, 1994
12. D. Goldfarb and S. Ma. Fast multiple-splitting algorithms for convex optimization. SIAM Journal on Optimization, 22(2):533-556, 2012.
13. Z. Lu and L. Xiao. On the complexity analysis of randomized block-coordinate descent methods. Technical Report arXiv:1305.4723, Simon Fraser University, 2013.
14. Z. Q. Luo and P. Tseng. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72:7-35, 1992.
15. O. L. Mangasarian. Parallel gradient distribution in unconstrained optimization. SIAM Journal on Optimization, 33(1):916-1925, 1995.
16. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19:1574-1609, 2009.
17. Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, 2004.
18. Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-362, 2012.
19. F. Niu, B. Recht, C. Ré, and S. J. Wright. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems 24, pages 693-701, 2011.
20. Z. Peng, M. Yan, and W. Yin. Parallel and distributed sparse optimization. Preprint, 2013.
21. P. Richtárik and M. Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Mathematrical Programming, 144:1-38, 2012a.
22. P. Richtárik and M. Takáč. Parallel coordinate descent methods for big data optimization. Technical Report arXiv:1212.0873, 2012b.
23. C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin. Feature clustering for accelerating parallel coordinate descent. Advances in Neural Information Processing Systems 25, pages 28-36, 2012.
24. S. Shalev-Shwartz and T. Zhang. Accelerated mini-batch stochastic dual coordinate ascent. Advances in Neural Information Processing Systems 26, pages 378-385, 2013.
25. O. Shamir and T. Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Proceedings of the 30th International Conference on Machine Learning, 2013.
26. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267-288, 1996.
27. P. Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications, 109:475-494, 2001.
28. P. Tseng and S. Yun. A coordinate gradient descent method for nonsmooth separable minimization. Mathematical Programming, Series B, 117:387-423, June 2009.
29. P. Tseng and S. Yun. A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training. Computational Optimization and Applications, 47(2):179-206, 2010.
30. P.-W. Wang and C.-J. Lin. Iteration complexity of feasible descent methods for convex optimization. Journal of Machine Learning Research, 15:1523-1548, 2014.
31. S. J. Wright. Accelerated block-coordinate relaxation for regularized optimization. SIAM Journal on Optimization, 22(1):159-186, 2012.
32. T. Yang. Trading computation for communication: Distributed stochastic dual coordinate ascent. Advances in Neural Information Processing Systems 26, pages 629-637, 2013.