Asynchronous stochastic coordinate descent: Parallelism and convergence properties

SIAM Journal on Optimization

Volumn 25, Issue 1, 2015, Pages 351-376

  Liu, Ji ^a   Wright, Stephen J ^a  

^a University of Wisconsin   (United States)

Author keywords

Asynchronous parallelism; Composite objective; Inconsistent read; Stochastic coordinate descent

Indexed keywords

ALGORITHMS; FUNCTIONS; NONLINEAR EQUATIONS; PARALLEL PROCESSING SYSTEMS;

ASYNCHRONOUS PARALLEL; ASYNCHRONOUS PARALLELISM; COMPOSITE OBJECTIVE FUNCTION; CONVERGENCE PROPERTIES; COORDINATE DESCENT; INCONSISTENT READ; LINEAR CONVERGENCE RATE; MULTI-CORE PROCESSOR;

STOCHASTIC SYSTEMS;

EID: 84925448697     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/140961134     Document Type: Article

References (45)

1. A. AGARWAL AND J. C. DUCHI, Distributed delayed stochastic optimization, in Proceedings of the Conference on Decision and Control, 2012, pp. 5451-5452.
2. M. ANITESCU, Degenerate nonlinear programming with a quadratic growth condition, SIAM J. Optim., 10 (2000), pp. 1116-1135.
3. H. AVRON, A. DRUINSKY, AND A. GUPTA, Revisiting asynchronous linear solvers: Provable convergence rate through randomization, in Proceedings of the IEEE International Parallel and Distributed Processing Symposium, 2014.
4. A. BECK AND M. TEBOULLE, A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM J. Imaging Sci., 2 (2009), pp. 183-202.
5. A. BECK AND L. TETRUASHVILI, On the convergence of block coordinate descent type methods, SIAM J. Optim., 23 (2013), pp. 2037-2060.
6. D. P. BERTSEKAS AND J. N. TSITSIKLIS, Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Englewood Cliffs, NJ, 1989.
7. S. BOYD, N. PARIKH, E. CHU, B. PELEATO, AND J. ECKSTEIN, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Machine Learning, 3 (2011), pp. 1-122.
8. J. K. BRADLEY, A. KYROLA, D. BICKSON, AND C. GUESTRIN, Parallel coordinate descent for L1-regularized loss minimization, in Proceedings of the International Conference on Machine Learning, 2011.
9. C. CORTES AND V. VAPNIK, Support vector networks, Machine Learning, (1995), pp. 273-297.
10. A. COTTER, O. SHAMIR, N. SREBRO, AND K. SRIDHARAN, Better mini-batch algorithms via accelerated gradient methods, in NIPS 2011, Adv. Neural Inf. Process. Syst. 24, 2011, pp. 1647-1655.
11. J. C. DUCHI, A. AGARWAL, AND M. J. WAINWRIGHT, Dual averaging for distributed optimization: Convergence analysis and network scaling, IEEE Trans. Automat. Control, 57 (2012), pp. 592-606.
12. L. ELSNER, I. KOLTRACHT, AND M. NEUMANN, Convergence of sequential and asynchronous paracontractions nonlinear paracontractuions, Numer. Math., 62 (1992), pp. 305-316.
13. F. FACCHINEI, S. SAGRATELLA, AND G. SCUTARI, Flexible Parallel Algorithms for Big Data Optimization, Technical report, arXiv:1311.2444v1, 2013.
14. O. FERCOQ AND P. RICHTÁRIK, Accelerated, Parallel, and Proximal Coordinate Descent, Technical report, arXiv:1312.5799, 2013.
15. O. FERCOQ AND P. RICHTÁRIK, Smooth Minimization of Nonsmooth Functions by Parallel Coordinate Descent, Technical report, arXiv:1309.5885, 2013.
16. M. C. FERRIS AND O. L. MANGASARIAN, Parallel variable distribution, SIAM J. Optim., 4 (1994), pp. 815-832.
17. A. FROMMER AND D. B. SZYLD, On asynchronous iterations, J. Comput. Appl. Math., 123 (2000), pp. 201-216.
18. D. GOLDFARB AND S. MA, Fast multiple-splitting algorithms for convex optimization, SIAM J. Optim., 22 (2012), pp. 533-556.
19. A. J. HOFFMAN, On approximate solutions of systems of linear inequalities, J. Res. Natl. Bureau Standards, 49 (1952), pp. 263-265.
20. M. LAI AND W. YIN, Augmented L1 and nuclear-norm models with a globally linearly convergent algorithm, SIAM J. Imaging Sci., 6 (2013), pp. 1059-1091.
21. J. LIU, S. J. WRIGHT, C. RÉ, V. Bittorf, and S. Sridhar, An Asynchronous Parallel Stochastic Coordinate Descent Algorithm, Technical report, arXiv:1311.1873, 2013.
22. J. LIU, S. J. WRIGHT, AND S. SRIDHAR, An asynchronous Parallel Randomized Kaczmarz Algorithm, Technical report, arXiv:1401.4780, 2014.
23. Z. LU AND L. XIAO, On the Complexity Analysis of Randomized Block-Coordinate Descent Methods, Technical report, arXiv:1305.4723, 2013.
24. Z.-Q. LUO AND P. TSENG, On the convergence of the coordinate descent method for convex differentiable minimization, J. Optim. Theory Appl., 72 (1992), pp. 7-35.
25. O. L. MANGASARIAN, Parallel gradient distribution in unconstrained optimization, SIAM J. Optim., 33 (1995), pp. 916-1925.
26. I. NECOARA AND D. CLIPICI, Efficient Parallel Ccoordinate Descent Algorithm for Convex Optimization Problems with Separable Constraints: Application to Distributed MPC, Technical report, arXiv:1302.3092, 2013.
27. I. NECOARA AND A. PATRASCU, A Random Coordinate Descent Algorithm for Optimization Problems with Composite Objective Function and Linear Coupled Constraints, Technical report, arXiv:1302.3074, 2013.
28. A. NEMIROVSKI, A. JUDITSKY, G. LAN, AND A. SHAPIRO, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19 (2009), pp. 1574-1609.
29. Y. NESTEROV, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic, Norwell, MA, 2004.
30. Y. NESTEROV, Efficiency of coordinate descent methods on huge-scale optimization problems, SIAM J. Optim., 22 (2012), pp. 341-362.
31. F. NIU, B. RECHT, C. RÉ, AND S. J. WRIGHT, Hogwild!: A lock-free approach to parallelizing stochastic gradient descent, Adv. Neural Inform. Process. Systems, 24 (2011), pp. 693-701.
32. Z. PENG, M. YAN, AND W. YIN, Parallel and Distributed Sparse Optimization, Technical report, Department of Mathematics, UCLA, 2013.
33. P. RICHTÁRIK AND M. TAKÁČ, Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function, Math. Program. Ser. A (2012).
34. P. RICHTÁRIK AND M. TAKÁČ, Parallel Coordinate Descent Methods for Big Data Optimization, Technical report, arXiv: 1212.0873, 2012.
35. A. SAHA AND A. TEWARI, On the nonasymptotic convergence of cyclic coordinate descent methods, SIAM J. Optim., 23 (2013), pp. 576-601.
36. C. SCHERRER, A. TEWARI, M. HALAPPANAVAR, AND D. HAGLIN, Feature clustering for accelerating parallel coordinate descent, in NIPS 2012, Adv. Neural Inf. Proces. Syst. 25, 2012, pp. 28-36.
37. S. SHALEV-SHWARTZ AND T. ZHANG, Accelerated mini-batch stochastic dual coordinate ascent, in NIPS 2013, Adv. Neural Inf. Process. Syst. 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, eds., 2013, pp. 378-385.
38. O. SHAMIR AND T. ZHANG, Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes, in Proceedings of the International Conference on Machine Learning, 2013.
39. S. SRIDHAR, V. BITTORF, J. LIU, C. ZHANG, C. RÉ, AND S. J. WRIGHT, An approximate efficient solver for LP rounding, in NIPS 2013, Adv. Neural Inf. Process. Syst. 26, 2013.
40. P. TSENG, Convergence of a block coordinate descent method for nondifferentiable minimization, J. Optim. Theory Appl., 109 (2001), pp. 475-494.
41. P. TSENG, On Accelerated Proximal Gradient Methods for Convex-Concave Optimization, Technical report, University of Washington, 2008.
42. P. TSENG AND S. YUN, A coordinate gradient descent method for nonsmooth separable minimization, Math. Program. Ser. B, 117 (2009), pp. 387-423.
43. P. TSENG AND S. YUN, A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training, Comput. Optim. Appl., 47 (2010), pp. 179-206.
44. P.-W. WANG AND C.-J. LIN, Iteration Complexity of Feasible Descent Methods for Convex Optimization, Technical report, Department of Computer Science, National Taiwan University, 2013.
45. S. J. WRIGHT, R. D. NOWAK, AND M. A. T. FIGUEIREDO, Sparse reconstruction by separable approximation, IEEE Trans. Signal Process., 57 (2009), pp. 2479-2493.