An asynchronous parallel stochastic coordinate descent algorithm

31st International Conference on Machine Learning, ICML 2014

Volumn 2, Issue , 2014, Pages 1840-1848

  Liu, Ji ^a   Wright, Stephen J ^a   Ré, Christopher ^b   Bittorf, Victor ^a   Sridharf, Srikrishna ^a  

^a University of Wisconsin   (United States)

^b Stanford University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; CONSTRAINED OPTIMIZATION; FUNCTIONS; LEARNING SYSTEMS; OPTIMIZATION; PARALLEL PROCESSING SYSTEMS;

ASYNCHRONOUS PARALLEL; CONSTRAINED FUNCTION; CONVEX FUNCTIONS; COORDINATE DESCENT; LINEAR SPEED-UP; MULTI-CORE SYSTEMS; STRONG CONVEXITIES; UNCONSTRAINED OPTIMIZATION;

STOCHASTIC SYSTEMS;

EID: 84919932688     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Agarwal, A. and Duchi, J. C. Distributed delayed stochastic optimization. CDC, pp. 5451-5452, 2012.
2. Avron, H., Druinsky, A., and Gupta, A. Revisiting asynchronous linear solvers: Provable convergence rate through randomization. IPDPS, 2014.
3. Beck, A. and Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sciences., 2(1): 183-202, 2009.
4. Beck, A. and Tetruashvili, L. On the convergence of block coordinate descent type methods, 2013. To appear in SIAM Journal on Optimization.
5. Bertsekas, D. P. and Tsitsiklis, J. N. Parallel and Distributed Computation: Numerical Methods, Pentice Hall, 1989.
6. Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1): 1-122, 2011.
7. Chang, C.-C. and Lin, C.-J. LIBSVM: A library for support vector machines, 2011. URL http://www.csie.ntu.edu.tw/~cjlin/libsvm/.
8. Cortes, C. and Vapnik, V. Support vector networks. Machine Learning, pp. 273-297, 1995.
9. Cotter, A., Shamir, O., Srebro, N., and Sridharan, K. Better mini-batch algorithms via accelerated gradient methods. arXiv: 1106.4574, 2011.
10. Duchi, J. C., Agarwal, A., and Wainwright, M. J. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic Control, 57 (3):592-606, 2012.
11. Ferris, M. C. and Mangasarian, O. L. Parallel variable distribution. SIAM Journal on Optimization, 4(4):815-832, 1994 .
12. Goldfarb, D. and Ma, S. Fast multiple-splitting algorithms for convex optimization. SIAM Journal on Optimization, 22(2): 533-556, 2012.
13. Liu, J., Wright, S. J., Re, C., Bittorf, V., and Sridhar, S. An asynchronous parallel stochastic coordinate descent algorithm. Arxiv: 1311.1873, 2013.
14. Lu, Z. and Xiao, L. On the complexity analysis of randomized block-coordinate descent methods. TechReport, 2013.
15. Luo, Z. Q. and Tseng, P. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72:7-35, 1992.
16. Mangasarian, O. L. Parallel gradient distribution in unconstrained optimization. SIAM Journal on Optimization, 33(1):916-1925, 1995.
17. Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19:1574-1609, 2009.
18. Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, 2004.
19. Nesterov, Y. Efficiency of coordinate descent methods on huge- scale optimization problems. SIAM Journal on Optimization, 22(2):341-362, 2012.
20. Niu, F., Recht, B., Ré, C., and Wright, S. J. Hogwild!: A lock- free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems 24, pp. 693- 701, 2011.
21. Peng, Z., Yan, M., and Yin, W. Parallel and distributed sparse optimization, preprint, 2013.
22. Richtárik, P. and Takáč, M. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. arXiv: 1107.2848, 2011.
23. Richtárik, P. and Takáč, M. Parallel coordinate descent methods for big data optimization. ArXiv: 1212.0873, 2012.
24. Scherrer, C., Tewari, A., Halappanavar, M., and Haglin, D. Feature clustering for accelerating parallel coordinate descent. NIPS, pp. 28-36, 2012.
25. Shalev-Shwartz, S. and Zhang, T. Accelerated mini-batch stochastic dual coordinate ascent. Arxiv: 1305.2581, 2013.
26. Shamir, O. and Zhang, T. Stochastic gradient descent for non- smooth optimization: Convergence results and optimal averaging schemes. ICML, 2013.
27. Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267-288, 1996.
28. Tseng, P. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications, 109:475-494, 2001.
29. Tseng, P. and Yun, S. A coordinate gradient descent method for nonsmooth separable minimization. Mathematical Programming, Series B, 117:387-423, June 2009.
30. Tseng, P. and Yun, S. A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training. Computational Optimization and Applications, 47(2): 179-206, 2010.
31. Wang, P.-W. and Lin, C.-J. Iteration complexity of feasible descent methods for convex optimization. Technical report, 2013.
32. Wright, S. J. Accelerated block-coordinate relaxation for regularized optimization. SIAM Journal on Optimization, 22(1): 159-186, 2012.
33. Yang, T. Trading computation for communication: Distributed stochastic dual coordinate ascent. NIPS, pp. 629-637, 2013.
34. Zhang, C. and Ré, C. Dimmwitted: A study of main-memory statistical analytics. ArXiv: 1403.7550, 2014.