Adding vs. averaging in distributed primal-dual optimization

32nd International Conference on Machine Learning, ICML 2015

Volumn 3, Issue , 2015, Pages 1973-1982

  Ma, Chenxin ^a   Smith, Virginia ^b   Jaggi, Martin ^c   Jordan, Michael I ^b   Richtárik, Peter ^d   Takáč, Martin ^a  

^a Lehigh University   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c ETH ZURICH   (Switzerland)

^d UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84970028927     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Balcan, M.-F., Blum, A., Fine, S., and Mansour, Y. Distributed Learning, Communication Complexity and Privacy. In COLT, pp. 26.1-26.22, 2012.
2. Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1): 1-122, 2011.
3. Duchi, J. C., Jordan, M. I., and McMahan, H. B. Estimation, Optimization, and Parallelism when Data is Sparse. In NIPS, 2013.
4. Fercoq, O. and Richtárik, P. Accelerated, parallel and proximal coordinate descent. arXiv:1312.5799, 2013.
5. Fercoq, O., Qu, Z., Richtárik, P., and Takáč, M. Fast distributed coordinate descent for non-strongly convex losses. IEEE Workshop on Machine Learning for Signal Processing, 2014.
6. Forero, P. A., Cano, A., and Giannakis, G. B. Consensus-Based Distributed Support Vector Machines. JMLR, 11: 1663-1707, 2010.
7. Jaggi, M., Smith, V., Takáč, M., Terhorst, J., Krishnan, S., Hofmann, T., and Jordan, M. I. Communication-efficient distributed dual coordinate ascent. In NIPS, 2014.
8. Lee, C.-P. and Roth, D. Distributed Box-Constrained Quadratic Optimization for Dual Linear SVM. In ICML, 2015.
9. Liu, J. and Wright, S. J. Asynchronous stochastic coordinate descent: Parallelism and convergence properties. arXiv: 1403.3862, 2014.
10. Liu, J., Wright, S. J., Ré, C., Bittorf, V., and Sridhar, S. An Asynchronous Parallel Stochastic Coordinate Descent Algorithm. In ICML, 2014.
11. Lu, Z. and Xiao, L. On the complexity analysis of randomized block-coordinate descent methods. arXiv preprint arXiv: 1305.4723, 2013.
12. Mann, G., McDonald, R., Mohri, M., Silberman, N., and Walker, D. D. Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models. NIPS, 2009.
13. Mareček, J., Richtárik, P., and Takáč, M. Distributed block coordinate descent for minimizing partially separable functions. arXiv: 1406.0238, 2014.
14. McWilliams, B., Heinze, C., Meinshausen, N., Krummenacher, G., and Vanchinathan, H. P. LOCO: Distributing Ridge Regression with Random Projections. arXiv stat.ML, June 2014.
15. Niu, F., Recht, B., Ré, C., and Wright, S. J. Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. In NIPS, 2011.
16. Pechyony, D., Shen, L., and Jones, R. Solving Large Scale Linear SVM with Distributed Block Minimization. In NIPS Workshop on Big Learning, 2011.
17. Qu, Z. and Richtárik, P. Coordinate descent with arbitrary sampling I: Algorithms and complexity. arXiv:1412.8060, 2014.
18. Qu, Z., Richtárik, P., and Zhang, T. Randomized dual coordinate ascent with arbitrary sampling. arXiv:1411.5873, 2014.
19. Richtárik, P. and Tákač, M. Distributed coordinate descent method for learning with big data. arXiv preprint arXiv: 1310.2059, 2013.
20. RichtǍrik, P. and TakǍč, M. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Mathematical Programming, 144(1-2): 1-38, April 2014.
21. RichtǍrik, P. and TakǍč, M. Parallel coordinate descent methods for big data optimization. Mathematical Programming, pp. 1-52, 2015.
22. Shalev-Shwartz, S. and Zhang, T. Accelerated mini-batch stochastic dual coordinate ascent. In NIPS, 2013a.
23. Shalev-Shwartz, S. and Zhang, T. Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. arXiv: 1309.2375, 2013b.
24. Shalev-Shwartz, S. and Zhang, T. Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization. JMLR, 14:567-599, 2013c.
25. Shamir, O. and Srebro, N. Distributed Stochastic Optimization and Learning. In Allerton, 2014.
26. Shamir, O., Srebro, N., and Zhang, T. Communication efficient distributed optimization using an approximate newton-type method. In ICML, 2014.
27. Tappenden, R., Takáč, M., and Richtárik, P. On the complexity of parallel coordinate descent. Technical report, 2015. ERGO 15-001, University of Edinburgh.
28. Yang, T. Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent. In NIPS, 2013.
29. Yang, T., Zhu, S., Jin, R., and Lin, Y. On Theoretical Analysis of Distributed Stochastic Dual Coordinate Ascent. arXiv.1312.1031, 2013.
30. Yu, H.-F., Hsieh, C.-J., Chang, K.-W., and Lin, C.-J. Large Linear Classification When Data Cannot Fit in Memory. TKDD, 5(4): 1-23, 2012.
31. Zaharia, M., Chowdhury, M., Das, T., Dave, A., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In NSDI, 2012.
32. Zhang, Y. and Lin, X. DiSCO: Distributed Optimization for Self-Concordant Empirical Loss. In ICML, pp. 362-370, 2015.
33. Zhang, Y., Duchi, J. C., and Wainwright, M. J. Communication-Efficient Algorithms for Statistical Optimization. JMLR, 14:3321-3363, 2013.
34. Zinkevich, M. A., Weimer, M., Smola, A. J., and Li, L. Parallelized Stochastic Gradient Descent. NIPS, 2010.