Communication-efficient distributed dual coordinate ascent

Advances in Neural Information Processing Systems

Volumn 4, Issue January, 2014, Pages 3068-3076

  Jaggi, Martin ^a   Smith, Virginia ^b   Takáč, Martin ^c   Terhorst, Jonathan ^b   Krishnan, Sanjay ^b   Hofmann, Thomas ^a   Jordan, Michael I ^b  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c LEHIGH UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; INFORMATION SCIENCE; LEARNING SYSTEMS; OPTIMIZATION;

COORDINATE ASCENT; DISTRIBUTED OPTIMIZATION; LARGE-SCALE MACHINE LEARNING; LOCAL COMPUTATION; PRIMAL-DUAL; SOLUTION QUALITY; STATE OF THE ART; STRONG CONVERGENCE;

ALGORITHMS;

EID: 84937863713     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. NSDI, 2012.
2. Tianbao Yang. Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent. NIPS, 2013.
3. Tianbao Yang, Shenghuo Zhu, Rong Jin, and Yuanqing Lin. On Theoretical Analysis of Distributed Stochastic Dual Coordinate Ascent. arXiv: 1312.1031, December 2013.
4. Martin Takáč, Avleen Bijral, Peter Richtárik, and Nathan Srebro. Mini-Batch Primal and Dual Methods for SVMs. ICML, 2013.
5. Feng Niu, Benjamin Recht, Christopher Ré, and Stephen J Wright. Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. NIPS, 2011.
6. Martin A Zinkevich, Markus Weimer, Alex J Smola, and Lihong Li. Parallelized Stochastic Gradient Descent. NIPS 23, 2010.
7. Joseph K Bradley, Aapo Kyrola, Danny Bickson, and Carlos Guestrin. Parallel Coordinate Descent for L1-Regularized Loss Minimization. ICML, 2011.
8. Jakub Mareček, Peter Richtárik, and Martin Takáč. Distributed Block Coordinate Descent for Minimizing Partially Separable Functions. arXiv: 1408.2467, June 2014.
9. Ion Necoara and Dragos Clipici. Efficient parallel coordinate descent algorithm for convex optimization problems with separable constraints: Application to distributed MPC. Journal of Process Control, 23(3): 243-253, 2013.
10. Martin Takáč, Peter Richtárik, and Nathan Srebro. Primal-Dual Parallel Coordinate Descent for Machine Learning Optimization. Manuscript, 2014.
11. Hsiang-Fu Yu, Cho-Jui Hsieh, Kai-Wei Chang, and Chih-Jen Lin. Large linear classification when data cannot fit in memory. the 16th ACM SIGKDD international conference, page 833, 2010.
12. Hsiang-Fu Yu, Cho-Jui Hsieh, Kai-Wei Chang, and Chih-Jen Lin. Large Linear Classification When Data Cannot Fit in Memory. ACM Transactions on Knowledge Discovery from Data, 5(4): 1-23, 2012.
13. Shai Shalev-Shwartz and Tong Zhang. Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization. JMLR, 14: 567-599, 2013.
14. Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3): 400-407, 1951.
15. Léon Bottou. Large-Scale Machine Learning with Stochastic Gradient Descent. COMPSTAT'2010 - Proceedings of the 19th International Conference on Computational Statistics, pages 177-187, 2010.
16. Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Primal Estimated Sub-Gradient Solver for SVM. Mathematical Programming, 127(1): 3-30, 2010.
17. Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S Sathiya Keerthi, and S Sundararajan. A Dual Coordinate Descent Method for Large-scale Linear SVM. ICML, 2008.
18. Peter Richtárik and Martin Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Mathematical Programming, 144(1-2): 1-38, April 2014.
19. Peter Richtárik and Martin Takáč. Parallel Coordinate Descent Methods for Big Data Optimization. arXiv: 1212.0873, 2012.
20. Peter Richtárik and Martin Takáč. Distributed Coordinate Descent Method for Learning with Big Data. arXiv: 1310.2059, 2013.
21. Olivier Fercoq, Zheng Qu, Peter Richtárik, and Martin Takáč. Fast Distributed Coordinate Descent for Non-Strongly Convex Losses. IEEE Workshop on Machine Learning for Signal Processing, May 2014.
22. Yuchen Zhang, John C Duchi, and Martin J Wainwright. Communication-Efficient Algorithms for Statistical Optimization. JMLR, 14: 3321-3363, November 2013.
23. Gideon Mann, Ryan McDonald, Mehryar Mohri, Nathan Silberman, and Daniel D Walker. Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models. NIPS, 1231-1239, 2009.
24. Ohad Shamir, Nathan Srebro, and Tong Zhang. Communication-Efficient Distributed Optimization using an Approximate Newton-type Method. ICML, 32(1): 1000-1008, 2014.
25. John N Tsitsiklis, Dimitri P Bertsekas, and Michael Athans. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Trans. on Automatic Control, 31(9): 803-812, 1986.
26. Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Prediction Using Mini-Batches. JMLR, 13: 165-202, 2012.
27. Alekh Agarwal and John C Duchi. Distributed Delayed Stochastic Optimization. NIPS, 873-881, 2011.
28. Ji Liu, Stephen J Wright, Christopher Ré, Victor Bittorf, and Srikrishna Sridhar. An Asynchronous Parallel Stochastic Coordinate Descent Algorithm. ICML, 2014.
29. John C Duchi, Michael I Jordan, and H Brendan McMahan. Estimation, Optimization, and Parallelism when Data is Sparse. NIPS, 2013.
30. Maria-Florina Balcan, Avrim Blum, Shai Fine, and Yishay Mansour. Distributed Learning, Communication Complexity and Privacy. COLT, 23: 26.1-26.22, 2012.