DiSCO: Distributed optimization for self-concordant empirical loss

32nd International Conference on Machine Learning, ICML 2015

Volumn 1, Issue , 2015, Pages 362-370

  Zhang, Yuchen ^a   Xiao, Lin ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; CONJUGATE GRADIENT METHOD; ITERATIVE METHODS; LEARNING SYSTEMS; NEWTON-RAPHSON METHOD; OPTIMIZATION; REGRESSION ANALYSIS;

BINARY CLASSIFICATION; COMMUNICATION EFFICIENCY; DISTRIBUTED OPTIMIZATION; EMPIRICAL RISK MINIMIZATION; LOGISTIC REGRESSIONS; PRECONDITIONED CONJUGATE GRADIENT METHOD; REGULARIZATION PARAMETERS; RIDGE REGRESSION;

LEARNING ALGORITHMS;

EID: 84969505806     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Agarwal, A. and Duchi, J. C. Distributed delayed stochastic optimization. In Advances in NIPS, pp. 873-881, 2011.
2. Bekkerman, R., Bilenko, M., and Langford, J. Scaling up Machine Learning: Parallel and Distributed Approaches. Cambridge University Press, 2011.
3. Bertsekas, D. P. and Tsitsiklis, J. N. Parallel and Distributed Computation: Numerical Methods. Prentice-Hall, 1989.
4. Bousquet, O. and Elisseeff, A. Stability and generalization. Journal of Machine Learning Research, 2: 499-526, 2002.
5. Boyd, S. and Vandenberghe, L. Convex optimization. Cambridge university press, 2004.
6. Boyd, S., Parikh, N., Chu, E., Peleato, B. and Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning. 3 (1): 1-122, 2010.
7. Chang, C.-C. and Lin, C.-J. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2 (3): 27, 2011.
8. Dean, J. and Ghemawat, S. MapReduce: Simplfied data processing on large clusters. Communications of the ACM, 51 (1): 107-113, 2008.
9. Defazio, A., Bach, F., and Lacoste-Julien, S. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in NIPS 27, pp. 1646-1654. 2014.
10. Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction using mini-batches. The Journal of Machine Learning Research, 13 (1): 165-202, 2012.
11. Deng, W. and Yin, W. On the global and linear convergence of the generalized alternating direction method of multipliers. CAAM Technical Report 12-14, Rice University, 2012.
12. Duchi, J. C, Agarwal, A., and Wainwright, M. J. Dual averaging for distributed optimization: convergence analysis and network scaling. IEEE Transactions on Automatic Control, 57 (3): 592-606, 2012.
13. Golub, G. H. and Van Loan, C. F. Matrix Computations. The John Hopkins University Press, Baltimore, MD, third edition, 1996.
14. Jaggi, M., Smith, V, Takac, M., Terhorst, J., Krishnan, S., Hofmann, T., and Jordan, M. I. Communication-efficient distributed dual coordinate ascent. In Advances in NIPS 27, pp. 3068-3076. 2014.
15. Le Roux, N, Schmidt, M., and Bach, F. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances in NIPS 25, pp. 2672-2680. 2012.
16. Lin, C.-Y, Tsai, C.-H., Lee, C.-R, and Lin, C.-J. Large-scale logistic regression and linear support vector machines using Spark. In Proceedings of the IEEE Conference on Big Data, Washington DC, USA, 2014.
17. Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer, Boston, 2004.
18. Nesterov, Y. Gradient methods for minimizing composite functions. Mathematical Programming, Ser. B, 140: 125-161, 2013.
19. Nesterov, Y. and Nemirovski, A. Interior Point Polynomial Time Methods in Convex Programming. SIAM, Philadelphia, 1994.
20. Nocedal, J. and Wright, S. J. Numerical Optimization. Springer, New York, 2nd edition, 2006.
21. Recht, B., Re, C, Wright, S. J., and Niu, F. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances inNIPS, pp. 693-701, 2011.
22. Shalev-Shwartz, S., Shamir, O., Srebro, N. and Sridharan, K. Stochastic convex optimization. In Proceedings of the 22nd Annual Conference on Learning Theory, 2009.
23. Shalf, J., Dosanjh, S., and Morrison, J. Exascale computing technology challenges. In Proceedings of the 9th International Conference on High Performance Computing for Computational Science, VECPAR'10, pp. 1-25. Springer-Verlag. 2011.
24. Shamir, O., Srebro, N., and Zhang, T. Communication efficient distributed optimization using an approximate Newton-type method. In Proceedings of ICML. JMLR: W&CP volume 32, 2014.
25. Zhang, Y., Wainwright, M. J., and Duchi, J. C. Communication-efficient algorithms for statistical optimization. In Advances in NIPS, pp. 1502-1510, 2012.
26. Zhang, Y. and Xiao, L. Communication-efficient distributed optimization of self-concordant empirical loss. arXiv preprint arXiv: 1501.00263, 2015.
27. Zhuang, Y, Chin, W.-S., Juan, Y.-C, and Lin, C.-J. Distributed newton method for regularized logistic regression. Technical report, Department of Computer Science, National Taiwan University, 2014.