Iteration complexity of feasible descent methods for onvex optimization

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1523-1548

  Wang, Po Wei ^a   Lin, Chih Jen ^a  

^a NATIONAL TAIWAN UNIVERSITY   (Taiwan)

Author keywords

Convergence rate; Convex optimization; Feasible descent methods; Iteration complexity

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; CONVEX OPTIMIZATION; OPTIMIZATION;

CONVERGENCE RATES; CYCLIC COORDINATE DESCENTS; DESCENT METHOD; ITERATION COMPLEXITY; MACHINE LEARNING PROBLEM; OBJECTIVE FUNCTIONS; OPTIMIZATION ALGORITHMS; SUPPORT VECTOR CLASSIFICATION AND REGRESSIONS;

ITERATIVE METHODS;

EID: 84901632905     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (31)

1. Amir Beck and Luba Tetruashvili. On the convergence of block coordinate descent type methods. SIAM Journal on Optimization, 23(4):2037-2060, 2013.
2. Bernhard E. Boser, Isabelle Guyon, and Vladimir Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pages 144-152. ACM Press, 1992. (Pubitemid 23615470)
3. Kai-Wei Chang, Cho-Jui Hsieh, and Chih-Jen Lin. Coordinate descent method for largescale L2-loss linear SVM. Journal of Machine Learning Research, 9:1369-1398, 2008. URL http://www.csie.ntu.edu.tw/~cjlin/papers/cdl2.pdf.
4. Eli M. Gafni and Dimitri P. Bertsekas. Two-metric projection methods for constrained optimization. SIAM Journal on Control and Optimization, 22:936-964
5. G. H. Golub and C. F. Van Loan. Matrix Computations. The Johns Hopkins University Press, third edition, 1996.
6. Clifiord Hildreth. A quadratic programming procedure. Naval Research Logistics Quarterly, 4:79-85, 1957.
7. Jean-Baptiste Hiriart-Urruty and Claude Lemarfiechal. Fundamentals of Convex Analysis. Springer Verlag, 2001.
8. Chia-Hua Ho and Chih-Jen Lin. Large-scale linear support vector regression. Journal of Machine Learning Research, 13:3323-3348, 2012. URL http://www.csie.ntu.edu.tw/ ~cjlin/papers/linear-svr.pdf.
9. Alan J Hofiman. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards, 49(4):263-265, 1952.
10. Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S. Sathiya Keerthi, and Sellamanickam Sundararajan. A dual coordinate descent method for large-scale linear SVM. In Proceedings of the Twenty Fifth International Conference on Machine Learning (ICML), 2008. URL http://www.csie.ntu.edu.tw/~cjlin/papers/ cddual.pdf.
11. Wu Li. Sharp Lipschitz constants for basic optimal solutions and basic feasible solutions of linear programs. SIAM Journal on Control and Optimization, 32(1):140-153, 1994.
12. Chih-Jen Lin, Ruby C. Weng, and S. Sathiya Keerthi. Trust region Newton method for large-scale logistic regression. Journal of Machine Learning Research, 9:627-650, 2008. URL http://www.csie.ntu.edu.tw/~cjlin/papers/ logistic.pdf.
13. Zhi-Quan Luo and Paul Tseng. On the convergence of coordinate descent method for convex difierentiable minimization. Journal of Optimization Theory and Applications, 72 (1):7-35, 1992a.
14. Zhi-Quan Luo and Paul Tseng. On the linear convergence of descent methods for convex essentially smooth minimization. SIAM Journal on Control and Optimization, 30(2): 408-425, 1992b.
15. Zhi-Quan Luo and Paul Tseng. Error bounds and convergence analysis of feasible descent methods: a general approach. Annals of Operations Research, 46:157-178, 1993.
16. Olvi L. Mangasarian and Tzong-Huei Shiau. Lipschitz continuity of solutions of linear inequalities, programs and complementarity problems. SIAM Journal on Control and Optimization, 25(3):583-595, 1987. (Pubitemid 17585883)
17. Yurii E. Nesterov. Eficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-362, 2012.
18. Jong-Shi Pang. A posteriori error bounds for the linearly-constrained variational inequality problem. Mathematics of Operations Research, 12(3):474-484, 1987. 1547
19. John C. Platt. Fast training of support vector machines using sequential minimal optimization. In Bernhard Sch, olkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods -Support Vector Learning, Cambridge, MA, 1998. MIT Press.
20. Jason D. M. Rennie. Regularized logistic regression is strictly convex. Technical report, MIT, 2005. URL http://qwone.com/~jason/writing/convexLR.pdf.
21. Peter Richtfiarik and Martin Takfiafic. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. 2011. URL http://link.springer. com/article/10.1007/s10107-012-0614-z. To appear in Mathematical Programming.
22. Ankan Saha and Ambuj Tewari. On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization, 23(1):576-601, 2013.
23. Ludwig Seidel. Ueber ein verfahren, die gleichungen, auf welche die methode der kleinsten quadrate f, uhrt, sowie line, are gleichungen , uberhaupt, durch successive ann, aherung aufzul, osen. Abhandlungen der Bayerischen Akademie der Wissenschaften. Mathematisch-Naturwissenschaftliche Abteilung, 11(3):81-108, 1874.
24. Shai Shalev-Shwartz and Ambuj Tewari. Stochastic methods for l1 regularized loss minimization. In Proceedings of the Twenty Sixth International Conference on Machine Learn-ing (ICML), 2009.
25. Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14:567-599, 2013a.
26. Shai Shalev-Shwartz and Tong Zhang. Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization, 2013b. arXiv preprint arXiv:1309.2375.
27. Rachael Tappenden, Peter Richtfiarik, and Jacek Gondzio. Inexact coordinate descent: complexity and preconditioning, 2013. arXiv preprint arXiv:1304.5530.
28. Paul Tseng and Sangwoon Yun. Block-coordinate gradient descent method for linearly constrained nonsmooth separable optimization. Journal of Optimization Theory and Applications, 140:513-535, 2009.
29. Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, New York, NY, 1995.
30. Jean-Philippe Vial. Strong and weak convexity of sets and functions. Mathematics of Operations Research, 8(2):231-259, 1983.
31. Stephen J Wright. Accelerated block-coordinate relaxation for regularized optimization. SIAM Journal on Optimization, 22(1):159-186, 2012.