Calculus of the Exponent of Kurdyka–Łojasiewicz Inequality and Its Applications to Linear Convergence of First-Order Methods

Foundations of Computational Mathematics

Volumn 18, Issue 5, 2018, Pages 1199-1232

  Li, Guoyin ^a   Pong, Ting Kei ^b  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

Author keywords

Convergence rate; First order methods; Kurdyka ojasiewicz inequality; Linear convergence; Luo Tseng error bound; Sparse optimization

Indexed keywords

CALCULATIONS; ERRORS; OPTIMIZATION;

CONVERGENCE RATES; ERROR BOUND; FIRST ORDER METHOD; LINEAR CONVERGENCE; SPARSE OPTIMIZATIONS;

NUMERICAL ANALYSIS;

EID: 85027109955     PISSN: 16153375     EISSN: 16153383     Source Type: Journal    
DOI: 10.1007/s10208-017-9366-8     Document Type: Article

References (47)

1. B. P. W. Ames and M. Hong, Alternating direction method of multipliers for sparse zero-variance discriminant analysis and principal component analysis, Comput. Optim. Appl. 64 (2016), 725–754
2. H. Attouch, J. Bolte, P. Redont, and A. Soubeyran, Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Lojasiewicz inequality, Math. Oper. Res. 35 (2010), 438–457
3. H. Attouch, J. Bolte, and B. F. Svaiter, Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods, Math. Program. 137 (2013), 91–129
4. H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Springer, New York, 2010
5. H. H. Bauschke and J. M. Borwein, On projection algorithms for solving convex feasibility problems, SIAM Rev. 38 (1996), 367–426
6. T. Blumensath and M. Davies, Iterative thresholding for sparse approximations, J. Fourier Anal. Appl. 14 (2008), 629–654
7. J. Bolte, T. P. Nguyen, J. Peypouquet, and B. W. Suter, From error bounds to the complexity of first-order descent methods for convex functions, Math. Program. DOI:10.1007/s10107-016-1091-6
8. J. Borwein and A. Lewis, Convex Analysis and Nonlinear Optimization, Springer, New York, 2006
9. R. I. Boţ and E. R. Csetnek, An inertial Tseng’s type proximal algorithm for nonsmooth and nonconvex optimization problems, J. Optim. Theory Appl. 171 (2016), 600–616
10. A. Chambolle and Ch. Dossal, On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”, J. Optim. Theory Appl. 166 (2015), 968–982
11. A. Daniilidis, W. Hare, and J. Malick, Geometrical interpretation of the predictor-corrector type algorithms in structured optimization problems, Optim. 55 (2006), 481–503
12. A. L. Dontchev and R. T. Rockafellar, Implicit Functions and Solution Mappings, Springer, New York, 2009
13. F. Fachinei and J.-S. Pang, Finite-Dimensional Variational Inequalities and Complementarity Problems. I and II, Springer, New York, 2003
14. J. Fan, Comments on “wavelets in statistics: a review” by A. Antoniadis, J. Ital. Stat. Soc. 6 (1997), 131–138
15. M. Forti, P. Nistri, and M. Quincampoix, Convergence of neural networks for programming problems via a nonsmooth Łojasiewicz inequality, IEEE Trans. Neural Netw. 17 (2006), 1471–1486
16. P. Frankel, G. Garrigos, and J. Peypouquet, Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates, J. Optim. Theory Appl. 165 (2015), 874–900
17. D. Geman and G. Reynolds, Constrained restoration and the recovery of discontinuities, IEEE Trans. Pattern Anal. Mach. Intell. 14 (1992), 367–383
18. W. L. Hare and A. S. Lewis, Identifying active constraints via partial smoothness and prox-regularity, J. Convex Anal. 11 (2004), 251–266
19. M. Hong, Z.-Q. Luo, and M. Razaviyayn, Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems, SIAM J. Optim. 26 (2016), 337–364
20. P. R. Johnstone and P. Moulin, Local and global convergence of an inertial version of forward-backward splitting, Preprint, 2017. Available at arXiv:1502.02281v5
21. A. Kyrillidis, S. Becker, V. Cevher, and C. Koch, Sparse projections onto the simplex, ICML (2013), 235–243
22. A. S. Lewis, Active sets, nonsmoothness, and sensitivity, SIAM J. Optim. 13 (2002), 702–725
23. G. Li, B. S. Mordukhovich, and T. S. Pham, New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors, Math. Program. 153 (2015), 333–362
24. G. Li and T. K. Pong, Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems, Math. Program. 159 (2016), 371–401
25. G. Li and T. K. Pong, Global convergence of splitting methods for nonconvex composite optimization, SIAM J. Optim. 25 (2015), 2434–2460
26. W. Li, Error bounds for piecewise convex quadratic programs and applications, SIAM J. Control Optim. 33 (1995), 1510–1529
27. H. Liu, W. Wu, and A. M.-C. So, Quadratic optimization with orthogonality constraints: explicit Łojasiewicz exponent and linear convergence of line-search methods, ICML (2016), 1158-1167
28. Z. Q. Luo and P. Tseng, On the linear convergence of descent methods for convex essentially smooth minimization, SIAM J. Control Optim. 30 (1992), 408–425
29. Z. Q. Luo and P. Tseng, Error bound and convergence analysis of matrix splitting algorithms for the affine variational inequality problem, SIAM J. Optim. 1 (1992), 43–54
30. Z. Q. Luo and P. Tseng, Error bounds and convergence analysis of feasible descent methods: A general approach, Ann. Oper. Res. 46 (1993), 157–178
31. Z. Q. Luo, J. S. Pang, and D. Ralph, Mathematical Programs with Equilibrium Constraints, Cambridge University Press, Cambridge, 1996
32. B. S. Mordukhovich and Y. Shao, On nonconvex subdifferential calculus in Banach spaces, J. Convex Anal. 2 (1995), 211–227
33. B. S. Mordukhovich, Variational Analysis and Generalized differentiation, I: Basic Theory, II: Applications, Springer, Berlin, 2006
34. M. Nikolova, M. K. Ng, S. Zhang, and W.-K. Ching, Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization, SIAM J. Imaging Sci. 1 (2008), 2–25
35. P. Ochs, Y. Chen, T. Brox, and T. Pock, iPiano: inertial proximal algorithm for non-convex optimization, SIAM J. Imaging Sci. 7 (2014), 1388–1419
36. R. T. Rockafellar, Convex Analysis, Princeton University Press, Princeton, 1970
37. S. M. Robinson, Some continuity properties of polyhedral multifunctions, in Mathematical Programming at Oberwolfach vol. 14 (H. König, B. Korte, and K. Ritter, eds), Springer Berlin Heidelberg, 1981, pp. 206–214
38. R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Springer, Berlin, 1998
39. 1 -regularized logistic regression, J. Mach. Learn. Res. 11 (2010), 713–741
40. P. Tseng, Approximation accuracy, gradient methods, and error bound for structured convex optimization, Math. Program 125 (2010), 263–295
41. P. Tseng and S. Yun, A coordinate gradient descent method for nonsmooth separable minimization, Math. Program. 117 (2009), 387–423
42. Y. Wang, Z. Luo, and X. Zhang, New improved penalty methods for sparse reconstruction based on difference of two norms, Preprint, 2015. Available at researchgate, DOI:10.13140/RG.2.1.3256.3369
43. Y. Xu and W. Yin, A block coordinate descent method for regularized multi-convex optimization with applications to nonnegative tensor factorization and completion, SIAM J. Imaging Sci. 6 (2013), 1758–1789
44. M. Yuan and Y. Lin, Model selection and estimation in regression with grouped variables, J. Royal Stat. Soc. B. 68 (2006), 49–67
45. C.-H. Zhang, Nearly unbiased variable selection under minimax concave penalty, Ann. Stat. 38 (2010), 894–942
46. Z. Zhou and A. M.-C. So, A unified approach to error bounds for structured convex optimization problems, Math. Program. DOI:10.1007/s10107-016-1100-9
47. p -norm regularization: error bounds and convergence rate analysis of first-order methods, ICML (2015), 1501–1510