From error bounds to the complexity of first-order descent methods for convex functions

Mathematical Programming

Volumn 165, Issue 2, 2017, Pages 471-507

  Bolte, Jérôme ^a,d   Nguyen, Trong Phong ^a   Peypouquet, Juan ^b   Suter, Bruce W ^c  

^a TOULOUSE SCHOOL OF ECONOMICS   (France)

^b DEPARTAMENTO DE FÍSICA   (Chile)

^c AIR FORCE RESEARCH LABORATORY   (United States)

^d YAMAGUCHI UNIVERSITY   (Japan)

Author keywords

Complexity of first order methods; Compressed sensing; Convex minimization; Error bounds; Forward backward method; KL inequality; LASSO

Indexed keywords

COMPRESSED SENSING; ERROR ANALYSIS; ERRORS; FUNCTIONS; NUMERICAL ANALYSIS;

CONVEX MINIMIZATION; ERROR BOUND; FIRST ORDER METHOD; FORWARD BACKWARD METHOD; KL INEQUALITY; LASSO;

ITERATIVE METHODS;

EID: 85000350010     PISSN: 00255610     EISSN: 14364646     Source Type: Journal    
DOI: 10.1007/s10107-016-1091-6     Document Type: Article

References (64)

1. Absil, P.-A., Mahony, R., Andrews, B.: Convergence of the iterates of descent methods for analytic cost functions. SIAM J. Optim. 16, 531–547 (2005)
2. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. Ser. B 116, 5–16 (2009)
3. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)
4. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. Ser. A 137(1–2), 91–129 (2013)
5. Auslender, A., Crouzeix, J.-P.: Global regularity theorems. Math. Oper. Res. 13, 243–253 (1988)
6. Auslender A.: Méthodes numériques pour la résolution des problèmes d’optimisation avec contraintes, PhD Thesis, Université Joseph Fourier Grenoble, France, (1969)
7. Bauschke, H.H., Borwein, J.M.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38(3), 367–426 (1996)
8. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)
9. Baillon, J.-B., Combettes, P.L., Cominetti, R.: There is no variational characterization of the cycles in the method of periodic projections. J. Funct. Anal. 262(1), 400–408 (2012)
10. Beck A., Shtern S.: Linearly Convergent Away-Step Conditional Gradient for Non-strongly Convex Functions, http://arxiv.org/abs/1504.05002
11. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2008)
12. Beck, A., Teboulle, M.: Convergence rate analysis and error bounds for projection algorithms in convex feasibility problem. Optim. Methods Softw. 18(4), 377–394 (2003)
13. Bégout, P., Bolte, J., Jendoubi, M.-A.: On damped second-order gradient systems. J. Differ. Equ. 259(7), 3115–3143 (2015)
14. Belousov, E.G., Klatte, D.: A Frank-Wolfe type theorem for convex polynomial programs. Comput. Optim. Appl. 22, 37–48 (2002)
15. Bochnak, J., Coste, M., Roy, M.-F.: Real Algebraic Geometry. Springer, Berlin (1998)
16. Bolte, J.: Sur quelques principes de convergence en Optimisation. Université Pierre et Marie Curie, Habilitation à diriger des recherches (2008)
17. Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2006)
18. Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. Trans. Amer. Math. Soc. 362(6), 3319–3363 (2010)
19. Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)
20. Bolte, J., Pauwels, E.: Majorization-minimization procedures and convergence of SQP methods for semi-algebraic and tame programs. Math. Oper. Res. 41(2), 442–465 (2016)
21. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. Ser. A 146, 1–16 (2013)
22. Bruck, R.E.: Asymptotic convergence of nonlinear contraction semigroups in Hilbert space. J. Funct. Anal. 18, 15–26 (1975)
23. Brézis H.: Opérateurs maximaux monotones et semi-groupes de contractions dans les espace Hilbert, North-Holland Mathematics studies 5, North-Holland Publishing Co., (1973)
24. Burke, J.V., Ferris, M.C.: Weak sharp minima in mathematical programming. SIAM J. Control Optim. 31, 1340–1359 (1993)
25. Candès, E.J., Wakin, M.B.: An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)
26. Combettes, P.L.: Inconsistent signal feasibility problems: least-squares solutions in a product space. IEEE Trans. Signal Process. 42(11), 2955–2966 (1994)
27. Combettes, P.L., Pesquet, J.-C.: Proximal Splitting Methods in Signal Processing. In: Bauschke, H.H., et al. (eds.) Fixed-Point Algorithms for Inverse Problems in Science and Engineering Springer Optimization and Its Applications, vol. 49, pp. 185–212 (2011)
28. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4, 1168–1200 (2005)
29. Cornejo, O., Jourani, A., Zǎlinescu, C.: Conditioning and upper-lipschitz inverse subdifferentials in nonsmooth optimization problems. J. Optim. Theory Appl. 95(1), 127–148 (1997)
30. Corvellec, J.-N., Motreanu, V.V.: Nonlinear error bounds for lower semicontinuous functions on metric spaces. Math. Program 114(2), 291–319 (2008)
31. Coste M.: An introduction to o-minimal geometry, RAAG Notes, 81 p., Institut de Recherche Mathématiques de Rennes, (1999)
32. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math. 57, 1413–1457 (2004)
33. Dedieu, J.-P.: Penalty functions in subanalytic optimization. Optimization 26, 27–32 (1992)
34. Drori, Y.: Contributions to the Complexity Analysis of Optimization Algorithms, PhD Thesis, Tel Aviv (2014)
35. Ferris, M.: Finite termination of the proximal point algorithm. Math. Program. 50, 359–366 (1991)
36. Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for KL functions. J. Optim. Theory Appl. 165(3), 874–900 (2015)
37. Hoffman, A.J.: On approximate solutions of systems of linear inequalities. J. Res. Natl. Bur. Stand. 49(4), 263–265 (1952)
38. Kantorovich, L.V., Akilov, G.P.: Functional Analysis in Normed Spaces. Pergamon, Oxford (1964)
39. Kurdyka, K.: On gradients of functions definable in o-minimal structures. Ann. Inst. Fourier 48, 769–783 (1998)
40. Li, G.: Global error bounds for piecewise convex polynomials. Math. Program. 137(1–2), 37–64 (2013)
41. Li, G., Mordukhovich, B.S., Nghia, T.T.A., Pham, T.S: Error Bounds for Parametric Polynomial Systems with Applications to Higher-Order Stability Analysis and Convergence Rates, ArXiv preprint arXiv:1509.03742, (2015)
42. Li, G., Mordukhovich B.S., Pham T.S.: New fractional error bound for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors, to appear in Math. Program., Ser. A.
43. Liang, J., Fadili, J., Peyré, G.: Local linear convergence of forward-backward under partial smoothness, http://arxiv.org/pdf/1407.5611
44. Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels, in: Les Équations aux Dérivées Partielles, pp. 87–89, Éditions du centre National de la Recherche Scientifique, Paris (1963).
45. Łojasiewicz, S.: Division d’une distribution par une fonction analytique de variables réelles. C. R. Acad. Sci., Paris 246, 683–686 (1958)
46. Łojasiewicz, S.: Sur la problème de la division. Studia Mathematica 18, 87–136 (1959)
47. Łojasiewicz, S.: Sur la géométrie semi- et sous-analytique. Ann. Inst. Fourier 43, 1575–1595 (1993)
48. Luo, X.D., Luo, Z.Q.: Extensions of Hoffman’s error bound to polynomial systems. SIAM J. Optim. 4, 383–392 (1994)
49. Luo, Z.-Q., Pang, J.S.: Error bounds for analytic systems and their application. Math. Program. 67, 1–28 (1994)
50. Luo, Z.-Q., Sturm, J.F.: Error bound for quadratic systems. Appl. Optim. 33, 383–404 (2000)
51. Luo, Z.-Q., Tseng, P.: Error bounds and convergence analysis of feasible descent methods: a general approach. Ann. Oper. Res. 46–47(1), 157–178 (1993)
52. Mangasarian, O.L.: A condition number for differentiable convex inequalities. Math. Oper. Res. 10, 175–179 (1985)
53. Mordukhovich, B.: Variational analysis and generalized differentiation. I. Basic theory, Grundlehren der Mathematischen Wissenschaften, 330, Springer, Berlin, xxii+579 pp (2006)
54. Nedić, A., Bertsekas, D.: Convergence rate of incremental subgradient algorithms. In: Uryasev, S., Pardalos, P.M. (eds.) Stochastic Optimization: Algorithms and Applications, pp. 263–304. Kluwer Academic Publishers, Dordrecht (2000)
55. Ng, K.F., Zheng, X.Y.: Global error bound with fractional exponents. Math. Program. 88, 357–370 (2000)
56. Pauwels, E.: The value function approach to convergence analysis in composite optimization. Oper Res Lett. arXiv preprint https://arxiv.org/abs/1604.01654 (to appear)
57. Pang, J.S.: Error bounds in mathematical programming. Math. Program. 79, 299–332 (1997)
58. Peypouquet, J.: Asymptotic convergence to the optimal value of diagonal proximal iterations in convex minimization. J. Convex Anal. 16(1), 277–286 (2009)
59. Peypouquet, J.: Convex Optimization in Normed Spaces: Theory, Methods and Examples. Springer, Cham (2015)
60. Robinson, S.M.: An application of error bounds for convex programming in a linear space. SIAM J. Control 13, 271–273 (1975)
61. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1972)
62. Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)
63. Vui, H.H.: Global Hölderian error bound for non degenerate polynomials. SIAM J. Optim. 23(2), 917–933 (2013)
64. Zǎlinescu, C.: Sharp estimates for Hoffmans constant for systems of linear inequalities and equalities. SIAM J. Optim. 14, 517–533 (2003)