Accelerated and inexact forward-backward algorithms

SIAM Journal on Optimization

Volumn 23, Issue 3, 2013, Pages 1607-1633

  Villa, Silvia ^a   Salzo, Saverio ^b   Baldassarre, Luca ^c   Verri, Alessandro ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF GENOA   (Italy)

^c EPFL   (Switzerland)

Author keywords

Accelerated forward backward splitting; Convex optimization; Estimate sequences; Inexact proximity operator; Total variation

Indexed keywords

ESTIMATE SEQUENCES; EXPERIMENTAL ANALYSIS; FORWARD BACKWARD ALGORITHMS; FORWARD-BACKWARD SPLITTING; FORWARD-BACKWARD SPLITTING METHODS; GRADIENT DESCENT ALGORITHMS; PROXIMITY OPERATOR; TOTAL VARIATION;

CONVEX OPTIMIZATION; MACHINERY;

ALGORITHMS;

EID: 84886247153     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/110844805     Document Type: Article

References (67)

1. Y. I. Alber, R. S. Burachik, and A. N. Iusem, A proximal point method for nonsmooth convex optimization problems in Banach spaces, Abstr. Appl. Anal., 2 (1997), pp. 97-120.
2. A. Argyriou, C. A. Micchelli, M. Pontil, L. Shen, and Y. Xu, Efficient First Order Methods for Linear Composite Regularizers, preprint, arXiv:1104.1436v1, 2011.
3. A. Auslender, Numerical methods for nondifferentiable convex optimization, Nonlinear Analysis and Optimization, Math. Programming Stud., (1987), pp. 102-126.
4. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski, Optimization with sparsity-inducing penalties, Found. Trends Mach. Learn., 4 (2012), pp. 1-106.
5. H. H. Bauschke and P. L. Combettes, The Baillon-Haddad theorem revisited, J. Convex Anal., 17 (2010), pp. 781-787.
6. A. Beck and M. Teboulle, Fast gradient-based algorithms for constrained total variation image denoising and deblurring, IEEE Trans. Image Process., 18 (2009), pp. 2419-2434.
7. A. Beck and M. Teboulle, A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM J. Imaging Sci., 2 (2009), pp. 183-202.
8. S. Becker, J. Bobin, and E. Cand̀es, NESTA: A fast and accurate first-order method for sparse recovery, SIAM J. Imaging Sci., 4 (2011), pp. 1-39.
9. S. Bonettini and V. Ruggiero, On the convergence of primal-dual hybrid gradient algorithms for total variation image restoration, J. Math. Imaging Vision, 44 (2012), pp. 1-18.
10. K. Bredies, A forward-backward splitting algorithm for the minimization of non-smooth convex functionals in Banach space, Inverse Problems, 25 (2009), 015005.
11. R. S. Burachik and B. F. Svaiter, A relative error tolerance for a family of generalized proximal point methods, Math. Oper. Res., 26 (2001), pp. 816-831.
12. A. Chambolle, An algorithm for total variation minimization and applications, J. Math. Imaging Vision, 20 (2004), pp. 89-97.
13. A. Chambolle and P.-L. Lions, Image recovery via total variation minimization and related problems, Numer. Math., 76 (1997), pp. 167-188.
14. A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), pp. 120-145.
15. C. Chaux, J.-C. Pesquet, and N. Pustelnik, Nested iterative algorithms for convex constrained image recovery problems, SIAM J. Imaging Sci., 2 (2009), pp. 730-762.
16. P. L. Combettes, D. Dũng, and B. C. Vũ, Dualization of signal recovery problems, Set- Valued Var. Anal., 18 (2010), pp. 373-404.
17. P. L. Combettes and J.-C. Pesquet, Proximal splitting methods in signal processing, in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, H. H. Bauschke, R. Burachik, P. L. Combettes, V. Elser, D. R. Luke, and H. Wolkowicz, eds., Springer- Verlag, New York, 2011, pp. 185-212.
18. P. L. Combettes and V. R. Wajs, Signal recovery by proximal forward-backward splitting, Multiscale Model. Simul., 4 (2005), pp. 1168-1200.
19. R. Cominetti, Coupling the proximal point algorithm with approximation methods, J. Optim. Theory Appl., 95 (1997), pp. 581-600.
20. R. Correa and C. Lemarechal, Convergence of some algorithms of convex minimization, Math. Program., 62 (1993), pp. 261-275.
21. I. Daubechies, G. Teschke, and L. Vese, Iteratively solving linear inverse problems under general convex constraints, Inverse Problems Imaging, 1 (2007), pp. 29-46.
22. O. Devolder, F. Glineur, and Y. Nesterov, First-order methods of smooth convex optimization with inexact oracle, Math. Program., (2011), pp. 1-39.
23. J. Duchi and Y. Singer, Efficient online and batch learning using forward backward splitting, J. Mach. Learn. Res., 10 (2009), pp. 2899-2934.
24. J. Eckstein, Approximate iterations in Bregman-function-based proximal algorithms, Math. Program., 83 (1998), pp. 113-123.
25. B. Eicke, Iteration methods for convexly constrained ill-posed problems in Hilbert space, Numer. Funct. Anal. Optim., 13 (1992), pp. 413-429.
26. E. Esser, X. Zhang, and T. F. Chan, A general framework for a class of first order primaldual algorithms for convex optimization in imaging science, SIAM J. Imaging Sci., 3 (2010), pp. 1015-1046.
27. M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems, IEEE J. Sel. Top. Signal Process., 1 (2007), pp. 586-597.
28. M. Fornasier, ed., Theoretical Foundations and Numerical Methods for Sparse Recovery, Radon Ser. Comput. Appl. Math. 9, De Gruyter, Berlin, 2010.
29. O. Güler, On the convergence of the proximal point algorithm for convex minimization, SIAM J. Control Optim., 29 (1991), pp. 403-419.
30. O. Güler, New proximal point algorithms for convex minimization, SIAM J. Optim., 2 (1992), pp. 649-664.
31. B. He and X. Yuan, An accelerated inexact proximal point algorithm for convex minimization, J. Optim. Theory Appl., 154 (2012), pp. 536-548.
32. J.-B. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms. II, Grundlehren Math. Wiss. 306, Springer-Verlag, Berlin, 1993.
33. R. Jenatton, J.-Y. Audibert, and F. Bach, Structured variable selection with sparsityinducing norms, J. Mach. Learn. Res., 12 (2011), pp. 2777-2824.
34. B. Lemaire, About the convergence of the proximal method, in Advances in Optimization (Lambrecht, 1991), Lecture Notes in Econom. and Math. Systems 382, Springer, Berlin, 1992, pp. 39-51.
35. P. L. Lions and B. Mercier, Splitting algorithms for the sum of two nonlinear operators, SIAM J. Numer. Anal., 16 (1979), pp. 964-979.
36. B. Martinet, Régularisation d'inéquations variationnelles par approximations successives, Rev. Fraņcaise Inform. Rech. Oper., 4 (1970), pp. 154-158.
37. R. D. C. Monteiro and B. F. Svaiter, Convergence Rate of Inexact Proximal Point Methods with Relative Error Criteria for Convex Optimization, http://www.optimizationonline.org/DBHTML/2010/08/2714.html (2010).
38. R. Monteiro and B. Svaiter, An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods, SIAM J. Optim., 23 (2013), pp. 1092-1125.
39. J.-J. Moreau, Fonctions convexes duales et points proximaux dans un espace hilbertien, C. R. Acad. Sci. Paris Ser. I Math., 255 (1962), pp. 2897-2899.
40. J.-J. Moreau, Propriétés des applications "prox," C. R. Acad. Sci. Paris Ser. I Math., 256 (1963), pp. 1069-1071.
41. J.-J. Moreau, Proximité et dualité dans un espace hilbertien, Bull. Soc. Math. France, 93 (1965), pp. 273-299.
42. S. Mosci, L. Rosasco, M. Santoro, A. Verri, and S. Villa, Solving structured sparsity regularization with proximal methods, in Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Comput. Sci. 6322, J. Balcázar, F. Bonchi, A. Gionis, and M. Sebag, eds., Springer, Berlin, 2010, pp. 418-433.
43. A. S. Nemirovsky and D. B. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley-Intersci. Ser. Discrete Math., Wiley, New York, 1983.
44. Y. Nesterov, Introductory Lectures on Convex Optimization. A Basic Course, Appl. Optim. 87, Kluwer Academic, Boston, MA, 2004.
45. Y. Nesterov, Gradient Methods for Minimizing Composite Objective Function, Technical report, CORE Discussion Papers from Université Catholique de Louvain, Center for Operations Research and Econometrics No 2007/076, 2009.
46. G. Peyré and J. Fadili, Group sparsity with overlapping partition functions, in Proc. EUSIPCO 2011, Barcelona, 2011, pp. 303-307.
47. R. T. Rockafellar, Augmented Lagrangians and applications of the proximal point algorithm in convex programming, Math. Oper. Res., 1 (1976), pp. 97-116.
48. R. T. Rockafellar, Monotone operators and the proximal point algorithm, SIAM J. Control Optim., 14 (1976), pp. 877-898.
49. L. Rosasco, S. Mosci, M. S. Santoro, A. Verri, and S. Villa, A regularization approach to nonlinear variable selection, JMLR Workshop Conf. Proc., 9 (2010), pp. 653-660.
50. L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D, 60 (1992), pp. 259-268.
51. A. Sabharwal and L. C. Potter, Convexly constrained linear inverse problems: Iterative least-squares and regularization, IEEE Trans. Signal Process., 46 (1998), pp. 2345-2352.
52. S. Salzo and S. Villa, Inexact and accelerated proximal point algorithm, J. Convex Anal., 19 (2012).
53. O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, and F. Lenzen, Variational Methods in Imaging, Appl. Math. Sci. 167, Springer, New York, 2009.
54. M. Schmidt, N. Le Roux, and F. Bach, Convergence rates of inexact proximal-gradient methods for convex optimization, in Advances in Neural Information Processing Systems 24, 2011.
55. M. V. Solodov and B. F. Svaiter, A hybrid approximate extragradient-proximal point algorithm using the enlargement of a maximal monotone operator, Set-Valued Anal., 7 (1999), pp. 323-345.
56. M. V. Solodov and B. F. Svaiter, A comparison of rates of convergence of two inexact proximal point algorithms, in Nonlinear Optimization and Related Topics (Erice, 1998), Appl. Optim. 36, Kluwer Academic, Dordrecht, 2000, pp. 415-427.
57. M. V. Solodov and B. F. Svaiter, Error bounds for proximal point subproblems and associated inexact proximal point algorithms, Math. Program., 88 (2000), pp. 371-389.
58. M. V. Solodov and B. F. Svaiter, An inexact hybrid generalized proximal point algorithm and some new results on the theory of Bregman functions, Math. Oper. Res., 25 (2000), pp. 214-230.
59. M. V. Solodov and B. F. Svaiter, A unified framework for some inexact proximal point algorithms, Numer. Funct. Anal. Optim., 22 (2001), pp. 1013-1035.
60. A. Subramanian et al., Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles, Proc. Natl. Acad. Sci. USA, 102 (2005), p. 15545.
61. P. Tseng, Approximation accuracy, gradient methods, and error bound for structured convex optimization, Math. Program., 125 (2010), pp. 263-295.
62. M. J. Van De Vijver et al., A gene-expression signature as a predictor of survival in breast cancer, New England J. Med., 347 (2002), pp. 1999-2009.
63. Y. Yao and N. Shahzad, Strong convergence of a proximal point algorithm with general errors, Optim. Lett., 6 (2012), pp. 621-628.
64. M. Yuan and Y. Lin, Model selection and estimation in regression with grouped variables, J. R. Stat. Soc. Ser. B Stat. Method, 68 (2006), pp. 49-67.
65. C. Zǎlinescu, Convex Analysis in General Vector Spaces, World Scientific Publishing, River Edge, NJ, 2002.
66. A. J. Zaslavski, Convergence of a proximal point method in the presence of computational errors in Hilbert spaces, SIAM J. Optim., 20 (2010), pp. 2413-2421.
67. P. Zhao, G. Rocha, and B. Yu, The composite absolute penalties family for grouped and hierarchical variable selection, Ann. Statist., 37 (2009), pp. 3468-3497.