Separable approximations and decomposition methods for the augmented Lagrangian

Optimization Methods and Software

Volumn 30, Issue 3, 2015, Pages 643-668

  Tappenden, Rachael ^a   Richtárik, Peter ^a   Büke, Burak ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

(block) coordinate descent methods; augmented Lagrangian; iteration complexity; separable approximations

Indexed keywords

APPROXIMATION THEORY; CONSTRAINED OPTIMIZATION; DECOMPOSITION; LAGRANGE MULTIPLIERS; LINEAR PROGRAMMING;

AUGMENTED LAGRANGIANS; COORDINATE DESCENT METHODS; DECOMPOSITION METHODS; DIAGONAL QUADRATIC APPROXIMATIONS; ITERATION COMPLEXITY; PARALLEL COORDINATES; PARTIAL SEPARABILITY; SEPARABLE APPROXIMATION;

ITERATIVE METHODS;

EID: 84933677250     PISSN: 10556788     EISSN: 10294937     Source Type: Journal    
DOI: 10.1080/10556788.2014.966824     Document Type: Article

References (45)

1. A.J. Berger, J.M. Mulvey, and A. Ruszczyński, An extension of the DQA algorithm to convex stochastic programs, SIAM J. Optim. 4(4) (1994), pp. 735-753.
2. D.P. Bertsekas, Extended monotropic programming and duality, Extended Monotropic Program Duality 139(2) (2008), pp. 209-225.
3. D. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods, Athena Scientific, Belmont, MA, 1996.
4. E.G. Birgin and J.M. Martínez, Improving ultimate convergence of an augmented Lagrangian method, Optim. Methods Softw. 23(2) (2011), pp. 177-195.
5. M. Blondel, K. Seki, and K. Uehara, Block coordinate descent algorithms for large-scale sparse multiclass classification, Mach. Learn. 93(1) (2013), pp. 31-52.
6. J.K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin, Parallel coordinate descent for L1-regularized loss minimization, in 28th International Conference on Machine Learning (ICML-11), L. Getoor and T. Scheffer, eds., ACM, June 2011, pp. 321-328.
7. J. Eckstein, Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results. Technical report, December 2012. RUTCOR Research Report RRR 32-2012.
8. J. Eckstein and D.P. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators, Math. Program. 55 (1992), pp. 293-318.
9. O. Fercoq, Parallel coordinate descent for the Adaboost problem, Technical report, University of Edinburgh, July 2013. arXiv:1310.1840v1 [cs.LG].
10. O. Fercoq and P. Richtárik, Smoothed parallel coordinate descent method, Technical report, University of Edinburgh, 2013.
11. A. Hamdi and A. Griewank, Properties of an augmented Lagrangian for design optimization, Optim. Methods Softw. 25(4) (2009), pp. 645-664.
12. B.S. He, M. Tao, and X.M. Yuan, Alternating direction method with Gaussian back substitution for separable convex programming, SIAM J. Optim. 22(2) (2012), pp. 313-340.
13. M.R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl. 4 (1969), pp. 303-320.
14. M. Hong and Z.Q. Luo, On the linear convergence of the alternating direction method of multipliers. Technical report, August 2012. arXiv:1208.3922.
15. C.-J. Hsieh, K.-W. Chang, C.-J. Lin, S. Sathiya Keerthi, and S. Sundararajan, A dual coordinate descent method for large-scale linear SVM. In ICML 2008, ACM, New York, 2008, pp. 408-415.
16. Y.T. Lee and A. Sidford, Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. Technical report, 2013. arXiv:1305:1922v1.
17. 1 minimization with application to compressed sensing; a greedy algorithm, Inverse Probl. Imag. 3 (2009), pp. 487-503.
18. Z. Lu and L. Xiao, On the complexity analysis of randomized block-coordinate descent methods. Technical report, May 2013. arXiv:1305.4723.
19. Z. Lu and L. Xiao, Randomized block coordinate non-monotone gradient method for a class of nonlinear programming. Technical report, June 2013. arXiv:1306.5918.
20. J.M. Mulvey and A. Ruszczyński, A diagonal quadratic approximation method for large scale linear programs, Oper. Res. Lett. 12 (1992), pp. 205-215.
21. J.M. Mulvey and A. Ruszczyński, A new scenario decomposition method for large scale stochastic optimization, Oper. Res. 43(3) (1995), pp. 477-490.
22. I. Necoara, Y. Nesterov, and F. Glineur, Efficiency of randomized coordinate descent methods on optimization problems with linearly coupled constraints. Technical report, June 2012.
23. I. Necoarə and A. Patrascu, A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints, Comput. Optim. Appl. 57(2) (2014), pp. 307-337.
24. Y. Nesterov, Efficiency of coordinate descent methods on huge-scale optimization problems, SIAM J. Optim. 22(2) (2012), pp. 341-362.
25. A. Patrascu and I. Necoara, Efficient random coordinate descent algorithms for large-scale structured nonconvex optimization. Technical report, University Politehnica Bucharest, May 2013.
26. M.J.D. Powell, A method for nonlinear constraints in minimization problems, in Optimization, R. Fletcher, ed., Academic Press, New York, 1972, pp. 283-298.
27. Z.T. Qin, K. Scheinberg and D. Goldfarb, Efficient block-coordinate descent algorithms for the group lasso, Math. Program. Comput. 5(2) (2013), pp. 143-169.
28. P. Richtárik and M. Takáč, Efficient serial and parallel coordinate descent methods for huge-scale truss topology design, in Operations Research Proceedings 2011, D. Klatte, H.-J. Lüthi, and K. Schmedders, (eds.), Springer, Berlin, 2012, pp. 27-32.
29. P. Richtárik and M. Takáč, Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function, Math. Program. Ser. A, 144 (2012), pp. 1-38.
30. P. Richtárik and M. Takáč, Parallel coordinate descent methods for big data optimization. Technical report, November 2012. arXiv:1212.0873.
31. P. Richtárik and M. Takáč, Efficiency of randomized coordinate descent methods on minimization problems with a composite objective function. 4th Workshop on Signal Processing with Adaptive Sparse Structured Representations, June 2011.
32. R. Tyrell Rockafellar, The multiplier method of Hestenes and Powell applied to convex programming, J. Optim. Theory Appl. 12 (1973), pp. 555-562.
33. R. Tyrell Rockafellar, Augmented Lagrangians and applications of the proximal point algorithm in convex programming, Math. Oper Res. 1 (1976), pp. 97-116.
34. R. Tyrell Rockafellar and R.J.-B. Wets, Scenarios and policy aggregation in optimization under uncertainty, Math. Oper. Res. 16 (1991), pp. 1-23.
35. A. Ruszczyński, An augmented Lagrangian method for block diagonal linear programming problems, Oper. Res. Lett. 8 (1989), pp. 287-294.
36. A. Ruszczyński, On convergence of an augmented Lagrangian decomposition method for sparse convex optimization, Math. Oper. Res. 20(3) (1995), pp. 634-656.
37. H. Schwarz, Über einen Grenzübergang durch alternierendes Verfahren, Vierteljahrsschrift der Naturforschenden Gesellschaft in Zürich 15 (1870), pp. 272-286.
38. 1 regularized loss minimization, in Proceedings of the 26th International Conference on Machine Learning, L. Bottou and M. Littman, eds., Omnipress, Montreal, June 2009, pp. 929-936.
39. S. Shalev-Shwartz and T. Zhang, Stochastic dual coordinate ascent methods for regularized loss minimization, J. Mach. Learn. Res. 14 (2013), pp. 567-599.
40. G. Stephanopoulos and A.W. Westerberg, The use of Hestenes' method of multipliers to resolve dual gaps in engineering system optimization, J. Optim. Theory Appl. 15 (1975), pp. 285-309.
41. M. Takáč, A. Bijral, P. Richtárik, and N. Srebro, Mini-batch primal and dual methods for SVMs, in 30th International Conference on Machine Learning, Volume 28 of JMLR: Workshop and Conference Proceedings, S. Dasgupta and D. McAllester, eds., Microtome Publishing, Atlanta, GA, 2013, pp. 1022-1030.
42. R. Tappenden,P. Richtárik, and J. Gondzio,Inexact coordinate descent: Complexity and preconditioning. Technical report, University of Edinburgh, April 2013. arXiv:1304.5530.
43. P. Tseng, Convergence of a block coordinate descent method for nondifferentiable minimization, J. Optim. Theory Appl. 109 (2001), pp. 475-494.
44. X. Wang, M. Hong, S. Ma, and Z.Q. Luo, Solving multiple-block separable convex minimization problems using two-block alternating direction method of multipliers. Technical report, August 2013. arXiv:1308.5294.
45. N. Watanabe, Y. Nishimura, and M. Matsubara, Decomposition in large system optimization using the method of multipliers, J. Optim. Theory Appl. 25 (1978), pp. 181-193.