Convergence of approximate and incremental subgradient methods for convex optimization

SIAM Journal on Optimization

Volumn 14, Issue 3, 2004, Pages 807-840

  Kiwiel, Krzysztof C ^a  

^a SYSTEMS RESEARCH INSTITUTE   (Poland)

Author keywords

Approximate subgradients; Convex programming; Efficiency; Nondifferentiable optimization; Subgradient optimization

Indexed keywords

EID: 4043072116     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/S1052623400376366     Document Type: Article

References (64)

1. [AIS98] YA. I. ALBER, A. N. IUSEM, AND M. V. SOLODOV, On the projected subgradient method for nonsmooth convex optimization in a Hilbert space, Math. Programming, 81 (1998), pp. 23-35.
2. [Ber97] D. P. BERTSEKAS, A new class of incremental gradient methods for least squares problems, SIAM J. Optim., 7 (1997), pp. 913-926.
3. [Ber99] D. P. BERTSEKAS, Nonlinear Programming, 2nd ed., Athena Scientific, Belmont, MA, 1999.
4. [BeT00] D. P. BERTSEKAS AND J. N. TSITSIKLIS, Gradient convergence in gradient methods with errors, SIAM J. Optim., 10 (2000), pp. 627-642.
5. [Brä93] U. BRÄNNLUND, On Relaxation Methods for Nonsmooth Convex Optimization, Ph.D. thesis, Department of Mathematics, Royal Institute of Technology, Stockholm, 1993.
6. [Brä95] U. BRÄNNLUND, A generalized subgradient method with relaxation step, Math. Programming, 71 (1995), pp. 207-219.
7. [BSS93] M. S. BAZARAA, H. D. SHERALI, AND C. M. SHETTY, Nonlinear Programming: Theory and Algorithms, 2nd ed., Wiley, New York, 1993.
8. [BTMN01] A. BEN-TAL, T. MARGALIT, AND A. NEMIROVSKI, The ordered subsets mirror descent optimization method with applications to tomography, SIAM J. Optim., 12 (2001), pp. 79-108.
9. [CoL93] R. CORREA AND C. LEMARÉCHAL, Convergence of some algorithms for convex minimization, Math. Programming, 62 (1993), pp. 261-275.
10. [DeV81] V. F. DEMYANOV AND L. V. VASILEV, Nondifferentiable Optimization, Nauka, Moscow, 1981 (in Russian); Optimization Software Inc., New York, 1985 (in English).
11. [DuS88] N. DUNFORD AND J. T. SCHWARTZ, Linear Operators, Part I: General Theory, Wiley-Interscience, New York, 1988.
12. [Erm66] YU. M. ERMOLIEV, Methods of solution of nonlinear extremal problems, Kibernetika, no. 4 (1966), pp. 1-17 (in Russian); Cybernetics, 2 (1966), pp. 1-16 (in English).
13. [Erm66] YU. M. ERMOLIEV, Methods of solution of nonlinear extremal problems, Kibernetika, no. 4 (1966), pp. 1-17 (in Russian); Cybernetics, 2 (1966), pp. 1-16 (in English).
14. [Erm76] YU. M. ERMOLIEV, Stochastic Programming Methods, Nauka, Moscow, 1976 (in Russian).
15. [ErS68] YU. M. ERMOLIEV AND N. Z. SHOR, A random search method for a two-stage stochastic programming problem and its generalization, Kibernetika, no. 1 (1968), pp. 90-92 (in Russian).
16. [Gai94] A. A. GAIVORONSKI, Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part 1, Optim. Methods Softw., 4 (1994), pp. 117-134.
17. [Gla65] E. G. GLADISHEV, On stochastic approximation, Theory Probab. Appl., 10 (1965), pp. 297-300 (in Russian).
18. [GoK99] J.-L. GOFFIN AND K. C. KIWIEL, Convergence of a simple subgradient level method, Math. Program., 85 (1999), pp. 207-211.
19. [Gri94] L. GRIPPO, A class of unconstrained minimization methods for neural network training, Optim. Methods Softw., 4 (1994), pp. 135-150.
20. [HUL93] J.-B. HIRIART-URRUTY AND C. LEMARÉCHAL, Convex Analysis and Minimization Algorithms, Springer-Verlag, Berlin, 1993.
21. [KiA91] S. KIM AND H. AHN, Convergence of a generalized subgradient method for nondifferentiable optimization, Math. Programming, 50 (1991), pp. 75-80.
22. [Kib79] V. M. KIBARDIN, Decomposition into functions in the minimization problem, Avtomat. i Telemekh., no. 9 (1979), pp. 66-79 (in Russian); Automat. Remote Control, 40 (1980), pp. 1311-1323 (in English).
23. [Kib79] V. M. KIBARDIN, Decomposition into functions in the minimization problem, Avtomat. i Telemekh., no. 9 (1979), pp. 66-79 (in Russian); Automat. Remote Control, 40 (1980), pp. 1311-1323 (in English).
24. [KiU93] S. KIM AND B.-S. UM, An improved subgradient method for constrained nondifferentiable optimization, Oper. Res. Lett., 14 (1993), pp. 61-64.
25. [Kiw96a] K. C. KIWIEL, The efficiency of subgradient projection methods for convex optimization, Part II: Implementations and extensions, SIAM J. Control Optim., 34 (1996), pp. 677-697.
26. [Kiw96b] K. G. KIWIEL, A Subgradient Method with Bregman Projections for Convex Constrained Nondifferentiable Minimization, Tech. report, Systems Research Institute, Warsaw, Poland, 1996.
27. [Kiw98] K. C. KIWIEL, Subgradient method with entropic projections for convex nondifferentiable minimization, J. Optim. Theory Appl., 96 (1998), pp. 159-173.
28. [KLL99a] K. C. KIWIEL, T. LARSSON, AND P. O. LINDBERG, Dual Properties of Ballstep Subgradient Methods, with Applications to Lagrangian Relaxation, Tech. report LiTH-MAT-R-1999-24, Department of Mathematics, Linköping University, Linköping, Sweden, 1999. Revised 2002.
29. [KLL99b] K. C. KIWIEL, T. LARSSON, AND P. O. LINDBERG, The efficiency of ballstep subgradient level methods for convex optimization, Math. Oper. Res., 24 (1999), pp. 237-254.
30. [Lis86] S. A. LISINA, A subgradient method for minimizing a convex function for the case when the infimum is not achieved, Vestnik Leningrad. Univ. Mat. Mekh. Astronom., no. 4 (1986), pp. 70-74 (in Russian).
31. [Lit68] B. M. LITVAKOV, Convergence of recurrent algorithms for pattern recognition learning, Avtomat. i Telemekh., no. 3 (1968), pp. 142-150 (in Russian); Automat. Remote Control, 29 (1968), pp. 121-128 (in English).
32. [Lit68] B. M. LITVAKOV, Convergence of recurrent algorithms for pattern recognition learning, Avtomat. i Telemekh., no. 3 (1968), pp. 142-150 (in Russian); Automat. Remote Control, 29 (1968), pp. 121-128 (in English).
33. [LPS96] T. LARSSON, M. PATRIKSSON, AND A.-B. STRÖMBERG, Conditional subgradient optimization - theory and applications, European J. Oper. Res., 88 (1996), pp. 382-403.
34. [LPS00] T. LARSSON, M. PATRIKSSON, AND A.-B. STRÖMBERG, On the Convergence of Conditional ∈-Subgradient Methods for Convex Programs and Convex-Concave Saddle-Point Problems. Tech. report, Department of Mathematics, Chalmers University of Technology, Linköping, Sweden, 2000.
35. [Luo91] Z. Q. LUO, On the convergence of the LMS algorithm with adaptive learning rate for linear feedforward networks, Neural Computation, 3 (1991), pp. 226-245.
36. [LuT94] Z.-Q. LUO AND P. TSENG, Analysis of an approximate gradient projection method with applications to the backpropagation algorithm, Optim. Methods Softw., 4 (1994), pp. 85-101.
37. [MaS94] O. L. MANGASARIAN AND M. V. SOLODOV, Serial and parallel backpropagation convergence via nonmonotone perturbed minimization, Optim. Methods Softw., 4 (1994), pp. 103-116.
38. [MGN87] V. S. MIKHALEVICH, A. M. GUPAL, AND V. I. NORKIN, Methods of Nonconvex Optimization, Nauka, Moscow, 1987 (in Russian).
39. [Min86] M. MINOUX, Mathematical Programming, Theory and Algorithms, John Wiley and Sons, Chichester, UK, 1986.
40. [MiU82] F. MIRZOAKHMEDOV AND S. P. URYASEV, Adaptive stepsize control for the stochastic approximation algorithm, Zh. Vychisl. Mat. i Mat. Fiz., 23 (1982), pp. 1314-1325 (in Russian).
41. [NeB01] A. NEDIĆ AND D. P. BERTSEKAS, Incremental subgradient methods for nondifferentiable optimization, SIAM J. Optim., 12 (2001), pp. 109-138.
42. [Nes84] YU. E. NESTEROV, Minimization methods for nonsmooth convex and quasiconvex functions, Èkonom. i Mat. Metody, 20 (1984), pp. 519-531 (in Russian); Matekon, 29 (1984), pp. 519-531 (in English).
43. [Nes84] YU. E. NESTEROV, Minimization methods for nonsmooth convex and quasiconvex functions, Èkonom. i Mat. Metody, 20 (1984), pp. 519-531 (in Russian); Matekon, 29 (1984), pp. 519-531 (in English).
44. [Nes89] YU. E. NESTEROV, Effective Methods in Nonlinear Programming, Radio i Sviaz, Moscow, 1989 (in Russian).
45. [NeY78] A. S. NEMIROVSKII AND D. B. YUDIN, Cesaro convergence of the gradient method for approximating saddle points of convex-concave functions, Dokl. Akad. Nauk SSSR, 239 (1978), pp. 1056-1059 (in Russian).
46. [Nur79] E. A. NURMINSKII, Numerical Methods for Solving Deterministic and Stochastic Minimax Problems, Naukova Dumka, Kiev, 1979 (in Russian).
47. [Nur82] E. A. NURMINSKI, Subgradient method for minimizing weakly convex functions and ∈-subgradient methods of convex optimization, in Progress in Nondifferentiable Optimization, E. A. Nurminski, ed., CP-82-S8, International Institute for Applied Systems Analysis, Laxenburg, Austria, 1982, pp. 97-123.
48. [Nur91] E. A. NURMINSKII, Numerical Methods for Convex Optimization, Nauka, Moscow, 1991 (in Russian).
49. [NuZ77] E. A. NURMINSKII AND A. A. ZHELIKHOVSKII, ∈-Quasigradient method for solving non-smooth extremal problems, Kibernetika, no. 1 (1977), pp. 109-113 (in Russian); Cybernetics, 13 (1977), pp. 109-114 (in English).
50. [NuZ77] E. A. NURMINSKII AND A. A. ZHELIKHOVSKII, ∈-Quasigradient method for solving non-smooth extremal problems, Kibernetika, no. 1 (1977), pp. 109-113 (in Russian); Cybernetics, 13 (1977), pp. 109-114 (in English).
51. [Pol67] B. T. POLYAK, A general method for solving extremum problems, Dokl. Akad. Nauk SSSR, 174 (1967), pp. 33-36 (in Russian); Soviet Math. Dokl., 8 (1967), pp. 593-597 (in English).
52. [Pol67] B. T. POLYAK, A general method for solving extremum problems, Dokl. Akad. Nauk SSSR, 174 (1967), pp. 33-36 (in Russian); Soviet Math. Dokl., 8 (1967), pp. 593-597 (in English).
53. [Pol69] B. T. POLYAK, Minimization of unsmooth functional, Zh. Vychisl. Mat. i Mat. Fiz., 9 (1969), pp. 509-521 (in Russian); U.S.S.R. Comput. Math. and Math. Phys., 9 (1969), pp. 14-29 (in English).
54. [Pol69] B. T. POLYAK, Minimization of unsmooth functional, Zh. Vychisl. Mat. i Mat. Fiz., 9 (1969), pp. 509-521 (in Russian); U.S.S.R. Comput. Math. and Math. Phys., 9 (1969), pp. 14-29 (in English).
55. [Pol78] B. T. POLYAK, Subgradient methods: A survey of Soviet research, in Nonsmooth Optimization, C. Lemaréchal and R. Mifflin, eds., Pergamon Press, Oxford, UK, 1978, pp. 5-29.
56. [Pol83] B. T. POLYAK, Introduction to Optimization, Nauka, Moscow, 1983 (in Russian); Optimization Software Inc., New York, 1987 (in English).
57. [PoT73] B. T. POLYAK AND YA. Z. TSYPKIN, Pseudogradient adaptation and training algorithms, Avtomat. i Telemekh., no. 3 (1973), pp. 45-68 (in Russian); Automat. Remote Control, 34 (1973), pp. 377-397 (in English).
58. [PoT73] B. T. POLYAK AND YA. Z. TSYPKIN, Pseudogradient adaptation and training algorithms, Avtomat. i Telemekh., no. 3 (1973), pp. 45-68 (in Russian); Automat. Remote Control, 34 (1973), pp. 377-397 (in English).
59. [Roc70] R. T. ROCKAFELLAR, Convex Analysis, Princeton University Press, Princeton, NJ, 1970.
60. [Sch83] K. SCHULZ, A note on the convergence of subgradient optimization methods, Math. Operationsforsch. Statist. Ser. Optim., 14 (1983), pp. 537-541.
61. [SCT00] H. D. SHERALI, G. CHOI, AND C. H. TUNCBILEK, A variable target value method for nondifferentiable optimization, Oper. Res. Lett., 26 (2000), pp. 1-8.
62. [Sho62] N. Z. SHOR, An application of the generalized gradient descent method to the solution of a network transportation problem, in Proceedings of the Scientific Seminar on Theoretical and Application Problems of Cybernetics and Operations Research, no. 1, Scientific Council on Cybernetics of the Academy of Sciences of the Ukrainian SSR, Kiev, 1962, pp. 9-17 (in Russian).
63. [Sho79] N. Z. SHOR, Minimization Methods for Non-Differentiable Functions, Naukova Dumka, Kiev, 1979 (in Russian); Springer-Verlag, Berlin, 1985 (in English).
64. [ShW96] A. SHAPIRO AND Y. WARDI, Convergence analysis of gradient descent stochastic algorithms, J. Optim. Theory Appl., 91 (1996), pp. 439-454.