Convergence of Stochastic Proximal Gradient Algorithm

Applied Mathematics and Optimization

Volumn 82, Issue 3, 2020, Pages 891-917

  Rosasco, Lorenzo ^a,b   Villa, Silvia ^a   Vũ, Bằng Công ^c  

^a UNIVERSITY OF GENOA   (Italy)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c EPFL   (Switzerland)

Author keywords

Forward backward splitting algorithm; Proximal methods; Stochastic optimization

Indexed keywords

ITERATIVE METHODS; OPTIMIZATION;

ALMOST SURE CONVERGENCE; CONVERGENCE RATES; FUNCTION VALUES; GRADIENT ALGORITHM; PROXIMAL METHODS; SPLITTING ALGORITHMS; STOCHASTIC GRADIENT; STOCHASTIC OPTIMIZATIONS;

STOCHASTIC SYSTEMS;

EID: 85074528968     PISSN: 00954616     EISSN: 14320606     Source Type: Journal    
DOI: 10.1007/s00245-019-09617-7     Document Type: Article

References (52)

1. Atchade, Y.F., Fort, G., Moulines, E.: On perturbed proximal gradient algorithms. J. Mach. Learn. Res. 18, 310–342 (2017)
2. Bach, F., Jenatton, R., Mairal, J., Obozinski, G.: Structured sparsity through convex optimization. Stat. Sci. 27, 450–468 (2012)
3. Bach, F., Moulines, E.: Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In: Advances in Neural Information Processing Systems, vol. 24 (2011)
4. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)
5. Benveniste, A., Métivier, M., Priouret, P.: Adaptive Algorithms and Stochastic Approximations, vol. 22. Springer, Berlin (1990)
6. Bertsekas, D.P.: Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. In: Optimization for Machine Learning, pp. 85–104 (2011)
7. Bertsekas, D.P., Tsitsiklis, J.N.: Gradient convergence in gradient methods with errors. SIAM J. Optim. 10, 627–642 (2000). (electronic)
8. Bianchi, P., Hachem, W.: Dynamical behavior of a stochastic forward-backward algorithm using random monotone operators. J. Optim. Theory Appl. 171, 90–120 (2016)
9. Blatt, D., Hero, A., Gauchman, H.: A convergent incremental gradient method with a constant step size. SIAM J. Optim. 18, 29–51 (2007)
10. Bottou, L., Curtis, F., Nocedal, J.: Optimization methods for large-scale machine learning. SIAM Rev. 60, 223–311 (2018)
11. Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press, Cambridge (2006)
12. Combettes, P.L., Pesquet, J.-C.: Stochastic quasi-fejér block-coordinate fixed point iterations with random sweeping. SIAM J. Optim. 25, 1221–1248 (2015)
13. Combettes, P.L., Pesquet, J.-C.: Stochastic approximations and perturbations in forward-backward splitting for monotone operators. Pure Appl. Funct. Anal. 1, 13–37 (2016)
14. Combettes, P.L., Pesquet, J.-C.: Proximal splitting methods in signal processing. In: Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212. Springer, New York (2011)
15. Combettes, P.L., Wajs, Valérie R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4, 1168–1200 (2005). (electronic)
16. De Mol, C., De Vito, E., Rosasco, L.: Elastic-net regularization in learning theory. J. Complex. 25, 201–230 (2009)
17. Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: a fast incremental gradient method with support for non-strongly convex composite objectives. Adv. Neural Inf. Process. Syst. 27, 1646–1654 (2014)
18. Duchi, J., Singer, Y.: Efficient online and batch learning using forward backward splitting. J. Mach. Learn. Res. 10, 2899–2934 (2009)
19. Ermol’ev, YuM: On the method of generalized stochastic gradients and quasi-fejér sequences. Cybernetics 5, 208–220 (1969)
20. Ermol’ev, YuM, Tuniev, A.D.: Random fejér and quasi-fejér sequences. Theory Optim. Solut. Akademiya Nauk Ukrainskoĭ SSR Kiev 2, 76–83 (1968)
21. Ghadimi, S., Lan, G.: Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, I: a generic algorithmic framework. SIAM J. Optim. 22, 1469–1492 (2012)
22. Ghadimi, S., Lan, G.: Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, II: shrinking procedures and optimal algorithms. SIAM J. Optim. 23, 2061–2089 (2013)
23. Hazan, E., Kale, S.: Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization. J. Mach. Learn. Res. 15, 2489–2512 (2014)
24. Jofré, A., Thompson, P.: On variance reduction for stochastic smooth convex optimization with multiplicative noise. In: Mathematical Programming, Series B, Published Online, pp. 1–40 (2018)
25. Juditski, A., Nesterov, Y.: Deterministic and stochastic primal-dual subgradient methods for uniformly convex minimization. Stoch. Syst. 4, 44–80 (2014)
26. Kiefer, J., Wolfowitz, J.: Stochastic estimation of the maximum of a regression function. Ann. Math. Stat. 23, 462–466 (1952)
27. Kushner, H.J., Yin, G.G.: Stochastic Approximation Algorithms and Applications, vol. 35. Springer, New York (1997)
28. Kwok, J.T., Hu, C., Pan, W.: Accelerated gradient methods for stochastic optimization and online learning. Adv. Neural Inf. Process. Syst. 22, 781–789 (2009)
29. Lan, G.: An optimal method for stochastic composite optimization. Math. Progr. 133, 365–397 (2012)
30. Lin, J., Rosasco, L., Villa, S., Zhou, D.-X.: Modified Fejér sequences and applications. Comput. Optim. Appl. 71, 95–113 (2018)
31. Lin, Q., Chen, X., Peña, J.: A sparsity preserving stochastic gradient methods for sparse regression. Comput. Optim. Appl. 58, 455–482 (2014)
32. Mosci, S., Rosasco, L., Santoro, M., Verri, A., Villa, S.: Solving structured sparsity regularization with proximal methods. In: Machine Learning and Knowledge discovery in Databases European Conference, ECML PKDD, pp. 418–433. Springer, Berlin (2010)
33. Nekrylova, Z.V.: The rate of convergence of the stochastic gradient method. Cybernetics 11, 218–222 (1975)
34. Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19, 1574–1609 (2008)
35. Nemirovski, A., Yudin, D.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)
36. Polyak, B.: Introduction to Optimization. Optimization Software, New York (1987)
37. Polyak, B.T., Juditsky, A.B.: Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30, 838–855 (1992)
38. Rakhlin, A., Shamir, O., Sridaran, K.: Making gradient descent optimal for strongly convex stochastic optimization. In: Proceedings of the 29th International Conference on Machine Learning, pp. 449–456 (2012)
39. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951)
40. Rosasco, L., Villa, S., Vu, B.C.: Stochastic forward-backward splitting for monotone inclusions. J. Optim. Theory Appl. 169, 388–406 (2016)
41. Rosasco, L., Villa, S., Vu, B.C.: A first-order stochastic primal-dual algorithm with correction step. Numer. Funct. Anal. Optim. 38, 602–626 (2017)
42. Schmidt, M., Le Roux, N., Bach, F.: Minimizing finite sums with the stochastic average gradient. Math. Progr. Ser. B 162, 83–112 (2017)
43. Schmidt, M.W., Le Roux, N., Bach, F.: Convergence rates of inexact proximal-gradient methods for convex optimization. Adv. Neural Inf. Process. Syst. 24, 1458–1466 (2011)
44. Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, Cambridge (2014)
45. Shalev-Shwartz, S., Singer, Y., Srebro, N., Cotter, A.: Pegasos: primal estimated sub-gradient solver for SVM. Math. Progr. Ser. B 127, 3–30 (2011)
46. Shalev-Shwartz, S., Zhang, T.: Accelerated mini-batch stochastic dual coordinate ascent. In: Advances in Neural Information Processing Systems, vol. 26 (2013)
47. Shamir, O., Zhang, T.: Stochastic gradient descent for non-smooth optimization: convergence results and optimal averaging schemes. In: Proceedings of the 30th International Conference on Machine Learning, pp. 71–79 (2013)
48. Shor, N.Z.: Minimization Methods for Non-differentiable Functions. Springer, Berlin (1985)
49. Villa, S., Rosasco, L., Mosci, S., Verri, A.: Proximal methods for the latent group lasso penalty. Comput. Optim. Appl. 58, 381–407 (2014)
50. Villa, S., Salzo, S., Baldassarre, L., Verri, A.: Accelerated and inexact forward-backward algorithms. SIAM J. Optim. 23, 1607–1633 (2013)
51. Xiao, L.: Dual averaging methods for regularized stochastic learning and online optimization. J. Mach. Learn. Res. 11, 2543–2596 (2010)
52. Zou, Z., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B 67, 301–320 (2005)