A globally convergent incremental Newton method

Mathematical Programming

Volumn 151, Issue 1, 2015, Pages 283-313

  Gurbuzbalaban M ^a   Ozdaglar, A ^a   Parrilo, P ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Convex optimization; EKF algorithm; Gauss Newton method; Incremental methods; Newton method; Strong convexity

Indexed keywords

ARTIFICIAL INTELLIGENCE; CONVEX OPTIMIZATION; FUNCTIONS; LEARNING SYSTEMS;

DISTRIBUTED OPTIMIZATION; GLOBALLY CONVERGENT; INCREMENTAL METHOD; LINEAR CONVERGENCE; LINEAR CONVERGENCE RATE; MACHINE LEARNING PROBLEM; STRONG CONVEXITIES; VARIABLE STEP SIZE;

NEWTON-RAPHSON METHOD;

EID: 84937762700     PISSN: 00255610     EISSN: 14364646     Source Type: Journal    
DOI: 10.1007/s10107-015-0897-y     Document Type: Article

References (37)

1. Bertsekas, D.: Incremental least squares methods and the extended Kalman filter. SIAM J. Optim. 6(3), 807–822 (1996)
2. Bertsekas, D.: A new class of incremental gradient methods for least squares problems. SIAM J. Optim. 7(4), 913–926 (1997)
3. Bertsekas, D.: Nonlinear Programming. Athena Scientific, United States (1999)
4. Bertsekas, D.: Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. Optim. Mach. Learn. 2010, 1–38 (2011)
5. Bertsekas, D.: Convex Optimization Algorithms. Athena Scientific, United States (2015)
6. Blatt, D., Hero, A., Gauchman, H.: A convergent incremental gradient method with a constant step size. SIAM J. Optim. 18(1), 29–51 (2007)
7. Bordes, A., Bottou, L., Gallinari, P.: SGD-QN: careful quasi-Newton stochastic gradient descent. J. Mach. Learn. Res. 10, 1737–1754 (2009)
8. Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Lechevallier, Y., Saporta, G. (eds.) Proceedings of COMPSTAT’2010, pp. 177–186. Physica-Verlag HD, Heidelberg (2010)
9. Bottou, L., Le Cun, Y.: On-line learning for very large data sets. Appl. Stoch. Models Bus. Ind. 21(2), 137–151 (2005)
10. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)
11. Byrd, R.H., Hansen, S.L., Nocedal, J., Singer, Y.: A Stochastic Quasi-Newton Method for Large-Scale Optimization. arXiv preprint arXiv:1401.7020 (2014)
12. Cătinaş, E.: Inexact perturbed Newton methods and applications to a class of Krylov solvers. J. Optim. Theory Appl. 108(3), 543–570 (2001)
13. Davidon, W.C.: New least-square algorithms. J. Optim. Theory Appl. 18(2), 187–197 (1976)
14. Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: a fast incremental gradient method with support for non-strongly convex composite objectives. arXiv preprint arXiv:1407.0202 (2014)
15. Dembo, R., Eisenstat, S., Steihaug, T.: Inexact Newton methods. SIAM J. Numer. Anal. 19(2), 400–408 (1982)
16. Dennis, J.E., Moré, J.J.: A characterization of superlinear convergence and its application to quasi-newton methods. Math. Comput. 28(126), 549–560 (1974)
17. Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)
18. Mairal, J.: Optimization with first-order surrogate functions. In: ICML, Volume 28 of JMLR Proceedings, pp. 783–791, Atlanta, United States (2013)
19. Mangasarian, O.L., Solodov, M.V.: Serial and parallel backpropagation convergence via nonmonotone perturbed minimization. Optim. Methods Softw. 4(2), 103–116 (1994)
20. Mokhtari, A., Ribeiro, A.: Res: regularized Stochastic BFGS algorithm. arXiv preprint arXiv:1401.7625 (2014)
21. Moriyama, H., Yamashita, N., Fukushima, M.: The incremental Gauss–Newton algorithm with adaptive stepsize rule. Comput. Optim. Appl. 26(2), 107–141 (2003)
22. Nedić, A., Bertsekas, D.: Convergence rate of incremental subgradient algorithms. In: Uryasev, S., Pardalos, P.M. (eds.) Stochastic Optimization: Algorithms and Applications. Applied Optimization, vol. 54, pp. 223–264. Springer, US (2001)
23. Nedić, A., Ozdaglar, A.: On the rate of convergence of distributed subgradient methods for multi-agent optimization. In: Proceedings of IEEE CDC, pp. 4711–4716 (2007)
24. Nedić, A., Ozdaglar, A.: Distributed subgradient methods for multi-agent optimization. IEEE Trans. Autom. Control 54(1), 48–61 (2009)
25. Polyak, B.T.: Introduction to optimization. Translations series in mathematics and engineering. Optimization Software, Publications Division, New York (1987)
26. Ram, S.S., Nedic, A., Veeravalli, V.V.: Stochastic incremental gradient descent for estimation in sensor networks. In: Signals, Systems and Computers, 2007. ACSSC 2007. Conference Record of the Forty-First Asilomar Conference on, pp. 582–586 (2007)
27. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22(3), 400–407 (1951)
28. Roux, N.L., Schmidt, M., Bach, F.R.: A stochastic gradient method with an exponential convergence rate for finite training sets. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 2663–2671. Curran Associates Inc., NY, USA (2012)
29. Schmidt, M., Roux, N.L.: Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370, 2013
30. Schraudolph, N., Yu, J., Günter, S.: A stochastic quasi-Newton method for online convex optimization. In: Proceedings of the 11th International Conference Artificial Intelligence and Statistics (AISTATS), pp. 433–440 (2007)
31. Shamir, O., Srebro, N., Zhang, T.: Communication efficient distributed optimization using an approximate Newton-type method. ICML 32(1), 1000–1008 (2014)
32. Sohl-Dickstein, J., Poole, B., Ganguli, S.: Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In: Jebara, T., Xing, E.P. (eds.) ICML, pp. 604–612. JMLR Workshop and Conference Proceedings (2014)
33. Solodov, M.V.: Incremental gradient algorithms with stepsizes bounded away from zero. Comput. Optim. Appl. 11(1), 23–35 (1998)
34. Sparks, E.R., Talwalkar, A., Smith, V., Kottalam, J., Xinghao, P., Gonzalez, J., Franklin, M.J., Jordan, M.I, Kraska, T.: MLI: an API for distributed machine learning. In: IEEE 13th International Conference on Data Mining (ICDM), pp. 1187–1192 (2013)
35. Tseng, P.: An incremental gradient(-projection) method with momentum term and adaptive stepsize rule. SIAM J. Optim. 8(2), 506–531 (1998)
36. Tseng, P., Yun, S.: Incrementally updated gradient methods for constrained and regularized optimization. J. Optim. Theory Appl. 160(3), 832–853 (2014)
37. Zhang, T.: Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Proceedings of the Twenty-First International Conference on Machine Learning, ICML, pp. 919–926. ACM, New York (2004)