A stochastic quasi-Newton method for large-scale optimization

SIAM Journal on Optimization

Volumn 26, Issue 2, 2016, Pages 1008-1031

  Byrd, R H ^a   Hansen, S L ^b   Nocedal, Jorge ^b   Singer, Y ^c  

^a UCB 450   (United States)

^b Northwestern University   (United States)

^c GOOGLE INC   (United States)

Author keywords

Large scale optimization; Quasi Newton; Stochastic optimization; Sub sampling

Indexed keywords

APPROXIMATION THEORY; ARTIFICIAL INTELLIGENCE; CURVE FITTING; ITERATIVE METHODS; LEARNING SYSTEMS; NEWTON-RAPHSON METHOD; NUMERICAL METHODS; STOCHASTIC SYSTEMS;

CURVATURE INFORMATION; DETERMINISTIC OPTIMIZATION; HESSIAN-VECTOR PRODUCTS; LARGE-SCALE OPTIMIZATION; QUASI-NEWTON; STOCHASTIC APPROXIMATION METHODS; STOCHASTIC OPTIMIZATIONS; SUB-SAMPLING;

OPTIMIZATION;

EID: 84976910543     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/140954362     Document Type: Article

References (28)

1. S.-I. AMARI, Natural gradient works efficiently in learning, Neural Comput., 10(1998), pp. 251-276.
2. S. ASMUSSEN and P. W. GLYNN, Stochastic Simulation: Algorithms and Analysis, Stoch. Model. Appl. Probab. 57, Springer, New York, 2007.
3. F. BACH and E. MOULINES, Non-strongly-convex smooth stochastic approximation with convergence rate o (1/n), in Advances in Neural Information Processing Systems 26, Curran, Red Hook, NY, 2013, pp. 773-781.
4. A. BORDES, L. BOTTOU, and P. GALLINARI, SGD-QN: Careful quasi-Newton stochastic gradient descent, J. Mach. Learn. Res., 10(2009), pp. 1737-1754.
5. L. BOTTOU and O. BOUSQUET, The tradeoffs of large scale learning, in Advances in Neural Information Processing Systems 20, J. C. Platt, D. Koller, Y. Singer, and S. Roweis, eds., MIT Press, Cambridge, MA, 2008, pp. 161-168.
6. L. BOTTOU and Y. LECUN, Large scale online learning, in Advances in Neural Information Processing Systems 16, S. Thrun, L. Saul, and B. Schölkopf, eds., MIT Press, Cambridge, MA, 2004.
7. R. H. BYRD, G. M. CHIN, J. NOCEDAL, and Y. WU, Sample size selection in optimization methods for machine learning, Math. Program., 134(2012), pp. 127-155.
8. R. H BYRD, G. M CHIN, W. NEVEITT, and J. NOCEDAL, On the use of stochastic Hessian information in optimization methods for machine learning, SIAM J. Optim., 21(2011), pp. 977-995.
9. J. DUCHI, E. HAZAN, and Y. SINGER, Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res., 12(2011), pp. 2121-2159.
10. R. FLETCHER, Practical Methods of Optimization, 2nd ed., Wiley, Chichester, 1987.
11. D. D LEWIS, Y. YANG, T. G. ROSE, and F. LI, Rcv 1: A new benchmark collection for text categorization research, J. Mach. Learn. Res., 5(2004), pp. 361-397.
12. A. MOKHTARI and A. RIBEIRO, Regularized stochastic BFGS algorithm, in IEEE Global Conference on Signal and Information Processing, IEEE, Piscataway, NJ, 2013.
13. A. MOKHTARI and A. RIBEIRO, Global convergence of online limited memory bfgs, preprint, arXiv:1409.2045, 2014.
14. I. MUKHERJEE, K. CANINI, R. FRONGILLO, and Y. SINGER, Parallel boosting with momentum, in ECML PKDD 2013, Part III, Lecture Notes in Comput. Sci. 8190, Springer, Heidelberg, 2013, pp. 17-32.
15. N. MURATA, A statistical study of on-line learning, in On-line Learning in Neural Networks, Cambridge University Press, Cambridge, UK, 1998, pp. 63-92.
16. A. NEDIĆ and D. BERTSEKAS, Convergence rate of incremental subgradient algorithms, in Stochastic Optimization: Algorithms and Applications, Springer, Boston, 2001, pp. 223-264.
17. A. NEMIROVSKI, A. JUDITSKY, G. LAN, and A. SHAPIRO, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19(2009), pp. 1574-1609.
18. J. NOCEDAL and S. WRIGHT, Numerical Optimization, 2nd ed., Springer New York, 1999.
19. H. PARK, S.-I. AMARI, and K. FUKUMIZU, Adaptive natural gradient learning algorithms for various stochastic models, Neural Networks, 13(2000), pp. 755-764.
20. A. PLAKHOV and P. CRUZ, A stochastic approximation algorithm with step-size adaptation, J. Math. Sci., 120(2004), pp. 964-973.
21. M. J. D. POWELL, Some global convergence properties of a variable metric algorithm for minimization without exact line searches, in Nonlinear Programming, R. W. Cottle and C. E. Lemke, eds., SIAM-AMS Proc. 9, AMS, Providence, RI, 1976, pp. 53-72.
22. H. ROBBINS and S. MONRO, A stochastic approximation method, Ann. Math. Statist., 22(1951), pp. 400-407.
23. N. L. ROUX and A. W. FITZGIBBON, A fast natural Newton method, in Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010, pp. 623-630.
24. N. L. ROUX, P.-A. MANZAGOL, and Y. BENGIO, Topmoumoute online natural gradient algorithm, in Advances in Neural Information Processing Systems 20, MIT Press, Cambridge, MA, 2007, pp. 849-856.
25. N. SCHRAUDOLPH, J. YU, and S. GÜNTER, A stochastic quasi-Newton method for online convex optimization, in Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, Microtome Publishing, Brookline, MA, 2007, pp. 436-443.
26. P. SUNEHAG, J. TRUMPF, S. V. N. VISHWANATHAN, and N. SCHRAUDOLPH, Variable metric stochastic approximation theory, in Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, Microtome Publishing, Brookline, MA, (2007), pp. 436-443.
27. P. TSENG and S. YUN, A coordinate gradient descent method for nonsmooth separable minimization, Math. Program., 117(2009), pp. 387-423.
28. F. YOUSEFIAN, A. NEDIĆ, and U. V. SHANBHAG, On stochastic gradient and subgradient methods with adaptive steplength sequences, Automatica J. IFAC, 48(2012), pp. 56-67.