Global convergence of online limited memory BFGS

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 3151-3181

  Mokhtari, Aryan ^a   Ribeiro, Alejandro ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Large scale optimization; Quasi Newton methods; Stochastic optimization

Indexed keywords

ARTIFICIAL INTELLIGENCE; EIGENVALUES AND EIGENFUNCTIONS; LEARNING SYSTEMS; NEWTON-RAPHSON METHOD; NONLINEAR PROGRAMMING; SEARCH ENGINES; STOCHASTIC SYSTEMS;

BROYDEN-FLETCHER-GOLDFARB-SHANNO; EXPERIMENTAL EVALUATION; LARGE-SCALE MACHINE LEARNING; LARGE-SCALE OPTIMIZATION; LOWER AND UPPER BOUNDS; QUASI-NEWTON METHODS; STOCHASTIC GRADIENT DESCENT ALGORITHM; STOCHASTIC OPTIMIZATIONS;

OPTIMIZATION;

EID: 84961737543     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (26)

1. John R. Birge, Liqun Qi, and Xiaochun Chen. A Stochastic Newton Method for Stochastic Quadratic Programs with Recourse. School of Mathematics, University of New South Wales Sydney, Australia, 1994.
2. Antoine Bordes, Léon Bottou, and Patrick Gallinari. Sgd-qn: Careful quasi-newton stochastic gradient descent. The Journal of Machine Learning Research, 10:1737-1754, 2009.
3. Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pages 144-152. ACM, 1992.
4. Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pages 177-186. Springer, 2010.
5. Léon Bottou and Yann Le Cun. On-line learning for very large data sets. Applied Stochastic Models in Business and Industry, 21(2):137-151, 2005.
6. Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge university press, 2004.
7. Charles G. Broyden, John E. Dennis, and Jorge J. Moré. On the local and superlinear convergence of quasi-newton methods. IMA Journal of Applied Mathematics, 12(3):223-245, 1973.
8. Richard H. Byrd, Jorge Nocedal, and Ya-Xiang Yuan. Global convergence of a class of quasi-newton methods on convex problems. SIAM Journal on Numerical Analysis, 24(5):1171-1190, 1987.
9. John E. Dennis and Jorge J. Moré. A characterization of superlinear convergence and its application to quasi-newton methods. Mathematics of Computation, 28(126):549-560, 1974.
10. Roger Fletcher. Practical Methods of Optimization. John Wiley & Sons, 2013.
11. Dong-Hui Li and Masao Fukushima. A modified bfgs method and its global convergence in nonconvex minimization. Journal of Computational and Applied Mathematics, 129(1):15-35, 2001.
12. Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45(1-3):503-528, 1989.
13. Aryan Mokhtari and Alejandro Ribeiro. Res: Regularized stochastic bfgs algorithm. IEEE Transactions on Signal Processing, 62:6089-6104, 2014a.
14. Aryan Mokhtari and Alejandro Ribeiro. A quasi-newton method for large scale support vector machines. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 8302-8306. IEEE, 2014b.
15. Andrew Y. Ng. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the Twenty-First International Conference on Machine Learning, page 78. ACM, 2004.
16. Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer-Verlag, New York, NY, 2 edition, 1999.
17. Michael J. D. Powell. Some global convergence properties of a variable metric algorithm for minimization without exact line searches. Nonlinear Programming, 9:53-72, 1976.
18. Nicol N. Schraudolph, Jin Yu, and Simon Günter. A stochastic quasi-newton method for online convex optimization. In International Conference on Artificial Intelligence and Statistics, pages 436-443, 2007.
19. Shai Shalev-Shwartz and Nathan Srebro. Svm optimization: inverse dependence on training set size. In Proceedings of the 25th International Conference on Machine Learning, pages 928-935. ACM, 2008.
20. Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Primal estimated sub-gradient solver for svm. Mathematical programming, 127(1):3-30, 2011.
21. Victor Solo and Xuan Kong. Adaptive signal processing algorithms: stability and performance. Prentice Hall, New Jersey, 1995.
22. G. Sun. Kdd cup track 2 soso. com ads prediction challenge, 2012. Accessed August, 1, 2012.
23. Peter Sunehag, Jochen Trumpf, Nicol N. Schraudolph, and Swaminathan V. N. Vishwanathan. Variable metric stochastic approximation theory. In International Conference on Artificial Inteligence and Statistics, pages 560-566, 2009.
24. Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer Science & Business Media, 2000.
25. Michael Zargham, Alejandro Ribeiro, and Ali Jadbabaie. Accelerated backpressure algorithm. In Global Communications Conference (GLOBECOM), 2013 IEEE, pages 2269-2275. IEEE, 2013.
26. Tong Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the Twenty-First International Conference on Machine Learning, page 116. ACM, 2004.