A linearly-convergent stochastic L-BFGS algorithm

Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016

Volumn , Issue , 2016, Pages 249-258

  Moritz, Philipp ^a   Nishihara, Robert ^a   Jordan, Michael I ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CONVEX OPTIMIZATION; GRADIENT METHODS;

LINEAR CONVERGENCE; LINEAR CONVERGENCE RATE; NONCONVEX OPTIMIZATION; OPTIMIZATION PROBLEMS; ORDERS OF MAGNITUDE; SMOOTH FUNCTIONS; STOCHASTIC GRADIENT DESCENT; VARIANCE REDUCTIONS;

STOCHASTIC SYSTEMS;

EID: 85067064967     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. A. Agarwal, O. Chapelle, M. Dudík, and J. Langford. A reliable effective terascale linear learning system. The Journal of Machine Learning Research, 15(1): 1111–1133, 2014.
2. T. Bertin-Mahieux, D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. In International Conference on Music Information Retrieval, 2011.
3. A. Bordes, L. Bottou, and P. Gallinari. SGD-QN: Careful quasi-Newton stochastic gradient descent. The Journal of Machine Learning Research, 10: 1737–1754, 2009.
4. L. Bottou. Large-scale machine learning with stochastic gradient descent. In International Conference on Computational Statistics, pages 177–186, 2010.
5. L. Bottou and Y. LeCun. Large scale online learning. Advances in neural information processing systems, 16:217, 2004.
6. R. H. Byrd, S. L. Hansen, J. Nocedal, and Y. Singer. A stochastic quasi-Newton method for large-scale optimization. arXiv preprint arXiv:1401.7020, 2014.
7. A. Defazio, F. Bach, and S. Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems 27, pages 1646–1654, 2014.
8. R. S. Dembo and T. Steihaug. Truncated-Newton algorithms for large-scale unconstrained optimization. Mathematical Programming, 26(2):190–212, 1983.
9. R. S. Dembo, S. C. Eisenstat, and T. Steihaug. Inexact Newton methods. SIAM Journal on Numerical analysis, 19(2):400–408, 1982.
10. J. E. Dennis, Jr and J. J. Moré. Quasi-Newton methods, motivation and theory. SIAM review, 19(1): 46–89, 1977.
11. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011.
12. R. Frostig, R. Ge, S. M. Kakade, and A. Sidford. Competing with the empirical risk minimizer in a single pass. In Conference on Learning Theory, 2015.
13. R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pages 315–323, 2013.
14. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
15. D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. RCV1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361–397, 2004.
16. D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1-3):503–528, 1989.
17. A. Lucchi, B. McWilliams, and T. Hofmann. A variance reduced stochastic Newton method. arXiv preprint arXiv:1503.08316, 2015.
18. A. Mokhtari and A. Ribeiro. Global convergence of online limited memory BFGS. arXiv preprint arXiv:1409.2045, 2014a.
19. A. Mokhtari and A. Ribeiro. RES: Regularized stochastic BFGS algorithm. IEEE Transactions on Signal Processing, 62(23):6089–6104, 2014b.
20. Y. Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming, 120 (1):221–259, 2009.
21. J. Nocedal and S. J. Wright. Numerical Optimization. Springer, 2006.
22. B. A. Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, 6(1):147–160, 1994.
23. B. Recht and C. Ré. Parallel stochastic gradient algorithms for large-scale matrix completion. Mathematical Programming Computation, 5(2):201–226, 2013.
24. H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pages 400–407, 1951.
25. N. L. Roux, M. Schmidt, and F. R. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances in Neural Information Processing Systems, pages 2663–2671, 2012.
26. N. N. Schraudolph, J. Yu, and S. Günter. A stochastic quasi-Newton method for online convex optimization. In International Conference on Artificial Intelligence and Statistics, pages 436–443, 2007.
27. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss. The Journal of Machine Learning Research, 14(1):567–599, 2013.
28. J. Sohl-Dickstein, B. Poole, and S. Ganguli. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In International Conference on Machine Learning, 2014.
29. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, pages 1139–1147, 2013.
30. C. Wang, X. Chen, A. J. Smola, and E. P. Xing. Variance reduction for stochastic gradient optimization. In Advances in Neural Information Processing Systems, pages 181–189, 2013.
31. X. Wang, S. Ma, and W. Liu. Stochastic quasi-Newton methods for nonconvex stochastic optimization. arXiv preprint arXiv:1412.1196, 2014.