Adaptive Newton method for empirical risk minimization to statistical accuracy

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 4069-4077

  Mokhtari, Aryan ^a   Daneshmand, Hadi ^b   Lucchi, Aurelien ^b   Hofmann, Thomas ^b   Ribeiro, Alejandro ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b ETH ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE ALGORITHMS; NEWTON-RAPHSON METHOD; SAMPLING; SELF ORGANIZING MAPS;

EMPIRICAL RISK MINIMIZATION; LARGE-SCALE DATASETS; LOCAL NEIGHBORHOODS; NEWTON ITERATIONS; NUMBER OF SAMPLES; NUMERICAL EXPERIMENTS; QUADRATIC CONVERGENCE; STATISTICAL ACCURACY;

ITERATIVE METHODS;

EID: 85019113979     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. Peter L. Bartlett, Michael I. Jordan, and Jon D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 138-156, 2006.
2. Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1): 183-202, 2009.
3. Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems 20, Vancouver, British Columbia, Canada, pages 161-168, 2007.
4. Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004.
5. Hadi Daneshmand, Aurélien Lucchi, and Thomas Hofmann. Starting small - learning with adaptive sample sizes. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, pages 1463-1471, 2016.
6. Aaron Defazio, Francis R. Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems 27, Montreal, Quebec, Canada, pages 1646-1654, 2014.
7. Murat A. Erdogdu and Andrea Montanari. Convergence rates of sub-sampled Newton methods. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, Montreal, Quebec, Canada, pages 3052-3060, 2015.
8. Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford. Competing with the empirical risk mini-mizer in a single pass. In Proceedings of The 28th Conference on Learning Theory, COLT 2015, Paris, France, July 3-6, 2015, pages 728-763, 2015.
9. Mert Gürbüzbalaban, Asuman Ozdaglar, and Pablo Parrilo. A globally convergent incremental Newton method. Mathematical Programming, 151(1): 283-313, 2015.
10. Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems 26. Lake Tahoe, Nevada, United States, pages 315-323, 2013.
11. Aurelien Lucchi, Brian McWilliams, and Thomas Hofmann. A variance reduced stochastic newton method. arXiv, 2015.
12. Aryan Mokhtari and Alejandro Ribeiro. Res: Regularized stochastic BFGS algorithm. IEEE Transactions on Signal Processing, 62(23): 6089-6104, 2014.
13. Aryan Mokhtari and Alejandro Ribeiro. Global convergence of online limited memory BFGS. Journal of Machine Learning Research, 16: 3151-3181, 2015.
14. Philipp Moritz, Robert Nishihara, and Michael I. Jordan. A linearly-convergent stochastic L-BFGS algorithm. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, pages 249-258, 2016.
15. Yu Nesterov. Introductory Lectures on Convex Programming Volume I: Basic course. Citeseer, 1998.
16. Yurii Nesterov et al. Gradient methods for minimizing composite objective function. 2007.
17. Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4): 838-855, 1992.
18. Zheng Qu, Peter Richtárik, Martin Takác, and Olivier Fercoq. SDNA: stochastic dual Newton ascent for empirical risk minimization. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1823-1832, 2016.
19. Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pages 400-407, 1951.
20. Nicolas Le Roux, Mark W. Schmidt, and Francis R. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances in Neural Information Processing Systems 25. Lake Tahoe, Nevada, United States, pages 2672-2680, 2012.
21. Nicol N. Schraudolph, Jin Yu, and Simon Günter. A stochastic quasi-Newton method for online convex optimization. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, pages 436-443, 2007.
22. Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11: 2635-2670, 2010.
23. Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss. The Journal of Machine Learning Research, 14: 567-599, 2013.
24. Shai Shalev-Shwartz and Tong Zhang. Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. Mathematical Programming, 155(1-2): 105-145, 2016.
25. Vladimir Vapnik. The nature of statistical learning theory. Springer Science & Business Media, 2013.
26. Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization, 24(4): 2057-2075, 2014.