Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Advances in Neural Information Processing Systems

Volumn 2, Issue January, 2014, Pages 1017-1025

  Needell, Deanna ^a   Srebro, Nathan ^b   Ward, Rachel ^c  

^a CLAREMONT MCKENNA COLLEGE   (United States)

^b TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO   (United States)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; INFORMATION SCIENCE; STOCHASTIC SYSTEMS;

CONVEX OBJECTIVES; KACZMARZ ALGORITHM; LINEAR CONVERGENCE; LINEAR DEPENDENCE; QUADRATIC DEPENDENCE; SAMPLING DISTRIBUTION; STOCHASTIC GRADIENT DESCENT; STRONG CONVEXITIES;

IMPORTANCE SAMPLING;

EID: 84937892482     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. F. Bach and E. Moulines. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in Neural Information Processing Systems (NIPS), 2011.
2. Y. Censor, P. P. B. Eggermont, and D. Gordon. Strong underrelaxation in Kaczmarz's method for inconsistent systems. Numerische Mathematik, 41(1): 83-92, 1983.
3. R. Foygel and N. Srebro. Concentration-based guarantees for low-rank matrix reconstruction. 24th Ann. Conf. Learning Theory (COLT), 2011.
4. M. Hanke and W. Niethammer. On the acceleration of Kaczmarz's method for inconsistent linear systems. Linear Algebra and its Applications, 130: 83-98, 1990.
5. G. T. Herman. Fundamentals of computerized tomography: image reconstruction from projections. Springer, 2009.
6. G. N. Hounsfield. Computerized transverse axial scanning (tomography): Part 1. description of system. British Journal of Radiology, 46(552): 1016-1022, 1973.
7. S. Kaczmarz. Angenäherte auflösung von systemen linearer gleichungen. Bull. Int. Acad. Polon. Sci. Lett. Ser. A, pages 335-357, 1937.
8. N. Le Roux, M. W. Schmidt, and F. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. Advances in Neural Information Processing Systems (NIPS), pages 2672-2680, 2012.
9. F. Natterer. The mathematics of computerized tomography, Volume 32 of Classics in Applied Mathematics. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2001. ISBN 0-89871-493-1. doi: 10.1137/1.9780898719284. URL http://dx.doi.org/10.1137/1.9780898719284. Reprint of the 1986 original.
10. D. Needell. Randomized Kaczmarz solver for noisy linear systems. BIT, 50 (2): 395-403, 2010. ISSN 0006-3835. doi: 10.1007/s10543-010-0265-5. URL http://dx.doi.org/10.1007/s10543-010-0265-5.
11. D. Needell and R. Ward. Two-subspace projection method for coherent overdetermined linear systems. Journal of Fourier Analysis and Applications, 19(2): 256-269, 2013.
12. Arkadi Nemirovski. Efficient methods in convex programming. 2005.
13. Y. Nesterov. Introductory Lectures on Convex Optimization. Kluwer, 2004.
14. Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optimiz., 22(2): 341-362, 2012.
15. P. Richtárik and M. Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Math. Program., pages 1-38, 2012.
16. M. Schmidt, N. Roux, and F. Bach. Minimizing finite sums with the stochastic average gradient. arXiv preprint arXiv: 1309.2388, 2013.
17. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss. J. Mach. Learn. Res., 14(1): 567-599, 2013.
18. N. Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates. In Advances in Neural Information Processing Systems, 2010.
19. T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl., 15(2): 262-278, 2009. ISSN 1069-5869. doi: 10.1007/s00041-008-9030-4. URL http://dx.doi.org/10.1007/s00041-008-9030-4.
20. K. Tanabe. Projection method for solving a singular system of linear equations and its applications. Numerische Mathematik, 17(3): 203-214, 1971.
21. T. M. Whitney and R. K. Meany. Two algorithms related to the method of steepest descent. SIAM Journal on Numerical Analysis, 4(1): 109-118, 1967.
22. P. Zhao and T. Zhang. Stochastic optimization with importance sampling. Submitted, 2014.