A proximal stochastic gradient method with progressive variance reduction

SIAM Journal on Optimization

Volumn 24, Issue 4, 2014, Pages 2057-2075

  Xiao, Lin ^a   Zhang, Tong ^b,c  

^a MICROSOFT RESEARCH   (United States)

^b RUTGERS UNIVERSITY   (United States)

^c BAIDU INC   (China)

Author keywords

Proximal mapping; Stochastic gradient method; Variance reduction

Indexed keywords

ARTIFICIAL INTELLIGENCE; FUNCTIONS; GRADIENT METHODS; LEARNING SYSTEMS; MAPPING;

CONVEX FUNCTIONS; EMPIRICAL RISK MINIMIZATION; GEOMETRIC RATES; OBJECTIVE FUNCTIONS; PROXIMAL MAPPING; STOCHASTIC GRADIENT; STOCHASTIC GRADIENT METHODS; VARIANCE REDUCTIONS;

STOCHASTIC SYSTEMS;

EID: 84919793228     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/140961791     Document Type: Article

References (35)

1. A. BECK AND M. TEBOULLE, A fast iterative shrinkage-threshold algorithm for linear inverse problems, SIAM J. Imaging Sci., 2(2009), pp. 183-202.
2. D. P. BERTSEKAS, Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey, Report LIDS-P-2848, Laboratory for Information and Decision Systems, MIT, Cambridge, MA, 2010.
3. D. P. BERTSEKAS, Incremental proximal methods for large scale convex optimization, Math. Program. Ser. B, 129(2011), pp. 163-195.
4. J. A. BLACKARD, D. J. DEAN, AND C. W. ANDERSON, Covertype data set, in UCI Machine Learning Repository, K. Bache and M. Lichman, eds., http://archive.ics.uci.edu/ml (2013).
5. D. BLATT, A. O. HERO, AND H. GAUCHMAN, A convergent incremental gradient method with a constant step size, SIAM J. Optim., 18(2007), pp. 29-51.
6. R. H. BYRD, G. M. CHIN, J. NOCEDAL, AND Y. WU, Sample size selection in optimization methods for machine learning, Math. Program. Ser. B, 134(2012), pp. 127-155.
7. G. H.-G. CHEN AND R. T. ROCKAFELLAR, Convergence rates in forward-backward splitting, SIAM J. Optim., 7(1997), pp. 421-444.
8. J. DUCHI AND Y. SINGER, Efficient online and batch learning using forward backward splitting, J. Mach. Learn. Res., 10(2009), pp. 2873-2898.
9. R.-E. FAN AND C.-J. LIN, LIBSVM Data: Classification, Regression and Multi-label, http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets (2011).
10. M. P. FRIEDLANDER AND G. GOH, Tail Bounds for Stochastic Approximation, arXiv:1304.5586, 2013.
11. M. P. FRIEDLANDER AND M. SCHMIDT, Hybrid deterministic-stochastic methods for data fitting, SIAM J. Sci. Comput., 34(2012), pp. 1380-1405.
12. I. GUYON, Sido: A Pharmacology Dataset, http://www.causality.inf.ethz.ch/data/SIDO.html (2008).
13. T. HASTIE, R. TIBSHIRANI, AND J. FRIEDMAN, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer, New York, 2009.
14. C. HU, J. T. KWOK, AND W. PAN, Accelerated gradient methods for stochastic optimization and online learning, in Adv. Neural Inf. Process. Syst. 22, MIT Press, Cambridge, MA, 2009, pp. 781-789.
15. R. JOHNSON AND T. ZHANG, Accelerating stochastic gradient descent using predictive variance Reduction, in Adv. Neural Inf. Process. Syst. 26, MIT Press, Cambridge, MA, 2013, pp. 315-323.
16. J. KONEČNÝ AND P. RICHTÁRIK, Semi-Stochastic Gradient Descent Methods, arXiv:1312.1666, 2013.
17. J. LANGFORD, L. LI, AND T. ZHANG, Sparse online learning via truncated gradient, J. Mach. Learn. Res., 10(2009), pp. 777-801.
18. Y. T. LEE AND A. SIDFORD, Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems, arXiv:1305.1922, 2013.
19. D. D. LEWIS, Y. YANG, T. ROSE, AND F. LI, RCV 1: A new benchmark collection for text categorization research, J. Mach. Learn. Res., 5(2004), pp. 361-397.
20. P.-L. LIONS AND B. MERCIER, Splitting algorithms for the sum of two nonlinear operators, SIAM J. Numer. Anal., 16(1979), pp. 964-979.
21. M. MAHDAVI, L. ZHANG, AND R. JIN, Mixed optimization for smooth functions, in Adv. Neural Inf. Process. Syst. 26, MIT Press, Cambridge, MA, 2013, pp. 674-682.
22. D. NEEDELL, N. SREBRO, AND R. WARD, Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz Algorithm, preprint, arXiv:1310.5715, 2014.
23. YU. NESTEROV, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer, Boston, 2004.
24. Y. NESTEROV, Efficiency of coordinate descent methods on huge-scale optimization problems, SIAM J. Optim., 22(2012), pp. 341-362.
25. YU. NESTEROV, Gradient methods for minimizing composite functions, Math. Program. Ser. B, 140(2013), pp. 125-161.
26. P. RICHTÁRIK AND M. TAKÁČ, Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function, Math. Program., 144(2014), pp. 1-38.
27. R. T. ROCKAFELLAR, Convex Analysis, Princeton University Press, Princeton, NJ, 1970.
28. N. LE ROUX, M. SCHMIDT, AND F. BACH, A stochastic gradient method with an exponential convergence rate for finite training sets, in Adv. Neural Inf. Process. Syst. 25, MIT Press, Cambridge, MA, 2012, pp. 2672-2680.
29. M. SCHMIDT, N. LE ROUX, AND F. BACH, Minimizing Finite Sums with the Stochastic Average Gradient, Technical report HAL 00860051, INRIA, Paris, 2013.
30. S. SHALEV-SHWARTZ AND T. ZHANG, Proximal Stochatic Dual Coordinate Ascent, arXiv:1211.2772, 2012.
31. S. SHALEV-SHWARTZ AND T. ZHANG, Stochastic dual coordinate ascent methods for regularized loss minimization, J. Mach. Learn. Res., 14(2013), pp. 567-599.
32. P. TSENG, A modified forward-backward splitting method for maximal monotone mappings, SIAM J. Control Optim., 38(2000), pp. 431-446.
33. L. XIAO, Dual averaging methods for regularized stochastic learning and online optimization, J. Mach. Learn. Res., 11(2010), pp. 2534-2596.
34. L. ZHANG, M. MAHDAVI, AND R. JIN, Linear convergence with condition number independent access of full gradients, in Adv. Neural Inf. Process. Syst. 26, MIT Press, Cambridge, MA, 2013, pp. 980-988.
35. P. ZHAO AND T. ZHANG, Stochastic Optimization with Importance Sampling, arXiv:1401.2753, 2014.