A sparsity preserving stochastic gradient methods for sparse regression

Computational Optimization and Applications

Volumn 58, Issue 2, 2014, Pages 455-482

  Lin, Qihang ^a   Chen, Xi ^b   Peña, Javier ^c  

^a UNIVERSITY OF IOWA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

First order methods; Sparse regression; Stochastic gradient

Indexed keywords

CONVEX OPTIMIZATION; ITERATIVE METHODS; REGRESSION ANALYSIS; STOCHASTIC SYSTEMS;

FIRST-ORDER ALGORITHMS; FIRST-ORDER METHODS; LARGE DEVIATION PROPERTY; OBJECTIVE FUNCTIONS; OPTIMAL CONVERGENCE; SPARSE REGRESSION; STOCHASTIC GRADIENT; STOCHASTIC GRADIENT METHODS;

ALGORITHMS;

EID: 84901193687     PISSN: 09266003     EISSN: 15732894     Source Type: Journal    
DOI: 10.1007/s10589-013-9633-9     Document Type: Article

References (36)

1. Beck, A., Teboulle, M.: A fast iterative shrinkage-threshold algorithm for linear inverse problems. SIAM J. Image Sci. 2(1), 183-202 (2009)
2. Chung, K.L.: On a stochastic approximation method. Ann. Math. Stat. 25, 463-483 (1954)
3. Cotter, A., Shamir, O., Srebro, N., Sridharan, K.: Better mini-batch algorithms via accelerated gradient methods. In: Advances in Neural Information Processing Systems (NIPS) (2011)
4. Dekel, O., Gilad-Bachrach, R., Shamir, O., Xiao, L.: Optimal distributed online prediction using mini-batches. J. Mach. Learn. Res. 13, 165-202 (2012)
5. Duchi, J., Singer, Y.: Efficient online and batch learning using forward-backward splitting. J. Mach. Learn. Res. 10, 2873-2898 (2009)
6. Duchi, J.C., Bartlett, P.L., Wainwright, M.J.: Randomized smoothing for stochastic optimization. SIAM J. Optim. 22(2), 674-701 (2012)
7. Ermoliev, Y.: Stochastic quasigradient methods and their application to system optimization. Stochastics 9, 1-36 (1983)
8. Gaivoronski, A.A.: Nonstationary stochastic programming problems. Kybernetika 4, 89-92 (1978)
9. Hu, C., Kwok, J.T., Pan, W.: Accelerated gradient methods for stochastic optimization and online learning. In: Advances in Neural Information Processing Systems (NIPS) (2009)
10. Kiefer, J., Wolfowitz, J.: Stochastic estimation of the maximum of a regression function. Ann. Math. Stat. 23, 462-466 (1952)
11. Kushner, H.J., Yin, G.G.: Stochastic Approximation Algorithms and Applications. Springer, New York (2003)
12. Lan, G.: An optimal method for stochastic composite optimization. Math. Program. 133(1), 365-397 (2012)
13. Lan, G., Ghadimi, S.: Optimal stochastic approximation algorithm for strongly convex stochastic composite optimization, I: a generic algorithmic framework. SIAM J. Optim. 22(4), 1469-1492 (2012)
14. Lan, G., Ghadimi, S.: Optimal stochastic approximation algorithm for strongly convex stochastic composite optimization, II: shrinking procedures and optimal algorithms. SIAM J. Optim. 23(4), 2061-2069 (2013)
15. Langford, J., Li, L., Zhang, T.: Sparse online learning via truncated gradient. J. Mach. Learn. Res. 10, 2873-2908 (2009)
16. Lee, S., Wright, S.J.: Manifold identification in dual averaging methods for regularized stochastic online learning. J. Mach. Learn. Res. 13, 1705-1744 (2012)
17. 2 norm minimization. In: The Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI) (2009)
18. Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574-1609 (2009)
19. Nemirovski, A., Yudin, D.: On Cezari's convergence of the steepest descent method for approximating saddle point of convex-concave functions. Soviet Mathematics Doklady 19 (1978)
20. Nemirovski, A., Yudin, D.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)
21. Nesterov, Y.E.: Gradient methods for minimizing composite objective function. Technical report, CORE (2007)
22. Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer, Amsterdam (2003)
23. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127-152 (2005)
24. Nesterov, Y.: Primal-dual subgradient methods for convex problems. Math. Program. 120, 221-259 (2009)
25. Pflug, G.C.: Optimization of Stochastic Models, the Interface Between Simulation and Optimization. Kluwer, Boston (1996)
26. Polyak, B.T., Juditsky, A.B.: Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30, 838-855 (1992)
27. Polyak, B.T.: New stochastic approximation type procedures. Automat. Telemekh. 7, 98-107 (1990)
28. Pong, T.K., Tseng, P., Ji, S., Ye, J.: Trace norm regularization: reformulations, algorithms, and multi-task learning. SIAM J. Optim. 20(6), 3465-3489 (2010)
29. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22(3), 400-407 (1951)
30. Ruszczynski, A., Syski, W.: A method of aggregate stochastic subgradients with on-line stepsize rules for convex stochastic programming problems. Math. Program. Stud. 28, 113-131 (1986)
31. Sacks, J.: Asymptotic distribution of stochastic approximation. Ann. Math. Stat. 29, 373-409 (1958)
32. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B 58, 267-288 (1996)
33. Toh, K.-C., Yun, S.: An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems. Pac. J. Optim. 6, 615-640 (2010)
34. Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization. Technical report, University of Washington (2008)
35. Xiao, L.: Dual averaging methods for regularized stochastic learning and online optimization. J. Mach. Learn. Res. 11, 2543-2596 (2010)
36. Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. B 68, 49-67 (2006)