Accelerated block-coordinate relaxation for regularized optimization

SIAM Journal on Optimization

Volumn 22, Issue 1, 2012, Pages 159-186

  Wright, Stephen J ^a  

^a University of Wisconsin Madison   (United States)

Author keywords

Active manifold identification; Block coordinate relaxation; Regularized optimization

Indexed keywords

ACCELERATION TECHNIQUE; BLOCK-COORDINATE RELAXATION; COMPUTATIONAL RESULTS; CRITICAL POINTS; LOGISTIC REGRESSIONS; MANIFOLD IDENTIFICATION; NON-DEGENERACY; REDUCED SPACE; REGULARIZATION FUNCTION; SMOOTH FUNCTIONS; SUB-PROBLEMS;

ALGORITHMS; LOGISTICS;

OPTIMIZATION;

EID: 84861594597     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/100808563     Document Type: Article

References (35)

1. S. Bakin, Adaptive Regression and Model Selection in Data Mining Problems, Ph.D. thesis, Australian National University, 1999.
2. R. H. Byrd, G. M. Chin, W. Neveitt, and J. Nocedal, On the Use of Stochastic Hessian Information in Unconstrained Optimization, Technical report, Northwestern University, 2010.
3. J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra, Efficient projections on the l1-ball for learning in higher dimensions, in Proceedings of the International Conference on Machine Learning, Helsinki, 2008.
4. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Ann. Statist., 32 (2004), pp. 407-499.
5. L. El Ghaoui, V. Viallon, and T. Rabbani, Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems, Technical report, EECS Department, University of California-Berkeley, Berkeley, CA, 2011.
6. M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems, IEEE J. Selected Topics in Signal Processing, 1 (2007), pp. 586-597.
7. J. Friedman, T. Hastie, and R. Tibshirani, Regularization Paths for Generalized Linear Models Via Coordinate Descent, Technical report, Department of Statistics, Stanford University, 2009.
8. E. T. Hale, W. Yin, and Y. Zhang, A fixed-point continuation method for l1-minimization: Methodology and convergence, SIAM J. Optim., 19 (2008), pp. 1107-1130.
9. W. Hare and A. Lewis, Identifying active constraints via partial smoothness and proxregularity, J. Convex Anal., 11 (2004), pp. 251-266.
10. W. L. Hare, A proximal method for identifying active manifolds, Comput. Optim. Appl., 43 (2009), pp. 295-306.
11. W. L. Hare, Identifying active manifolds in regularization problems, in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, Optim. and its Appl. 49, H. H. Bauschke, R. S. Burachik, P. L. Combettes, V. Elser, C. R. Luke, and H. Wolkowicz, eds., Springer, 2010, pp. 261-271.
12. J. Kim, Y. Kim, and Y. Kim, Gradient LASSO Algorithm, Technical report, Seoul National University, 2006.
13. Y. Kim, J. Kim, and Y. Kim, Blockwise sparse regression, Statist. Sinica, 16 (2006), pp. 375-390.
14. K. Koh, S.-J. Kim, and S. Boyd, An interior-point method for large-scale l1-regularized logistic regression, J. Mach. Learn. Res., 8 (2007), pp. 1519-1555.
15. A. Lewis, Active sets, nonsmoothness, and sensitivity, SIAM J. Optim., 13 (2003), pp. 702-725.
16. A. S. Lewis and S. J. Wright, A Proximal Method for Composite Minimization, Optimization technical report, University of Wisconsin-Madison, 2008.
17. L. Meier, S. van de Geer, and P. Buhlmann, The group lasso for logistic regression, J. Roy. Statist. Soc. B, 70 (2008), pp. 53-71.
18. J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed., Springer, New York, 2006.
19. B. Recht, M. Fazel, and P. Parrilo, Guaranteed minimum-rank solutions to linear matrix equations via nuclear norm minimization, SIAM Rev., 52 (2010), pp. 471-501.
20. R. T. Rockafellar, Convex Analysis, Princeton University Press, Princeton, NJ, 1970.
21. S. K. Shevade and S. S. Keerthi, A simple and efficient algorithm for gene selection using sparse logistic regression, Bioinformatics, 19 (2003), pp. 2246-2253.
22. J. Shi, W. Yin, S. Osher, and P. Sajda, A fast hybrid algorithm for large scale l1-regularized logistic regression, J. Mach. Learn. Res., 11 (2010), pp. 713-741.
23. W. Shi, G. Wahba, S. J. Wright, K. Lee, R. Klein, and B. Klein, LASSO-Patternsearch algorithm with application to opthalmology data, Stat. Interface, 1 (2008), pp. 137-153.
24. R. Tibshirani, Regression shrinkage and selection via the LASSO, J. Roy. Statist. Soc. B, 58 (1996), pp. 267-288.
25. P. Tseng, Approximation accuracy, gradient methods, and error bound for structured convex optimization, Math. Program. Ser. B, 125 (2010), pp. 263-295.
26. P. Tseng, Further results on a stable recovery of sparse overcomplete representations in the presence of noise, IEEE Trans. Inform. Theory, 55 (2009), pp. 888-899.
27. P. Tseng and S. Yun, A block-coordinate gradient descent method for linearly constrained nonsmooth separable optimization, J. Optim. Theory Appl., 140 (2009), pp. 513-535.
28. P. Tseng and S. Yun, A coordinate gradient descent method for nonsmooth separable minimization, Math. Program. Ser. B, 117 (2009), pp. 387-423.
29. P. Tseng and S. Yun, A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training, Comput. Optim. Appl., 47 (2010), pp. 179-206.
30. B. Turlach, W. N. Venables, and S. J. Wright, Simultaneous variable selection, Technometrics, 47 (2005), pp. 349-363.
31. I. Vaisman, A First Course in Differential Geometry, Monogr. Textbooks Pure Appl. Math., Marcel Dekker, New York, 1984.
32. S. J. Wright, Identifiable surfaces in constrained optimization, SIAM J. Control Optim., 31 (1993), pp. 1063-1079.
33. S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, Sparse reconstruction by separable approximation, IEEE Trans. Signal Process., 57 (2009), pp. 2479-2493.
34. G.-X. Yuan, K.-W. Chang, C.-J. Hsieh, and C. Lin, A Comparison of Optimization Methods for Large-Scale l1-Regularized Linear Classification, Technical report, Department of Computer Science, National Taiwan University, 2009.
35. M. Yuan and Y. Lin, Model selection and estimation in regression with grouped variables, J. Roy. Statist. Soc. B, 68 (2006), pp. 49-67.