Least angle and ℓ 1 penalized regression: A review

Statistics Surveys

Volumn 2, Issue , 2008, Pages 61-93

  Hesterberg, Tim ^a   Choi, Nam Hee ^a   Meier, Lukas ^a   Fraley, Chris ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

1; Lasso; Penalty; Regression; Regularization; Variable selection

Indexed keywords

EID: 58349104057     PISSN: None     EISSN: 19357516     Source Type: Journal    
DOI: 10.1214/08-SS035     Document Type: Article

References (117)

1. Adams, J. L. (1990) A computer experiment to evaluate regression strategies. In Proceedings of the Statistical Computing Section, 55-62. American Statistical Association.
2. Avalos, M., Grandvalet, Y. and Ambroise, C. (2007) Parsimonious additivemodels. Computational Statistics and Data Analysis, 51, 2851-2870.
3. Bakin, S. (1999) Adaptive regression and model selection in data mining problems. Ph.D. thesis, The Australian National University.
4. Balakrishnan, S. and Madigan, D. (2007) Finding predictive runs with laps. International Conference on Machine Learning (ICML), 415-420.
5. Banerjee, O., El Ghaoui, L. and d'Aspremont, A. (2008) Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data. Journal of Machine Learning Research. (to appear).
6. Breiman, L. (1995) Better subset regression using the nonnegative garrote. Technometrics, 37, 373-384.
7. Bühlmann, P. and Meier, L. (2008) Discussion of "One-step sparse estimates in nonconcave penalized likelihood models" by H. Zou and R. Li. Annals of Statistics. (to appear).
8. Bühlmann, P. and Yu, B. (2006) Sparse boosting. Journal of Machine Learning Research, 7, 1001-1024.
9. Bunea, F., Tsybakov, A. and Wegkamp, M. H. (2007) Sparsity oracle inequalities for the Lasso. Electronic Journal of Statistics, 1, 169-194.
10. Candes, E. and Tao, T. (2007) The Dantzig selector: statistical estimation when p is much larer than n. Annals of Statistics, 35, 2313-2351.
11. Candes, E. J., Wakin, M. and Boyd, S. (2007) Enhancing sparsity by reweighted L1 minimization. Tech. rep., California Institute of Technology.
12. Chen, S., Donoho, D. and Saunders, M. (1998) Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20, 33-61.
13. Choi, N. H. and Zhu, J. (2006) Variable selection with strong heredity/ marginality constraints. Tech. rep., Department of Statistics, University of Michigan.
14. Dahl, J., Vandenberghe, L. and Roychowdhury, V. (2008) Covariance selection for non-chordal graphs via chordal embedding. Optimization Methods and Software. (to appear).
15. Draper, N. R. and Smith, H. (1998) Applied regression analysis. Wiley, 3rd edn.
16. Efron, B. and Hastie, T. (2003) LARS software for R and Splus. http://www-stat.stanford.edu/~hastie/Papers/LARS.
17. Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004) Least angle regression. Annals of Statistics, 32, 407-451.
18. Efron, B., Hastie, T. and Tibshirani, R. (2007) Discussion of "the Dantzig selector" by E. Candes and T. Tan. Annals of Statistics, 35, 2358-2364.
19. Efroymson, M. A. (1960) Multiple regression analysis. In Mathematical Methods for Digital Computers (eds. A. Ralston and H. S.Wilf), vol. 1, 191-203. Wiley.
20. Fan, J. and Li, R. (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96, 1348-1360.
21. Fraley, C. and Hesterberg, T. (2007) Least-angle regression and Lasso for large datasets. Tech. rep., Insightful Corporation.
22. Frank, I. E. and Friedman, J. H. (1993) A statistical view of some chemometrics regression tools, with discussion. Technometrics, 35, 109-148.
23. Freund, Y. and Schapire, R. E. (1997) A decision-theoretic generalization of online learning and an application to boosting. Journal of Computer and System Sciences, 55, 119-139.
24. Friedman, J. (2006) Herding lambdas: fast algorithms for penalized regression and classification. Manuscript.
25. Friedman, J. H. (1991) Mulivariate adaptive regression splines. Annals of Statistics, 19, 1-67.
26. Friedman, J. H., Hastie, T., Höfling, H. and Tibshirani, R. (2007a) Pathwise coordinate optimization. Annals of Applied Statistics, 1, 302-332.
27. Friedman, J. H., Hastie, T. and Tibshirani, R. (2007b) Sparse inverse covariance estimation with the graphical lasso. Biostatistics. (published online December 12, 2007).
28. Fu, W. (1998) Penalized regressions: the Bridge versus the Lasso. Journal of Computational and Graphical Statistics, 7, 397-416.
29. Fu, W. (2000) S-PLUS package brdgrun for shrinkage estimators with bridge penalty. http://lib.stat.cmu.edu/S/brdgrun.shar.
30. Furnival, G. M. and Wilson, Jr., R. W. (1974) Regression by leaps and bounds. Technometrics, 16, 499-511.
31. Gao, H.-Y. (1998) Wavelet shrinkage denoising using the non-negative garrote. Journal of Computational and Graphical Statistics, 7, 469-488.
32. Genkin, A., Lewis, D. D. and Madigan, D. (2007) Large-scale Bayesian logistic regression for text categorization. Technometrics, 49, 291-304.
33. Ghosh, S. (2007) Adaptive elastic net: An improvement of elastic net to achieve oracle properties. Tech. rep., Department of Mathematical Sciences, Indiana University-Purdue University, Indianapolis.
34. Greenshtein, E. and Ritov, Y. (2004) Persistency in high dimensional linear predictor-selection and the virtue of over-parametrization. Bernoulli, 10, 971-988.
35. Gui, J. and Li, H. (2005) Penalized Cox regression analysis in the highdimensional and low-sample size settings, with applications to microarray gene expression data. Bioinformatics, 21, 3001-3008.
36. Guigue, V., Rakotomamonjy, A. and Canu, S. (2006) Kernel basis pursuit. Revue d'Intelligence Artificielle, 20, 757-774.
37. Gunn, S. R. and Kandola, J. S. (2002) Structural modeling with sparse kernels. Machine Learning, 10, 581-591.
38. Hamada, M. and Wu, C. (1992) Analysis of designed experiments with complex aliasing. Journal of Quality Technology, 24, 130-137.
39. Hastie, T., Rosset, S., Tibshirani, R. and Zhu, J. (2004) The entire regularization path for the support vector machine. Journal of Machine Learning Research, 5, 1391-1415. 3/5/04.
40. Hastie, T., Taylor, J., Tibshirani, R. and Walther, G. (2007) Forward stagewise regression and the monotone lasso. Electronic Journal of Statistics, 1, 1-29.
41. Hastie, T., Tibshirani, R. and Friedman, J. (2001) The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer Verlag.
42. Hesterberg, T. and Fraley, C. (2006a) Least angle regression. Proposal to NIH, http://www.insightful.com/lars.
43. Hesterberg, T. and Fraley, C. (2006b) S-PLUS and R package for least angle regression. In Proceedings of the American Statistical Association, Statistical Computing Section [CD-ROM], 2054-2061. Alexandria, VA: American Statistical Association.
44. Huang, J., Ma, S. and Zhang, C.-H. (2008) Adaptive Lasso for sparse highdimensional regression models. Statisica Sinica. (to appear).
45. Hurvich, C.M. and Tsai, C.-L. (1990) The impact of model selection on inference in linear regression. The American Statistician, 44, 214-217.
46. Insightful Corportation (2006) GLARS: Generalized Least Angle Regression software for R and S-PLUS. http://www.insightful.com/lars.
47. Ishwaran, H. (2004) Discussion of "Least Angle Regression" by Efron et al. Annals of Statistics, 32, 452-458.
48. Jolliffe, I., Trendafilov, N. and Uddin, M. (2003) A modified principal component technique based on the LASSO. Journal of Computational and Graphical Statistics, 12, 531-547.
49. Keerthi, S. and Shevade, S. (2007) A fast tracking algorithm for generalized lars/lasso. IEEE Transactions on Neural Networks, 18, 1826-1830.
50. Khan, J. A., Van Aelst, S. and Zamar, R. H. (2007) Robust linear model selection based on least angle regression. Journal of the American Statistical Association, 102, 1289-1299.
51. Kim, J., Kim, Y. and Kim, Y. (2005a) glasso: R-package for Gradient LASSO algorithm. http://idea.snu.ac.kr/Research/glassojskim/glasso.htm.
52. R package version 0.9, http://idea.snu.ac.kr/Research/glassojskim/glasso.htm.
53. Kim, J., Kim, Y. and Kim, Y. (2005b) Gradient LASSO algorithm. Technical report, Seoul National University.
54. Kim, Y., Kim, J. and Kim, Y. (2006) Blockwise sparse regression. Statistica Sinica, 16, 375-390.
55. Knight, K. (2004) Discussion of "Least Angle Regression" by Efron et al. Annals of Statistics, 32, 458-460.
56. Leng, C., Lin, Y. and Wahba, G. (2006) A note on the LASSO and related procedures in model selection. Statistica Sinica, 16, 1273-1284.
57. Lin, Y. and Zhang, H. (2006) Component selection and smoothing in multivariate nonparametric regression. Annals of Statistics, 34, 2272-2297.
58. Lokhorst, J. (1999) The LASSO and Generalised Linear Models. Honors Project, The University of Adelaide, Australia.
59. Lokhorst, J., Venables, B. and Turlach, B. (1999) Lasso2: L1 Constrained Estimation Routines. http://www.maths.uwa.edu.au/ berwin/software/lasso.html.
60. Loubes, J. and Massart, P. (2004) Discussion of "least angle regression" by Efron et al. Annals of Statistics, 32, 460-465.
61. Lu, W. and Zhang, H. (2007) Variable selection for proportional odds model. Statistics in Medicine, 26, 3771-3781.
62. Madigan, D. and Ridgeway, G. (2004) Discussion of "least angle regression" by Efron et al. Annals of Statistics, 32, 465-469.
63. McCullagh, P. and Nelder, J. A. (1989) Generalised Linear Models. London: Chapman and Hall.
64. Meier, L. and Bühlmann, P. (2007) Smoothing ℓ1-penalized estimators for highdimensional time-course data. Electronic Journal of Statistics, 1, 597-615.
65. Meier, L., van de Geer, S. and Bühlmann, P. (2008) The group lasso for logistic regression. Journal of the Royal Statistical Society, Series B, 70, 53-71.
66. Meinshausen, N. (2007) Lasso with relaxation. Computational Statistics and Data Analysis, 52, 374-393.
67. Meinshausen, N. and Bühlmann, P. (2006) High dimensional graphs and variable selection with the lasso. Annals of Statistics, 34, 1436-1462.
68. Meinshausen, N., Rocha, G. and Yu, B. (2007) A tale of three cousins: Lasso, L2Boosting, and Dantzig. Annals of Statistics, 35, 2373-2384.
69. Meinshausen, N. and Yu, B. (2008) Lasso-type recovery of sparse representations for high-dimensional data. Annals of Statistics. (to appear).
70. Miller, A. (2002) Subset Selection in Regression. Chapman & Hall, second edn.
71. Monahan, J. F. (2001) Numerical Methods of Statistics. Cambridge University Press.
72. Osborne, M. R., Presnell, B. and Turlach, B. A. (2000a) A new approach to variable selection in least squares problems. IMA Journal of Numerical Analysis, 20, 389-403.
73. Osborne, M. R., Presnell, B. and Turlach, B. A. (2000b) On the LASSO and its dual. Journal of Computational and Graphical Statistics, 9, 319-337.
74. Owen, A. (2006) A robust hybrid of lasso and ridge regression. From the web.
75. Park, M. Y. and Hastie, T. (2006a) glmpath: L1 Regularization Path for Generalized Linear Models and Proportional Hazards Model. URL http://cran.r-project.org/src/contrib/Descriptions/glmpath.html. R package version 0.91.
76. Park, M. Y. and Hastie, T. (2006b) Regularization path algorithms for detecting gene interactions. Tech. rep., Department of Statistics, Stanford University.
77. Park, M.-Y. and Hastie, T. (2007) An L1 regularization-path algorithm for generalized linear models. Journal of the Royal Statistical Society Series B, 69, 659-677.
78. Roecker, E. B. (1991) Prediction error and its estimation for subset-selected models. Technometrics, 33, 459-468.
79. Rosset, S. (2005) Following curved regularized optimization solution paths. In Advances in Neural Information Processing Systems 17 (eds. L. K. Saul, Y. Weiss and L. Bottou), 1153-1160. Cambridge, MA: MIT Press.
80. Rosset, S. and Zhu, J. (2004) Discussion of "Least Angle Regression" by Efron et al. Annals of Statistics, 32, 469-475.
81. Rosset, S. and Zhu, J. (2007) Piecewise linear regularized solution paths. Annals of Statistics, 35, 1012-1030.
82. Roth, V. (2004) The generalized LASSO. IEEE Transactions on Neural Networks, 15, 16-28.
83. Segal, M. R., Dahlquist, K. D. and Conklin, B. R. (2003) Regression approaches for microarray data analysis. Journal of Computational Biology, 10, 961-980.
84. Shi, W., Wahba, G., Wright, S., Lee, K., Klein, R. and Klein, B. (2008) Lassopatternsearch algorithmwith application to ophthalmology and genomic data. Statistics and Its Interface. (to appear).
85. Silva, J., Marques, J. and Lemos, J. (2005) Selecting landmark points for sparse manifold learning. In Advances in Neural Information Processing Systems 18 (eds. Y. Weiss, B. Schölkopf and J. Platt), 1241-1248. Cambridge, MA: MIT Press.
86. Similä, T. and Tikka, J. (2006) Common subset selection of inputs in multiresponse regression. In IEEE International Joint Conference on Neural Networks, 1908-1915. Vancouver, Canada.
87. Stine, R. A. (2004) Discussion of "Least Angle Regression" by Efron et al. Annals of Statistics, 32, 475-481.
88. Thisted, R. A. (1988) Elements of Statistical Computing. Chapman and Hall.
89. Tibshirani, R. (1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58, 267-288.
90. Tibshirani, R. (1997) The lasso method for variable selection in the Cox model. Statistics in Medicine, 16, 385-395.
91. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005) Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society, Series B, 67, 91-108.
92. Trendafilov, N. and Joilliffe, I. (2007) Dlass: Variable selection in discriminant analysis via the lasso. Computational Statistics and Data Analysis, 51, 3718-3736.
93. Turlach, B. A. (2004) Discussion of "Least Angle Regression" by Efron et al. Annals of Statistics, 32, 481-490.
94. Turlach, B. A., Venables, W. N. and Wright, S. J. (2005) Simultaneous variable selection. Technometrics, 47, 349-363.
95. van de Geer, S. (2008) High-dimensional generalized linear models and the lasso. Annals of Statistics, 36, 614-645.
96. Wang, G., Yeung, D.-Y. and Lochovsky, F. (2007a) The kernel path in kernelized LASSO. In International Conference on Artificial Intelligence and Statistics. San Juan, Puerto Rico.
97. Wang, H. and Leng, C. (2006) Improving grouped variable selection via aglasso. Tech. rep., Peking University & National University of Singapore.
98. Wang, H. and Leng, C. (2007) Unified LASSO estimation via least squares approximation. Journal of the American Statistical Association, 102, 1039-1048.
99. Wang, H., Li, G. and Tsai, C. (2007b) Regression coefficient and autoregressive order shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 69, 63-78.
100. Yuan, M. (2008) Efficient computation of the ℓ1 regularized solution path in Gaussian graphical models. Journal of Computational and Graphical Statistics. (to appear).
101. Yuan, M., Joseph, R. and Lin, Y. (2007) An efficient variable selection approach for analyzing designed experiments. Technometrics, 49, 430-439.
102. Yuan, M. and Lin, Y. (2006) Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, Series B, 68, 49-68.
103. Yuan, M. and Lin, Y. (2007a) Model selection and estimation in the Gaussian graphical model. Biometrika, 94, 19-35.
104. Yuan, M. and Lin, Y. (2007b) On the non-negative garrote estimator. Journal of the Royal Statistical Society, Series B, 69, 143-161.
105. Zhang, C.-H. and Huang, J. (2007) The sparsity and bias of the lasso selection in high-dimensional linear regression. Annals of Statistics. To appear.
106. Zhang, H. and Lu, W. (2007) Adaptive Lasso for Cox's proportional hazards model. Biometrika, 94, 691-703.
107. Zhang, H., Wahba, G., Lin, Y., Voelker, M., Ferris, M., Klein, R. and Klein, B. (2004) Variable selection and model building via likelihood basis pursuit. Journal of the American Statistical Association, 99, 659-672.
108. Zhao, P., Rocha, G. and Yu, B. (2008) Grouped and hierarchical model selection through composite absolute penalties. Annals of Statistics. (to appear).
109. Zhao, P. and Yu, B. (2006) On model selection consistency of Lasso. Journal of Machine Learning Research, 7, 2541-2567.
110. Zhao, P. and Yu, B. (2007) Stagewise Lasso. Journal of Machine Learning Research, 8, 2701-2726.
111. Zhu, J., Rosset, S., Hastie, T. and Tibshirani, R. (2003) 1-norm support vector machines. In Advances in Neural Information Processing Systems 16, 49-56. MIT Press. NIPS 2003 Proceedings.
112. Zou, H. (2006) The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101, 1418-1429.
113. Zou, H. and Hastie, T. (2005a) elasticnet: Elastic Net Regularization and Variable Selection. R package version 1.0-3.
114. Zou, H. and Hastie, T. (2005b) Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67, 301-320.
115. Zou, H., Hastie, T. and Tibshirani, R. (2006) Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15, 265-286.
116. Zou, H., Hastie, T. and Tibshirani, R. (2007) On the "Degrees of Freedom" of the Lasso. Annals of Statistics, 35, 2173-2192.
117. Zou, H. and Li, R. (2008) One-step sparse estimates in nonconcave penalized likelihood models. Annals of Statistics. (to appear).