Nearly unbiased variable selection under minimax concave penalty

Annals of Statistics

Volumn 38, Issue 2, 2010, Pages 894-942

  Zhang, Cun Hui ^a  

^a RUTGERS UNIVERSITY   (United States)

Author keywords

Correct selection; Degrees of freedom; Least squares; Mean squared error; Minimax; Model selection; Nonconvex minimization; Penalized estimation; Risk estimation; Selection consistency; Sign consistency; Unbiasedness; Variable selection

Indexed keywords

EID: 77649284492     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/09-AOS729     Document Type: Article

References (54)

1. AKAIKE, H. (1973). Information theory and an extension of the maximum likelihood principle. In Proc. 2nd International Symposium on Information Theory (V. Petrov and F. Csáki, eds.) 267-281. Akadmiai Kiadó, Budapest.
2. ANTONIADIS, A. and FAN, J. (2001). Regularized wavelet approximations (with discussion). J. Amer. Statist. Assoc. 96 939-967.
3. BACH, F. R. (2008). Bolasso: Model consistent Lasso estimation through the bootstrap. In Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008, Helsinki, Finland) (A. McCallum and S. Roweis, eds.) 33-40.
4. BUNEA, F., TSYBAKOV, A. and WEGKAMP, M. (2007). Sparsity oracle inequalities for the lasso. Electron. J. Stat. 1 169-194 (electronic).
5. CANDÉS, E. and TAO, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51 4203-4215.
6. CANDÉS, E. and TAO, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n (with discussion). Ann. Statist. 35 2313-2404.
7. CHEN, S. and DONOHO, D.L. (1994). On basis pursuit. Technical report, Dept. Statistics, Stanford Univ.
8. DAVIDSON, K. and SZAREK, S. (2001). Local operator theory, random matrices and Banach spaces. In Handbook on the Geometry of Banach Spaces (W. B. Johnson and J. Lindenstrauss, eds.) I 317-366. North-Holland, Amsterdam.
9. DONOHO, D. L. and JOHNSTONE, I. (1994a). Minimax risk over ℓp-balls for ℓq -error. Probab. Theory Related Fields 99 277-303.
10. DONOHO, D. L. and JOHNSTONE, I. M. (1994b). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 425-455.
11. DONOHO, D. L., JOHNSTONE, I. M., HOCH, J. C. and STERN, A. S. (1992). Maximum entropy and the nearly black object (with discussion). J. Roy. Statist. Soc. Ser. B 54 41-81.
12. EFRON, B. (1986). How biased is the apparent error of a prediction rule? J. Amer. Statist. Assoc. 81 461-470.
13. EFRON, B., HASTIE, T., JOHNSTONE, I. and TIBSHIRANI, R. (2004). Least angle regression (with discussion). Ann. Statist. 32 407-499.
14. EFRON, B., HASTIE, T. and TIBSHIRANI, R. (2007). Discussion: The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2358-2364.
15. FAN, J. (1997). Comments on "Wavelets in statistics: A review" by A. Antoniadis. J. Italian Statist. Assoc. 6 131-138.
16. FAN, J. and LI, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360.
17. FAN, J. and LV, JINCHI. (2008). Sure independence screening for ultrahigh-dimensional feature space. J. Roy. Statist. Soc. Ser. B 70 849-911.
18. FAN, J. and PENG, H. (2004). On nonconcave penalized likelihood with diverging number of parameters. Ann. Statist. 32 928-961.
19. FOSTER, D. P. and GEORGE, E. I. (1994). The risk inflation criterion for multiple regression. Ann. Statist. 22 1947-1975. MR1329177
20. FREUND, Y. and SCHAPIRE, R. E. (1996). Experiments with a new boosting algorithm. In Machine Learning: Proceedings of the Thirteenth International Conference 148-156. Morgan Kauffmann, San Francisco.
21. FRIEDMAN, J., HASTIE, T. and TIBSHIRANI, R. (2000). Additive logistic regression: A statistical view of boosting (with discussion). Ann. Statist. 28 337-1307.
22. GAO, H.-Y. and BRUCE, A. G. (1997). Waveshrink with firm shrinkage. Statist. Sinica 7 855-874.
23. GENKIN, A. LEWIS, D. D. andMADIGAN, D. (2004). Large-scale Bayesian logistic regression for text categorization. Technical report, DIMACS, Rutgers Univ.
24. GREENSHTEIN E. and RITOV Y. (2004). Persistence in high-dimensional linear predictor selection and the virtue of overparametrization. Bernoulli 10 971-988.
25. HUANG, J.,MA, S. and ZHANG, C.-H. (2008). Adaptive Lasso for sparse high-dimensional regression models. Statist. Sinica 18 1603-1618.
26. HUNTER, D. R. and LI, R. (2005). Variable selection using MM algorithms. Ann. Statist. 33 1617-1642.
27. MALLOWS, C. L. (1973). Some comments on Cp. Technometrics 12 661-675.
28. MEINSHAUSEN, N. (2007). Relaxed Lasso. Comput. Statist. Data Anal. 52 374-393.
29. MEINSHAUSEN, N. and BUHLMANN, P. (2006). High-dimensional graphs and variable selection with the Lasso. Ann. Statist. 34 1436-1462.
30. MEINSHAUSEN, N., ROCHA, G. and YU, B. (2007). Discussion: The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2373-2384.
31. MEINSHAUSEN, N. and YU, B. (2009). Lasso-type recovery of sparse representations for highdimensional data. Ann. Statist. 37 2246-2270.
32. MEYER, M. andWOODROOFE, M. (2000). On the degrees of freedom in shape-restricted regression. Ann. Statist. 28 1083-1104.
33. OSBORNE, M., PRESNELL, B. and TURLACH, B. (2000a). A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20 389-404.
34. OSBORNE, M., PRESNELL, B. and TURLACH, B. (2000b). On the lasso and its dual. J. Comput. Graph. Statist. 9 319-337.
35. PARK, M. Y. and HASTIE, T. (2007). An L1 regularization-path algorithm for generalized linear models. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 659-677.
36. ROSSET, S. and ZHU, J. (2007). Piecewise linear regularized solution paths. Ann. Statist. 35 1012-1030.
37. SCHAPIRE, R. E. (1990). The strength of weak learnability. Machine Learning 5 197-227.
38. SCHWARZ, G. (1978). Estimating the dimension of a model. Ann. Statist. 6 461-464.
39. STEIN, C. (1981). Estimation of the mean of a multivariate normal distribution. Ann. Statist. 9 1135-1151. MR0630098
40. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the Lasso. J. Roy. Statist. Soc. Ser. B 58 267-288. MR1379242
41. TROPP, J. A. (2006). Just relax: Convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory 52 1030-1051.
42. VAN DE GEER, S. (2008). High-dimensional generalized linear models and the Lasso. Ann. Statist. 36 614-645.
43. WAINWRIGHT, M. (2006). Sharp thresholds for high-dimensional and noisy recovery of sparsity. Technical Report 708, Dept. Statistics, Univ. California, Berkeley.
44. YE, F. and ZHANG, C.-H. (2009). Rate Minimaxity of the Lasso and Dantzig Estimators. Technical Report No. 2009-3001 Dept. Statistics, Rutgers Univ.
45. YUAN, M. and LIN, Y. (2007). On the nonnegative garrote estimator. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 143-161.
46. ZHANG, C.-H. (2007a). Continuous generalized gradient descent. J. Comput. Graph. Statist. 16 761-781.
47. ZHANG, C.-H. (2007b). Penalized linear unbiased selection. Technical Report 2007-3003 Dept. Statistics, Rutgers Univ.
48. ZHANG, C.-H. (2007c). Information-theoretic optimality of variable selection with concave penalty. Technical Report No. 2007-2008 Dept. Statistics, Rutgers Univ.
49. ZHANG, C.-H. (2008). Discussion: One-step sparse estimates in nonconcave penalized likelihood models. Ann. Statist. 36 1553-1560.
50. ZHANG, C.-H. and HUANG, J. (2008). The sparsity and bias of the LASSO selection in highdimensional regression. Ann. Statist. 36 1567-1594.
51. ZHAO, P. and YU, B. (2006). On model selection consistency of LASSO. J. Mach. Learn. Res. 7 2541-2567.
52. ZHAO, P. and YU, B. (2007). Stagewise Lasso. J.Mach. Learn. Res. 8 2701-2726.
53. ZOU, H. (2006). The adaptive Lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418-1429.
54. ZOU, H. and LI, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models (with discussion). Ann. Statist. 36 1509-1533.