A selective overview of variable selection in high dimensional feature space

Statistica Sinica

Volumn 20, Issue 1, 2010, Pages 101-148

  Fan, Jianqing ^a   Lv, Jinchi ^b  

^a Princeton University   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Dimensionality reduction; Folded concave penalty; High dimensionality; LASSO; Model selection; Oracle property; Penalized least squares; Penalized likelihood; SCAD; Sure independence screening; Sure screening; Variable selection

Indexed keywords

EID: 77949352853     PISSN: 10170405     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Review

References (135)

1. Abramovich, F., Benjamini, Y., Donoho, D. and Johnstone, I. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34, 584-653.
2. Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (Edited by B. N. Petrov and F. Csaki), 267-281. Akademiai Kiado, Budapest.
3. Akaike, H. (1974). A new look at the statistical model identification. IEEE Trans. Auto. Control 19, 716-723.
4. Antoniadis, A. (1996). Smoothing noisy data with tapered coiflets series. Scand. J. Statist. 23, 313-330.
5. Antoniadis, A. and Fan, J. (2001). Regularization of wavelets approximations (with discussion). J. Amer. Statist. Assoc. 96, 939-967.
6. Antoniadis, A., Lambert-Lacroix, S. and Leblanc, F. (2003). Effective dimension reduction methods for tumor classification using gene expression data. Biostatistics 19, 563-570.
7. Bai, Z. D. (1999). Methodologies in spectral analysis of large dimensional random matrices, a review. Statist. Sinica 9, 611-677.
8. Bai, Z. D. and Yin, Y. Q. (1993). Limit of smallest eigenvalue of a large dimensional sample covariance matrix. Ann. Probab. 21, 1275-1294.
9. Barron, A., Birge, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113, 301-413.
10. Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57, 289-300.
11. Bickel, P. J. (1983). Minimax estimation of a normal mean subject to doing well at a point. In Recent Advances in Statistics (Edited by M. H. Rizvi, J. S. Rustagi and D. Siegmund), 511-528. Academic Press, New York.
12. Bickel, P. J. (2008). Discussion of "Sure independence screening for ultrahigh dimensional feature space". J. Roy. Statist. Soc. Ser. B 70, 883-884.
13. Bickel, P. J. and Levina, E. (2004). Some theory for Fisher's linear discriminant function, "naive Bayes", and some alternatives when there are many more variables than observations.Bernoulli 10, 989-1010.
14. Bickel, P. J. and Li, B. (2006). Regularization in statistics (with discussion). Test 15, 271-344.
15. Bickel, P. J., Ritov, Y. and Tsybakov, A. (2008). Simultaneous analysis of LASSO and Dantzig selector. Ann. Statist., to appear.
16. Boulesteix, A. (2004). PLS Dimension reduction for classification with microarray data. Statist. Appl. Genet. Mol. Biol. 3, 1-33.
17. Bradic, J., Fan, J. and Wang, W. (2009). Penalized composite quasi-likelihood for highdimensional variable selection. Manuscript.
18. Breiman, L. (1995). Better subset regression using the non-negative garrote. Technometrics 37, 373-384.
19. Breiman, L. (1996). Heuristics of instability and stabilization in model selection. Ann. Statist. 24, 2350-2383.
20. Bunea, F., Tsybakov, A. and Wegkamp, M. H. (2007). Sparsity oracle inequalities for the LASSO. Elec. Jour. Statist. 1, 169-194.
21. Bura, E. and Pfeiffer, R. M. (2003). Graphical methods for class prediction using dimension reduction techniques on DNA microarray data. Bioinformatics 19, 1252-1258.
22. Cai, T. and Lv, J. (2007). Discussion: The Dantzig selector: statistical estimation when p is much larger than n. Ann. Statist. 35, 2365-2369.
23. 1 minimization. IEEE Trans. Inform. Theory, to appear.
24. Candes, E. J. and Tao, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51, 4203-4215.
25. Candes, E. J. and Tao, T. (2006). Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. Inform. Theory 52, 5406-5425.
26. Candes, E. J. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n (with discussion). Ann. Statist. 35, 2313-2404.
27. 1 minimization. J. Fourier Anal. Appl. 14, 877-905.
28. Cao, H. Y. (2007). Moderate deviations for two sample t-statistics. ESAIM, Probab. Stat. 11, 264-627.
29. Chen, S., Donoho, D. L. and Saunders, M. (1999). Atomic decomposition by basis pursuit. SIAM, J. Sci. Comput. 20, 33-61.
30. Chiaromonte, F. and Martinelli, J. (2002). Dimension reduction strategies for analyzing global gene expression data with a response. Math. Biosci. 176, 123-144.
31. Claerbout, J. F. and Muir, F. (1973). Robust modeling of erratic data. Geophysics 38, 826-844.
32. Clarke, S. and Hall, P. (2009). Robustness of multiple testing procedures against dependence. Ann. Statist. 37, 332-358.
33. Daubechies, I., Defrise, M. and De Mol, C. (2004). An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math. 57, 1413-1457.
34. Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. Ser. B 39, 1-38.
35. Donoho, D. L. (2000). High-dimensional data analysis: The curses and blessings of dimensionality. Aide-Memoire of a Lecture at AMS Conference on Math Challenges of the 21st Century.
36. Donoho, D. L. (2004). Neighborly polytopes and sparse solution of underdetermined linear equations. Technical Report, Department of Statistics, Stanford University.
37. Donoho, D. L. (2006a). Compressed sensing. IEEE Trans. Inform. Theory 52, 1289-1306.
38. 1-norm solution is the sparsest solution. Comm. Pure Appl. Math. 59, 797-829.
39. 1 minimization. Proc. Nat. Acad. Sci. 100, 2197-2202.
40. Donoho, D. L., Elad, M. and Temlyakov, V. (2006). Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inform. Theory 52, 6-18.
41. Donoho, D. and Huo, X. (2001). Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inform. Theory 47, 2845-2862.
42. Donoho, D. and Jin, J. (2006). Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data. Ann. Statist. 34, 2980-3018.
43. Donoho, D. and Jin, J. (2008). Higher criticism thresholding: optimal feature selection when useful features are rare and weak. Proc. Nat. Acad. Sci. 105, 14790-14795.
44. Donoho, D. L. and Johnstone, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425-455.
45. Donoho, D. L. and Stark, P. B. (1989). Uncertainty principles and signal recovery. SIAM J. Appl. Math. 49, 906-931.
46. Dudoit, S., Shaffer, J. P. and Boldrick, J. C. (2003). Multiple hypothesis testing in microarray experiments. Statist. Sci. 18, 71-103.
47. Efron, B. (2007). Correlation and large-scale simultaneous significance testing. J. Amer. Statist. Assoc. 102, 93-103.
48. Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression (with discussion). Ann. Statist. 32, 407-499.
49. Efron, B., Hastie, T. and Tibshirani, R. (2007). Discussion: The Dantzig selector: statistical estimation when p is much larger than n. Ann. Statist. 35, 2358-2364.
50. Efron, B., Tibshirani, R., Storey, J. and Tusher, V. (2001). Empirical Bayes analysis of a microarray analysis experiment. J. Amer. Statist. Assoc. 99, 96-104.
51. Fan, J. (1997). Comments on "Wavelets in statistics: A review" by A. Antoniadis. J. Ital. Statist. Soc. 6, 131-138.
52. Fan, J. and Fan, Y. (2008). High-dimensional classification using features annealed independence rules. Ann. Statist. 36, 2605-2637.
53. Fan, J., Fan, Y. and Lv, J. (2008). High dimensional covariance matrix estimation using a factor model. J. Econometrics 147, 186-197.
54. Fan, J., Fan, Y. and Wu, Y. (2010). High dimensional classification. To appear in Highdimensional Statistical Inference (Edited by T. Cai and X. Shen). World Scientific, New Jersey.
55. Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96, 1348-1360.
56. Fan, J. and Li, R. (2006). Statistical challenges with high dimensionality: Feature selection in knowledge discovery. Proceedings of the International Congress of Mathematicians (Edited by M. Sanz-Sole, J. Soria, J. L. Varona and J. Verdera), Vol. Ill, 595-622.
57. Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space (with discussion). J. Roy. Statist. Soc. Ser. B 70, 849-911.
58. Fan, J. and Lv, J. (2009). Properties of non-concave penalized likelihood with NP-dimensionality. Manuscript.
59. Fan, J. and Peng, H. (2004). Nonconcave penalized likelihood with diverging number of parameters. Ann. Statist. 32, 928-961.
60. Fan, J. and Ren, Y. (2006). Statistical analysis of DNA microarray data. Clinical Cancer Research 12, 4469-4473.
61. Fan, J., Samworth, R. and Wu, Y. (2009). Ultrahigh dimensional variable selection: beyond the linear model. J. Mach. Learn. Res. 10, 1829-1853.
62. Fan, J. and Song, R. (2009). Sure independence screening in generalized linear models with NP-dimensionality. Revised for Ann. Statist.
63. Foster, D. and George, E. (1994). The risk inflation criterion for multiple regression. Ann. Statist. 22, 1947-1975.
64. Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109-148.
65. Friedman, J., Hastie, T. and Tibshirani, R. (2007). Sparse inverse covariance estimation with the LASSO. Manuscript.
66. Fu, W. J. (1998). Penalized regression: the bridge versus the LASSO. J. Comput. Graph. Statist. 7, 397-416.
67. Fuchs, J.-J. (2004). Recovery of exact sparse representations in the presence of noise. Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, 533-536.
68. Ghosh, D. (2002). Singular value decomposition regression modeling for classification of tumors from microarray experiments. Proceedings of the Pacific Symposium on Biocomputing, 11462-11467.
69. Girosi, F. (1998). An equivalence between sparse approximation and support vector machines. Neural Comput. 10, 1455-1480.
70. 1 constraint. Ann. Statist. 34, 2367-2386.
71. Greenshtein, E. and Ritov, Y. (2004). Persistence in high-dimensional predictor selection and the virtue of overparametrization. Bernoulli 10, 971-988.
72. Hall, P. (1987). Edgeworth expansion for Student's t statistic under minimal moment conditions. Ann. Probab. 15, 920-931.
73. Hall, P. (2006). Some contemporary problems in statistical sciences. The Madrid Intelligencer, to appear.
74. Hall, P. and Chan, Y.-B. (2009). Scale adjustments for classifiers in high-dimensional, low sample size settings. Biometrika 96, 469-478.
75. Hall, P., Marron, J. S. and Neeman, A. (2005). Geometric representation of high dimension, low sample size data. J. Roy. Statist. Soc. Ser. B 67, 427-444.
76. Hall, P. and Miller, H. (2009a). Using generalized correlation to effect variable selection in very high dimensional problems. J. Comput. Graph. Statist., to appear.
77. Hall, P. and Miller, H. (2009b). Recursive methods for variable selection in very high dimensional classification. Manuscript.
78. Hall, P., Park, B. and Samworth, R. (2008). Choice of neighbor order in nearest-neighbor classification. Ann. Statist. 5, 2135-2152.
79. Hall, P., Pittelkow, Y. and Ghosh, M. (2008). Theoretic measures of relative performance of classifiers for high dimensional data with small sample sizes. J. Roy. Statist. Soc. Ser. B 70, 158-173.
80. Hall, P., Titterington, D. M. and Xue, J.-H. (2008). Discussion of "Sure independence screening for ultrahigh dimensional feature space". J. Roy. Statist. Soc. Ser. B 70, 889-890.
81. Hall, P., Titterington, D. M. and Xue, J.-H. (2009). Tilting methods for assessing the influence of components in a classifier. Jo. Roy. Statist. Soc. Ser. B, to appear.
82. Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd edition). Springer-Verlag, New York.
83. Huang, J., Horowitz, J. and Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36, 587-613.
84. Huang, X. and Pan, W. (2003). Linear regression and two-class classification with gene expression data. Bioinformatics 19, 2072-2078.
85. Hunter, D. R. and Lange, K. (2000). Rejoinder to discussion of "Optimization transfer using surrogate objective functions." J. Comput. Graph. Statist. 9, 52-59.
86. Hunter, D. R. and Li, R. (2005). Variable selection using MM algorithms. Ann. Statist. 33, 1617-1642.
87. James, G., Radchenko, P. and Lv, J. (2009). DASSO: connections between the Dantzig selector and LASSO. J. Roy. Statist. Soc. Ser. B 71, 127-142.
88. Jin, J. (2009). Impossibility of successful classification when useful features are rare and weak. Proc. Nat. Acad. Sci. 106, 8859-8864.
89. Jing, B. Y., Shao, Q.-M. and Wang, Q. Y. (2003). Self-normalized Cramer type large deviations for independent random variables. Ann. Probab. 31, 2167-2215.
90. Johnstone, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29, 295-327.
91. Kim, Y., Choi, H. and Oh, H.S. (2008). Smoothly clipped absolute deviation on high dimensions. J. Amer. Statist. Assoc. 103, 1665-1673.
92. Kim, Y. and Kwon, S. (2009). On the global optimum of the SCAD penalized estimator. Manuscript.
93. Koenker, R. (1984). A note on L-estimates for linear models. Statist. Probab. Lett. 2, 323-325.
94. Koenker, R. and Bassett, G. (1978). Regression quantiles. Econometrica 46, 33-50.
95. Koltchinskii, V. (2008). Sparse recovery in convex hulls via entropy penalization. Ann. Statist., to appear.
96. Lange, K. (1995). A gradient algorithm locally equivalent to the EM algorithm. J. Roy. Statist. Soc. Ser. B 57, 425-437.
97. Ledoux, M. (2001). The Concentration of Measure Phenomenon. American Mathematical Society, Cambridge.
98. Ledoux, M. (2005). Deviation inequalities on largest eigenvalues. Manuscript.
99. Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electronic Journal of Statistics 2, 90-102.
100. Lv, J. and Fan, Y. (2009). A unified approach to model selection and sparse recovery using regularized least squares. Ann. Statist. 37, 3498-3528.
101. Lv, J. and Liu, J. S. (2008). New principles for model selection when models are possibly misspecified. Manuscript.
102. p. Technometrics 15, 661-675.
103. McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models. Chapman and Hall, London.
104. Meier, L., van de Geer, S. and Bühlmann, P. (2008). The group LASSO for logistic regression. J. Roy. Statist. Soc. Ser. B 70, 53-71.
105. Meinshausen, N. (2007). Relaxed LASSO. Comput. Statist. Data Anal. 52, 374-393.
106. Meinshausen, N. and Bühlmann, P. (2006). High dimensional graphs and variable selection with the LASSO. Ann. Statist. 34, 1436-1462.
107. Meinshausen, N., Rocha, G. and Yu, B. (2007). Discussion: A tale of three cousins: LASSO, L2Boosting and Dantzig. Ann. Statist. 35, 2373-2384.
108. Osborne, M. R., Presnell, B. and Turlach, B. A. (2000). On the LASSO and its dual. Journal of Computational and Graphical Statistics 9, 319-337.
109. Ravikumar, P., Lafferty, J., Liu, H. and Wasserman, L. (2009). Sparse additive models. J. Roy. Statist. Soc. Ser. B, to appear.
110. Rosset, S. and Zhu, J. (2007). Piecewise linear regularized solution paths. Ann. Statist. 35, 1012-1030.
111. Santosa, F. and Symes, W. W. (1986). Linear inversion of band-limited reflection seismograms. SIAM J. Sci. Statist. Comput. 7, 1307-1330.
112. Schwartz, G. (1978). Estimating the dimension of a model. Ann. Statist. 6, 461-464.
113. Shen, X., Tseng, G. C., Zhang, X. and Wong, W. H. (2003). On ψ-learning. J. Amer. Statist. Assoc. 98, 724-734.
114. Silverstein, J. W. (1985). The smallest eigenvalue of a large dimensional Wishart matrix. Ann. Probab. 13, 1364-1368.
115. Storey, J. D. and Tibshirani R. (2003). Statistical significance for genome-wide studies. Proc. Nat. Acad. Sci. 100, 9440-9445.
116. 1 norm. Geophysics 44, 39-52.
117. Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. J. Roy. Statist. Soc. Ser. B 58, 267-288.
118. Tibshirani, R., Hastie, T., Narasimhan, B. and Chu, G. (2002). Diagnosis of multiple cancer types by shrunken centroid of gene expression. Proc. Nat. Acad. Sci. 99, 6567-6572.
119. Tibshirani, R., Hastie, T., Narasimhan, B. and Chu, G. (2003). Class prediction by nearest shrunken centroids, with applications to DNA microarrays. Statist. Sci. 18, 104-117.
120. Tropp, J. A. (2006). Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory 5, 1030-1051.
121. van de Geer, S. (2008). High-dimensional generalized linear models and the LASSO. Ann. Statist. 36, 614-645.
122. Vapnik, V. (1995). The Nature of Statistical Learning. Springer-Verlag, New York.
123. Wainwright, M. J. (2006). Sharp thresholds for high-dimensional and noisy recovery of sparsity. Technical Report, Department of Statistics, UC Berkeley.
124. Wang, H., Li, R. and Tsai, C.-L. (2007). Tuning parameter selectors for the smoothly clipped absolute deviation method. Biometrika 94, 553-568.
125. Wu, T. T. and Lange, K. (2008). Coordinate descent algorithms for LASSO penalized regression. Ann. Appl. Stat. 2, 224-244.
126. Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. J. Roy. Statist. Soc. Ser. B 68, 49-67.
127. Zhang, C.-H. (2009). Penalized linear unbiased selection. Ann. Statist., to appear.
128. Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the LASSO selection in highdimensional linear regression. Ann. Statist. 36, 1567-1594.
129. Zhang, Y. and Li, R. (2009). Iterative conditional maximization algorithm for nonconcave penalized likelihood. IMS Lecture Notes-Monograph Series, to appear.
130. Zhao, P. and Yu, B. (2006). On model selection consistency of LASSO. J. Mach. Learn. Res. 7, 2541-2563.
131. Zou, H. (2006). The adaptive LASSO and its oracle properties. J. Amer. Statist. Assoc. 101, 1418-1429.
132. Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. J. Roy. Statist. Soc. Ser. B 67, 301-320.
133. Zou, H., Hastie, T. and Tibshirani. R. (2004). Sparse principal component analysis. Technical report.
134. Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models (with discussion). Ann. Statist. 36, 1509-1566.
135. Zou, H. and Yuan, M. (2008). Composite quantile regression and the oracle model selection theory. Ann. Statist. 36, 1108-1126.