Stability selection

Journal of the Royal Statistical Society. Series B: Statistical Methodology

Volumn 72, Issue 4, 2010, Pages 417-473

  Meinshausen, Nicolai ^a   Bühlmann, Peter ^b  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b ETH ZURICH   (Switzerland)

Author keywords

High dimensional data; Resampling; Stability selection; Structure estimation

Indexed keywords

EID: 77958487535     PISSN: 13697412     EISSN: 14679868     Source Type: Journal    
DOI: 10.1111/j.1467-9868.2010.00740.x     Document Type: Article

References (99)

1. Bach, F. (2008) Bolasso: model consistent lasso estimation through the bootstrap. In Proc. 25th Int. Conf. Machine Learning, pp. 33-40. New York: Association for Computing Machinery.
2. Banerjee, O. and El Ghaoui, L. (2008) Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. J. Mach. Learn. Res., 9, 485-516.
3. Barbieri, M. and Berger, J. (2004) Optimal predictive model selection. Ann. Statist., 32, 870-897.
4. Bhattacharjee, A., Richards, W., Staunton, J., Li, C., Monti, S., Vasa, P., Ladd, C., Beheshti, J., Bueno, R., Gillette, M., Loda, M., Weber, G., Mark, E., Lander, E., Wong, W., Johnson, B., Golub, T., Sugarbaker, D. and Meyerson, M. (2005) Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Bioinformatics, 21, 3301-3307.
5. Bickel, P. and Levina, E. (2008) Regularized estimation of large covariance matrices. Ann. Statist., 36, 199-227.
6. Bickel, P., Ritov, Y. and Tsybakov, A. (2009) Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist., 37, 1705-1732.
7. Breiman, L. (2001) Random Forests. Mach. Learn., 45, 5-32.
8. Bühlmann, P. and Yu, B. (2002) Analyzing bagging. Ann. Statist., 30, 927-961.
9. Candes, E. and Tao, T. (2007) The Dantzig selector: statistical estimation when p is much larger than n. Ann. Statist., 35, 2312-2351.
10. Chen, S., Donoho, S. and Saunders, M. (2001) Atomic decomposition by basis pursuit. SIAM Rev., 43, 129-159.
11. Conlon, E., Liu, X., Lieb, J. and Liu, J. (2003) Integrating regulatory motif discovery and genome-wide expression analysis. Proc. Natn. Acad. Sci. USA, 100, 3339-3344.
12. Davis, C., Gerick, F., Hintermair, V., Friedel, C., Fundel, K., Kuffner, R. and Zimmer, R. (2006) Reliable gene signatures for microarray classification: assessment of stability and performance. Bioinformatics, 22, 2356-2363.
13. 1-minimization. Proc. Natn. Acad. Sci. USA, 100, 2197-2202.
14. Dudoit, S., Shaffer, J. and Boldrick, J. (2003) Multiple hypothesis testing in microarray experiments. Statist. Sci., 18, 71-103.
15. Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004) Least angle regression. Ann. Statist., 32, 407-451.
16. Ein-Dor, L., Kela, I., Getz, G., Givol, D. and Domany, E. (2005) Outcome signature genes in breast cancer: is there a unique set? Bioinformatics, 21, 171-178.
17. Fan, J. and Lv, J. (2008) Sure independence screening for ultrahigh dimensional feature space (with discussion). J. R. Statist. Soc. B, 70, 849-911.
18. Fan, J., Samworth, R. and Wu, Y. (2009) Ultrahigh dimensional variable selection: beyond the linear model. J. Mach. Learn. Res., 10, 1989-2014.
19. Freedman, D. (1977) A remark on the difference between sampling with and without replacement. J. Am. Statist. Ass., 72, 681.
20. Freund, Y. and Schapire, R. (1996) Experiments with a new boosting algorithm. In Proc. 13th Int. Conf. Machine Learning, pp. 148-156.
21. Friedman, J., Hastie, T., Hoefling, H. and Tibshirani, R. (2007) Pathwise coordinate optimization. Ann. Appl. Statist., 1, 302-332.
22. Friedman, J., Hastie, T. and Tibshirani, R. (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9, 432-441.
23. Van De Geer, S. (2008) High-dimensional generalized linear models and the lasso. Ann. Statist., 36, 614-645.
24. Van De Geer, S. and Van Houwelingen, H. (2004) High-dimensional data: p»n in mathematical statistics and bio-medical applications. Bernoulli, 10, 939-943.
25. Huang, J., Ma, S. and Zhang, C.-H. (2008) Adaptive lasso for sparse high-dimensional regression models. Statist. Sin., 18, 1603-1618.
26. Lauritzen, S. (1996) Graphical Models. Oxford: Oxford University Press.
27. Lee, K., Sha, N., Dougherty, E., Vannucci, M. and Mallick, B. (2003) Gene selection: a Bayesian variable selection approach. Bioinformatics, 19, 90-97.
28. Leng, C., Lin, Y. and Wahba, G. (2006) A note on the lasso and related procedures in model selection. Statist. Sin., 16, 1273-1284.
29. Mallat, S. and Zhang, Z. (1993) Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process., 41, 3397-3415.
30. Meier, L., Van De Geer, S. and Bühlmann, P. (2008) The group lasso for logistic regression. J. R. Statist. Soc. B, 70, 53-71.
31. Meinshausen, N. and Bühlmann, P. (2006) High dimensional graphs and variable selection with the lasso. Ann. Statist., 34, 1436-1462.
32. Meinshausen, N. and Yu, B. (2009) Lasso-type recovery of sparse representations from high-dimensional data. Ann. Statist., 37, 246-270.
33. Michiels, S., Koscielny, S. and Hill, C. (2005) Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet, 365, 488-492.
34. Monti, S., Tamayo, P., Mesirov, J. and Golub, T. (2003) Consensus clustering: a resampling-based method for class discovery and visualization of gene expression microarray data. Mach. Learn., 52, 91-118.
35. Rothman, A., Bickel, P., Levina, E. and Zhu, J. (2008) Sparse permutation invariant covariance estimation. Electron. J. Statist., 2, 494-515.
36. Sha, N., Vannucci, M., Tadesse, M., Brown, P., Dragoni, I., Davies, N., Roberts, T., Contestabile, A., Salmon, M., Buckley, C. and Falciani, F. (2004) Bayesian variable selection in multinomial probit models to identify molecular signatures of disease stage. Biometrics, 60, 812-819.
37. Temlyakov, V. (2000) Weak greedy algorithms. Adv. Computnl Math., 12, 213-227.
38. Tibshirani, R. (1996) Regression shrinkage and selection via the lasso. J. R. Statist. Soc. B, 58, 267-288.
39. Tropp, J. (2004) Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inform. Theory, 50, 2231-2242.
40. Valdar, W., Holmes, C., Mott, R. and Flint, J. (2009) Mapping in structured populations by resample model averaging. Genetics, 182, 1263-1277.
41. Wainwright, M. (2009) Sharp thresholds for high-dimensional and noisy recovery of sparsity. IEEE Trans. Inform. Theor., 55, 2183-2202.
42. Yuan, M. and Lin, Y. (2007) Model selection and estimation in the gaussian graphical model. Biometrika, 94, 19-35.
43. Zhang, C.-H. (2007) Penalized linear unbiased selection. Technical Report 2007-003 Department of Statistics, Rutgers University, Piscataway.
44. Zhang, T. (2008) Adaptive forward-backward greedy algorithm for sparse learning with linear models. In Proc. Neural Information Processing Systems.
45. Zhang, T. (2009) On the consistency of feature selection using greedy least squares regression. J. Mach. Learn. Res., 10, 555-568.
46. Zhang, C. and Huang, J. (2008) The sparsity and bias of the lasso selection in high-dimensional linear regression. Ann. Statist., 36, 1567-1594.
47. Zhao, P. and Yu, B. (2006) On model selection consistency of lasso. J. Mach. Learn. Res., 7, 2541-2563.
48. Zou, H. (2006) The adaptive lasso and its oracle properties. J. Am. Statist. Ass., 101, 1418-1429.
49. Zucknick, M., Richardson, S. and Stronach, E. A. (2008) Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods. Statist. Appl. Genet. Molec. Biol., 7, article 7.
50. Akaike, H. (1980) Seasonal adjustment by a Bayesian modeling. J. Time Ser. Anal., 1, 1-13.
51. Bach, F. (2008) Bolasso: model consistent Lasso estimation through the bootstrap. In Proc. 25th Int. Conf. Machine Learning, pp. 33-40. New York: Association for Computing Machinery.
52. Barbieri, M. and Berger, J. (2004) Optimal predictive model section. Ann. Statist., 32, 870-897.
53. Belsley, D. A., Kuh, E. and Welsch, R. E. (1980) Regression Diagnostics, ch. 2. New York: Wiley.
54. Ben-David, S., Von Luxburg, U. and Pall, D. (2006) A sober look at clustering stability. Learn. Theor., no. 4005, 5-19.
55. Ben-Hur, A., Elisseef, A. and Guyon, I. (2002) A stability based method for discovering structure in clustered data. Pacific Symp. Biocomputing.
56. Berger, J. O. (1985) Statistical Decision Theory and Bayesian Analysis. Berlin: Springer.
57. Bickel, P., Ritov, Y. and Tsybakov, A. (2009) Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist., 37, 1705-1732.
58. Bottolo, L. and Richardson, S. (2010) Evolutionary Stochastic Search for Bayesian model exploration. Preprint. (Available from .)
59. Breiman, L. (1996) Bagging predictors. Mach. Learn., 24, 123-140.
60. Brown, P. J., Fearn, T. and Vannucci, M. (1999) The choice of variables in multivariate regression: a non-conjugate Bayesian decision theory approach. Biometrika, 60, 627-641.
61. Bunea, F., Tsybakov, A. and Wegkamp, M. (2007) Sparsity oracle inequalities for the Lasso. Electron. J. Statist., 1, 169-194.
62. Dalmasso, C., Broët, P. and Moreau, T. (2005) A simple procedure for estimating the false discovery rate. Bioinformatics, 21, 660-668.
63. Dupuis, J. A. and Robert, C. P. (2003) Variable selection in qualitative models via an entropic explanatory power. J. Statist. Planng Inf., 111, 77-94.
64. Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004) Least angle regression. Ann. Statist., 32, 407-499.
65. Fan, J. and Lv, J. (2008) Sure independence screening for ultrahigh diemensional feature space (with discussion). J. R. Statist. Soc. B, 70, 849-911.
66. Fan, J., Samworth, R. and Wu, Y. (2009) Ultrahigh dimensional feature selection: beyond the linear model. J. Mach. Learn. Res., 10, 2013-2038.
67. Gandy, A. (2009) Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk. J. Am. Statist. Ass., 104, 1504-1511.
68. Van de Geer, S., Zhou, S. and Bühlmann, P. (2010) Prediction and variable selection with the adaptive Lasso. Preprint arXiv:1001.5176v1.
69. George, E. I. and McCullogh, R. E. (1997) Approaches for Bayesian variable selection. Statist. Sin., 7, 339-373.
70. Griffin, J. E. and Brown, P. J. (2007) Bayesian adaptive lassos with non-convex penalisation. Institute of Mathematics and Statistics, University of Kent, Canterbury. (Available from .)
71. Griffin, J. E. and Brown, P. J. (2010) Inference with normal-gamma prior distributions in regression problems. Bayesn Anal., 5, 171-181.
72. Hans, C., Dobra, A. and West, M. (2007) Shotgun Stochastic Search for large p regression. J. Am. Statist. Ass., 102, 507-517.
73. Hastie, T., Tibshirani, R. and Friedman, J. (2008) The Elements of Statistical Learning, 2nd edn. New York: Springer.
74. Haufe, S., Müller, K.-R., Nolte, G. and Krämer, N. (2010) Sparse casual discovery in multivariate time series. In Journal of Machine Learning Research Wrkshp Conf. Proc., vol. 6, Causality: Objectives and Assessment, pp. 97-106. (Availabe from .)
75. Hennig, C. (2010) Methods for merging Gaussian mixture components. In Adv. Data Anal. Classificn, 4, 3-34.
76. Johnstone, I. M. and Silverman, B. W. (2005) Empirical Bayes selection of wavelet thresholds. Ann. Statist., 33, 1700-1752.
77. Künsch, H.-R. (1989) The jackknife and the bootstrap for general stationary observations. Ann. Statist., 17, 1217-1241.
78. Lange, T., Roth, V., Braun, M. and Buhmann, J. (2004) Stability-based validation of clustering solutions. Neur. Computn, 16, 1299-1323.
79. Lee, H. K. H. (2004) Bayesian Nonparametrics via Neural Networks. Philadelphia: Society for Industrial and Applied Mathematics.
80. Lindley, D. V. (1968) The choice of variables in multiple regression (with discussion). J. R. Statist. Soc. B, 30, 31-66.
81. Meinshausen, N. (2008) A note on the Lasso for graphical Gaussian model selection. Statist. Probab. Lett., 78, 880-884.
82. Meinshausen, N., Meier, L. and Bühlmann, P. (2009) P-values for high-dimensional regression. J. Am. Statist. Ass., 104, 1671-1681.
83. Monti, S., Tamayo, P., Mesirov, J. and Golub, T. (2003) Consensus clustering: a resampling-based method for class discovery and visualization of gene expression microarray data. Mach. Learn., 52, 91-118.
84. Nott, D. J. and Leng, C. (2010) Bayesian projection approaches to variable selection in generalized linear models. Computnl Statist. Data Anal., to be published.
85. Park, T. and Casella, G. (2008) The Bayesian Lasso. J. Am. Statist. Ass., 103, 681-686.
86. 1-penalized log-determinant divergence. Preprint arXiv:0811.3628.
87. Rinaldo, A. and Wasserman, L. (2010) Generalized density clustering. Ann. Statist., to be published. (Available from .)
88. Sauerbrei, W. and Schumacher, M. (1992) A bootstrap resampling procedure for model-building-application to the Cox regression-model. Statist. Med., 11, 2093-2109.
89. Savage, L. J. (1954) The Foundations of Statistics. New York: Wiley.
90. Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T. and Zeileis, A. (2008) Conditional variable importance for random forests. BMC Bioinform., 9.
91. Tibshirani, R. and Walther, G. (2005) Cluster validation by prediction strength. J. Computnl Graph. Statist., 14, 511-528.
92. Tong, H. (2010) Obituary of Hirotugu Akaike. J. R. Statist. Soc. A, 173, 451-454.
93. Wainwright, M. (2009) Sharp thresholds for high-dimensional and noisy recovery of sparsity. IEEE Trans. Inform. Theor., 55, 2183-2202.
94. Wang, H., Li, G. and Tsai, C. L. (2007) Regression coefficient and autoregressive order shrinkage and selection via the lasso. J. R. Statist. Soc. B, 69, 63-78.
95. Zhao, P. and Yu, B. (2006) On model selection consistency of lasso. J. Mach. Learn. Res., 7, 2541-2563.
96. Zou, H. (2006) The adaptive lasso and its oracle properties. J. Am. Statist. Ass., 101, 1418-1429.
97. Zou, H. and Hastie, T. (2005) Regularization and variable selection via the elastic net. J. R. Statist. Soc. B, 67, 301-320.
98. Zou, H., Hastie, T. and Tibshirani, R. (2007) On the degrees of freedom of the LASSO. Ann. Statist., 35, 2173-2192.
99. Zucknick, M., Richardson, S. and Stronach, E. A. (2008) Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods. Statist. Applic. Genet. Molec. Biol., 7, no. 1, article 7.