Exponential screening and optimal rates of sparse estimation

Annals of Statistics

Volumn 39, Issue 2, 2011, Pages 731-771

  Rigollet, Philippe ^a   Tsybakov, Alexandre ^b  

^a PRINCETON UNIVERSITY   (United States)

^b Princeton University   (United States)

Author keywords

Adaptation; Aggregation; Bic; High Dimensional Regression; Lasso; Minimax Rates; Sparsity; Sparsity Oracle Inequalities

Indexed keywords

EID: 79952982333     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/10-AOS854     Document Type: Article

References (46)

1. ABRAMOVICH, F., BENJAMINI, Y., DONOHO, D. L. and JOHNSTONE, I. M. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584-653. MR2281879 (Pubitemid 44091387)
2. ALQUIER, P. and LOUNICI, K. (2010). PAC-Bayesian bounds for sparse regression estimation with exponential weights. HAL.
3. BARANIUK, R., DAVENPORT, M., DEVORE, R. and WAKIN, M. (2008). A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28 253-263. MR2453366
4. BARRON, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39 930-945. MR1237720
5. BICKEL, P., RITOV, Y. and TSYBAKOV, A. (2008). Hierarchical selection of variables in sparse high-dimensional regression. Available at ArXiv:0801.1158.
6. BICKEL, P. J. and DOKSUM, K. A. (2006). Mathematical Statistics: Basic Ideas and Selected Topics, Vol. 1, 2nd ed. Prentice-Hall, Upper Saddle River, NJ.
7. BICKEL, P. J., RITOV, Y. and TSYBAKOV, A. B. (2009). Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist. 37 1705-1732. MR2533469
8. BüHLMANN, P. and VAN DE GEER, S. (2009). On the conditions used to prove oracle results for the Lasso. Electron. J. Statist. 3 1360-1392. MR2576316
9. BUNEA, F., TSYBAKOV, A. andWEGKAMP, M. (2007a). Sparsity oracle inequalities for the Lasso. Electron. J. Statist. 1 169-194 (electronic). MR2312149
10. BUNEA, F., TSYBAKOV, A. B. andWEGKAMP, M. H. (2007b). Aggregation for Gaussian regression. Ann. Statist. 35 1674-1697. MR2351101
11. CANDES, E. (2008). The restricted isometry property and its implications for compressed sensing. C. R. Math. Acad. Sci. Paris 346 589-592. MR2412803
12. CANDES, E. and TAO, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313-2351. MR2382644
13. DALALYAN, A. and TSYBAKOV, A. (2008). Aggregation by exponential weighting, sharp pac-Bayesian bounds and sparsity. Machine Learning 72 39-61.
14. DALALYAN, A. and TSYBAKOV, A. B. (2010). Mirror averaging with sparsity priors. Available at ArXiv.org:1003.1189.
15. DALALYAN, A. S. and TSYBAKOV, A. B. (2007). Aggregation by exponential weighting and sharp oracle inequalities. In Learning Theory. Lecture Notes in Computer Science 4539 97-111. Springer, Berlin. MR2397581
16. DALALYAN, A. S. and TSYBAKOV, A. B. (2009). Sparse regression learning by aggregation and Langevin Monte Carlo. Available at ArXiv:0903.1223.
17. DONOHO, D. L. and JOHNSTONE, I. M. (1994a). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 425-455. MR1311089
18. DONOHO, D. L. and JOHNSTONE, I. M. (1994b). Minimax risk over lp-balls for lq -error. Probab. Theory Related Fields 99 277-303. MR1278886
19. DONOHO, D. L., JOHNSTONE, I. M., HOCH, J. C. and STERN, A. S. (1992). Maximum entropy and the nearly black object. J. Roy. Statist. Soc. Ser. B 54 41-81. With discussion and a reply by the authors. MR1157714
20. FAN, J. and LI, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360. MR1946581 (Pubitemid 33695585)
21. FOSTER, D. P. and GEORGE, E. I. (1994). The risk inflation criterion for multiple regression. Ann. Statist. 22 1947-1975. MR1329177
22. GEORGE, E. I. (1986a). Combining minimax shrinkage estimators. J. Amer. Statist. Assoc. 81 437-445. MR0845881
23. GEORGE, E. I. (1986b). Minimax multiple shrinkage estimation. Ann. Statist. 14 188-205. MR0829562
24. GIRAUD, C. (2008). Mixing least-squares estimators when the variance is unknown. Bernoulli 14 1089-1107. MR2543587
25. HASTIE, T., TIBSHIRANI, R. and FRIEDMAN, J. (2001). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York. MR1851606
26. KOH, K., KIM, S.-J. and BOYD, S. (2008). A Matlab solver for l1-regularized least squares problems. BETA version. Stanford Univ. Available at http://www.stanford.edu/~boyd-l1-ls.
27. KOLTCHINSKII, V. (2010). Oracle inequalities in empirical risk minimization and sparse recovery problems. St. Flour Lecture Notes. To appear.
28. KOLTCHINSKII, V. (2009a). The Dantzig selector and sparsity oracle inequalities. Bernoulli 15 799-828. MR2555200
29. KOLTCHINSKII, V. (2009b). Sparsity in penalized empirical risk minimization. Ann. Inst. H. Poincaré Probab. Statist. 45 7-57. MR2500227
30. LECUN, Y., BOSER, B., DENKER, J. S., HENDERSON, D., HOWARD, R. E., HUBBARD, W. and JACKEL, L. D. (1990). Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems 396-404.
31. Morgan Kaufmann, San Francisco. LEUNG, G. and BARRON, A. R. (2006). Information theory and mixing least-squares regressions. IEEE Trans. Inform. Theory 52 3396-3410. MR2242356 (Pubitemid 44145107)
32. LOUNICI, K. (2007). Generalized mirror averaging and D-convex aggregation. Math. Methods Statist. 16 246-259. MR2356820
33. NEMIROVSKI, A. (2000). Topics in non-parametric statistics. In Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math. 1738 85-277. Springer, Berlin. MR1775640
34. RASKUTTI, G., WAINWRIGHT, M. J. and YU, B. (2009). Minimax rates of estimation for highdimensional linear regression over-q -balls. Available at ArXiv:0910.2042.
35. REYNAUD-BOURET, P. (2003). Adaptive estimation of the intensity of inhomogeneous Poisson processes via concentration inequalities. Probab. Theory Related Fields 126 103-153. (Pubitemid 36737904)
36. RIGOLLET, P. (2009). Maximum likelihood aggregation and misspecified generalized linear models. Technical report. Available at ArXiv:0911.2919.
37. RIGOLLET, P. and TSYBAKOV, A. (2010). Exponential screening and optimal rates of sparse estimation. Available at ArXiv:1003.2654v3.
38. ROBERT, C. and CASELLA, G. (2004). Monte Carlo Statistical Methods. Springer, New York.
39. TSYBAKOV, A. B. (2003). Optimal rates of aggregation. In COLT (B. Schölkopf and M. K. Warmuth, eds.). Lecture Notes in Computer Science 2777 303-313. Springer, Berlin.
40. TSYBAKOV, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York. MR2724359
41. VAN DE GEER, S. A. (2008). High-dimensional generalized linear models and the Lasso. Ann. Statist. 36 614-645. MR2396809
42. WRIGHT, J.,YANG, A. Y.,GANESH, A., SASTRY, S. S. andMA, Y. (2009). Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31 210-227.
43. ZHANG, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894-942. MR2604701
44. ZHANG, C.-H. and HUANG, J. (2008). The sparsity and bias of the LASSO selection in highdimensional linear regression. Ann. Statist. 36 1567-1594. MR2435448
45. ZHANG, C.-H. and MELNIK, O. (2009). Plus: Penalized linear unbiased selection. R package version 0.8. CRAN. Available at http://CRAN.R-project.org/ package=plus.
46. 1 regularization. Ann. Statist. 37 2109-2144. MR2543687