Sparse estimation by exponential weighting

Statistical Science

Volumn 27, Issue 4, 2012, Pages 558-575

  Rigollet, Philippe ^a   Tsybakov, Alexandre B ^b  

^a Princeton University   (United States)

^b CREST   (France)

Author keywords

Exponential weights; Fused sparsity; Group sparsity; High dimensional regression; Sparse regression; Sparsity; Sparsity oracle inequalities; Sparsity pattern aggregation; Sparsity prior

Indexed keywords

EID: 84871564349     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-STS393     Document Type: Article

References (44)

1. ALQUIER, P. and LOUNICI, K. (2011). PAC-Bayesian bounds for sparse regression estimation with exponential weights. Electron. J. Stat. 5 127-145.
2. BARANIUK, R., DAVENPORT, M., DEVORE, R. andWAKIN, M. (2008). A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28 253-263.
3. BARTLETT, P. L., BOUCHERON, S. and LUGOSI, G. (2002). Model selection and error estimation. Mach. Learn. 48 85-113.
4. BICKEL, P. J., RITOV, Y. and TSYBAKOV, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705-1732.
5. BIRGÉ, L. and MASSART, P. (2001). Gaussian model selection. J. Eur. Math. Soc. (JEMS) 3 203-268.
6. BREHENY, P. and HUANG, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann. Appl. Stat. 5 232-253.
7. BUNEA, F., TSYBAKOV, A. B. andWEGKAMP, M. H. (2007). Aggregation for Gaussian regression. Ann. Statist. 35 1674-1697.
8. CANDES, E. and TAO, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313- 2351.
9. CATONI, O. (1999). Universal aggregation rules with exact bias bounds. Technical report, Laboratoire de Probabilités et Modeles Aléatoires, Preprint 510.
10. CATONI, O. (2004). Statistical Learning Theory and Stochastic Optimization. Lecture Notes in Math. 1851. Springer, Berlin.
11. CATONI, O. (2007). Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning. Institute of Mathematical Statistics Lecture Notes-Monograph Series 56. IMS, Beachwood, OH.
12. DALALYAN, A. S. and SALMON, J. (2011). Sharp oracle inequalities for aggregation of affine estimators. Available at ArXiv:1104.3969.
13. DALALYAN, A. S. and TSYBAKOV, A. B. (2007). Aggregation by exponential weighting and sharp oracle inequalities. In Learning Theory. Lecture Notes in Computer Science 4539 97-111. Springer, Berlin.
14. DALALYAN, A. and TSYBAKOV, A. B. (2008). Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Mach. Learn. 72 39-61.
15. DALALYAN, A. and TSYBAKOV, A. B. (2012). Mirror averaging with sparsity priors. Bernoulli 18 914-944.
16. DALALYAN, A. S. and TSYBAKOV, A. B. (2012). Sparse regression learning by aggregation and Langevin Monte-Carlo. J. Comput. System Sci. 78 1423-1443.
17. FAN, J. and LI, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360.
18. FRIEDMAN, J., HASTIE, T. and TIBSHIRANI, R. (2010). Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33 1-22.
19. GERCHINOVITZ, S. (2011). Prediction of individual sequences and prediction in the statistical framework: Some links around sparse regression and aggregation techniques. Ph.D. thesis, Univ. Paris Sud-Paris XI.
20. GIRAUD, C. (2007). Mixing least-squares estimators when the variance is unknown. Available at arXiv:0711.0372.
21. GIRAUD, C. (2008). Mixing least-squares estimators when the variance is unknown. Bernoulli 14 1089-1107.
22. HUANG, J. and ZHANG, T. (2010). The benefit of group sparsity. Ann. Statist. 38 1978-2004.
23. JUDITSKY, A., RIGOLLET, P. and TSYBAKOV, A. B. (2008). Learning by mirror averaging. Ann. Statist. 36 2183-2206.
24. JUDITSKY, A. B., NAZIN, A. V., TSYBAKOV, A. B. and VAYATIS, N. (2005). Recursive aggregation of estimators by the mirror descent method with averaging. Probl. Inf. Transm. 41 368-384.
25. KNEIP, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835- 866.
26. KOLTCHINSKII, V., LOUNICI, K. and TSYBAKOV, A. B. (2011). Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. Ann. Statist. 39 2302-2329.
27. LECUÉ, G. (2007). Simultaneous adaptation to the margin and to complexity in classification. Ann. Statist. 35 1698-1721.
28. LECUÉ, G. (2012). Empirical risk minimization is optimal for the Convex aggregation problem. Bernoulli. To appear.
29. LEUNG, G. and BARRON, A. R. (2006). Information theory and mixing least-squares regressions. IEEE Trans. Inform. Theory 52 3396-3410.
30. LOUNICI, K., PONTIL, M., VAN DE GEER, S. and TSYBAKOV, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39 2164-2204.
31. LUGOSI, G. and WEGKAMP, M. (2004). Complexity regularization via localized random penalties. Ann. Statist. 32 1679-1697.
32. NEMIROVSKI, A. (2000). Topics in non-parametric statistics. In Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math. 1738 85-277. Springer, Berlin.
33. RIGOLLET, P. (2012). Kullback-Leibler aggregation and misspecified generalized linear models. Ann. Statist. 40 639-665.
34. RIGOLLET, P. and TSYBAKOV, A. B. (2007). Linear and convex aggregation of density estimators. Math. Methods Statist. 16 260-280.
35. RIGOLLET, P. and TSYBAKOV, A. (2011). Exponential screening and optimal rates of sparse estimation. Ann. Statist. 39 731-771.
36. ROBERT, C. P. and CASELLA, G. (2004). Monte Carlo Statistical Methods, 2nd ed. Springer, New York.
37. RUDIN, L. I., OSHER, S. and FATEMI, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60 259-268.
38. TIBSHIRANI, R., SAUNDERS, M., ROSSET, S., ZHU, J. and KNIGHT, K. (2005). Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 91-108.
39. TSYBAKOV, A. B. (2003). Optimal rates of aggregation. In COLT (B. Schölkopf andM. K. Warmuth, eds.). Lecture Notes in Computer Science 2777 303-313. Springer, Berlin.
40. TSYBAKOV, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York.
41. WEGKAMP, M. (2003). Model selection in nonparametric regression. Ann. Statist. 31 252-273.
42. YANG, Y. (2004). Aggregating regression procedures to improve performance. Bernoulli 10 25-47.
43. YUAN, M. and LIN, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 49-67.
44. ZHANG, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894-942.