Estimation and variable selection with exponential weights

Electronic Journal of Statistics

Volumn 8, Issue 1, 2014, Pages 328-354

  Arias Castro, Ery ^a   Lounici, Karim ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Estimation; Exponential weights; Gibbs sampler; Identifiability con dition; Model selection; Sparse linear model; Variable selection

Indexed keywords

EID: 84902471524     PISSN: 19357524     EISSN: None     Source Type: Journal    
DOI: 10.1214/14-EJS883     Document Type: Article

References (49)

1. Abramovich, F. and V. Grinshtein (2010). Map model selection in gaussian regression. Electron. J. Stat. 4, 932-949. MR2721039.
2. Alquier, P. and K. Lounici (2011). Pac-bayesian theorems for sparse regression estimation with exponential weights. Electronic Journal of Statis-tics 5, 127-145. Arxiv: 1009. 2707. MR2786484.
3. Bach, F. R. (2008). Bolasso: model consistent lasso estimation through the bootstrap. In Proceedings of the 25th international conference on Machine learning, ICML '08, New York, NY, USA, pp. 33-40. ACM.
4. Bickel, P., Y. Ritov, and A. Tsybakov (2009). Simultaneous analysis of lasso and dantzig selector. Annals of Statistics 37(4), 1705-1732. MR2533469.
5. Birgé, L. and P. Massart (2001). Gaussian model selection. J. Eur. Math. Soc. (JEMS) 3(3), 203-268. MR1848946.
6. Bunea, F. (2008). Consistent selection via the Lasso for high dimensional approximating regression models. In Pushing the limits of contemporary statistics: contributions in honor of Jayanta K. Ghosh, Volume 3 of Inst. Math. Stat. Collect., pp. 122-137. Beachwood, OH: Inst. Math. Statist. MR2459221.
7. Bunea, F. and A. Nobel (2008). Sequential procedures for aggregating arbitrary estimators of a conditional mean. IEEE Trans. Inform. The-ory 54(4), 1725-1735. MR2450298.
8. Bunea, F., A. Tsybakov, and M. Wegkamp (2007). Sparsity oracle inequalities for the Lasso. Electronic Journal of Statistics 1, 169-194. MR2312149.
9. Cai, T. T. and L. Wang (2011). Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Trans. Inform. Theory 57(7), 4680-4688. MR2840484.
10. Candès, E. and T. Tao (2007). The Dantzig selector: statistical estimation when p is much larger than n. Ann. Statist. 35(6), 2313-2351. MR2382644.
11. Candès, E. J. and M. A. Davenport (2013). How well can we estimate a sparse vector? Appl. Comput. Harmon. Anal. 34(2), 317-323. MR3008569.
12. Candès, E. J. and Y. Plan (2009). Near-ideal model selection by ℓ1 minimization. Ann. Statist. 37(5A), 2145-2177. MR2543688.
13. Catoni, O. (2004). Statistical learning theory and stochastic optimization, Volume 1851 of Lecture Notes in Mathematics. Berlin: Springer-Verlag. Lecture notes from the 31st Summer School on Probability Theory held in Saint-Flour, July 8-25, 2001. MR2163920.
14. Chen, J. and Z. Chen (2008). Extended Bayesian information criteria for model selection with large model spaces. Biometrika 95(3), 759-771. MR2443189.
15. Chipman, H., E. I. George, and R. E. McCulloch (2001). The practical implementation of Bayesian model selection. In Model selection, Volume 38 of IMS Lecture Notes Monogr. Ser., pp. 65-134. Beachwood, OH: Inst. Math. Statist. With discussion by M. Clyde, Dean P. Foster, and Robert A. Stine, and a rejoinder by the authors. MR2000752.
16. Dalalyan, A. and J. Salmon (2011). Optimal aggregation of affine estimators. In Proceedings of the 24th annual conference on Computational Learning Theory, Budapest (Hungary).
17. Dalalyan, A. and J. Salmon (2012). Sharp oracle inequalities for aggregation of affine estimators. Ann. Statist. 40(4), 2327-2355. MR3059085.
18. Dalalyan, A. and A. Tsybakov (2007). Aggregation by exponential weighting and sharp oracle inequalities. In Learning theory, Volume 4539 of Lecture Notes in Comput. Sci., pp. 97-111. Berlin: Springer. MR2397581.
19. Fan, J. and J. Lv (2008). Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 70(5), 849-911. MR2530322.
20. Fan, J. and J. Lv (2011). Nonconcave penalized likelihood with NP-dimensionality. IEEE Trans. Inform. Theory 57(8), 5467-5484. MR2849368.
21. Fan, J. and H. Peng (2004). Nonconcave penalized likelihood with a diverging number of parameters. Ann. Statist. 32(3), 928-961. MR2065194.
22. Gautier, E. and A. Tsybakov (2011, October). High-dimensional instrumental variables regression and confidence sets. Technical report, Arxiv preprint 1105. 2454v3. MR2867623.
23. Giraud, C. (2008). Mixing least-squares estimators when the variance is unknown. Bernoulli 14(4), 1089-1107. MR2543587.
24. Ji, P. and J. Jin (2012). UPS delivers optimal phase diagram in highdimensional variable selection. Ann. Statist. 40(1), 73-103. MR3013180.
25. Jin, J., C. Zhang, and Q. Zhang (2012). Optimality of graphlet screening in high dimensional variable selection. Available online at http://arxiv. org/abs/1204. 6452.
26. Juditsky, A., P. Rigollet, and A. B. Tsybakov (2008). Learning by mirror averaging. Ann. Statist. 36(5), 2183-2206. MR2458184.
27. Leung, G. and A. Barron (2006). Information theory and mixing leastsquares regressions. IEEE Transactions on Information Theory 52(8), 3396-3410. MR2242356.
28. Lounici, K. (2007). Generalized mirror averaging and D-convex aggregation. Math. Methods Statist. 16(3), 246-259. MR2356820.
29. Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electronic Journal of Statis-tics 2, 90-102. MR2386087.
30. Lounici, K. (2009). Statistical Estimation in High-Dimension, Sparsity Oracle Inequalities. Ph. D. thesis, University Paris Diderot-Paris 7.
31. Lounici, K., M. Pontil, A. Tsybakov, and S. van de Geer (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39(4), 2164-2204. MR2893865.
32. Meinshausen, N., P. Bühlmann, and E. Zürich (2006). High dimensional graphs and variable selection with the lasso. Annals of Statistics 34, 1436-1462. MR2278363.
33. Meinshausen, N. and B. Yu (2009). Lasso-type recovery of sparse representations for high-dimensional data. Ann. Statist. 37(1), 246-270. MR2488351.
34. Raskutti, G., M. J. Wainwright, and B. Yu (2011). Minimax rates of estimation for high-dimensional linear regression over ℓq-balls. IEEE Trans. Inform. Theory 57(10), 6976-6994. MR2882274.
35. Rigollet, P. and A. Tsybakov (2011). Exponential screening and optimal rates of sparse estimation. Ann. Statist. 39(2), 731-771. MR2816337.
36. Rigollet, P. and A. B. Tsybakov (2012). Sparse estimation by exponential weighting. Statist. Sci. 27(4), 558-575. MR3025134.
37. Rudelson, M. and S. Zhou (2013). Reconstruction from anisotropic random measurements. IEEE Trans. Inform. Theory 59(6), 3434-3447. MR3061256.
38. Shao, J. (1997). An asymptotic theory for linear model selection. Statist. Sinica 7(2), 221-264. With comments and a rejoinder by the author. MR1466682.
39. Stewart, G. W. and J. G. Sun (1990). Matrix perturbation theory. Computer Science and Scientific Computing. Boston, MA: Academic Press Inc. MR1061154.
40. Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices. Available from http://arxiv. org/abs/1011. 3027. MR2963170.
41. Verzelen, N. (2012). Minimax risks for sparse regressions: Ultra-high dimensional phenomenons. Electron. J. Stat. 6, 38-90. MR2879672.
42. Wainwright, M. J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ1-constrained quadratic programming (Lasso). IEEE Trans. Inform. Theory 55(5), 2183-2202. MR2729873.
43. Yang, Y. (2004). Aggregating regression procedures to improve performance. Bernoulli 10(8), 25-47. MR2044592.
44. Yang, Y. (2005). Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation. Biometrika 92(4), 937-950. MR2234196.
45. Ye, F. and C.-H. Zhang (2010). Rate minimaxity of the Lasso and Dantzig selector for the ℓq loss in ℓr balls. J. Mach. Learn. Res. 11, 3519-3540. MR2756192.
46. Zhang, C.-H. (2007). Information-theoretic optimality of variable selection with concave penalty. Technical report, Dept. Statistics, Rutgers Univ.
47. Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38(2), 894-942. MR2604701.
48. Zhang, C.-H. and T. Zhang (2012). A general theory of concave regularization for high-dimensional sparse estimation problems. Statist. Sci. 27(4), 576-593. MR3025135.
49. Zhao, P. and B. Yu (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7, 2541-2563. MR2274449.