PAC-Bayesian bounds for sparse regression estimation with exponential weights

Electronic Journal of Statistics

Volumn 5, Issue , 2011, Pages 127-145

  Alquier, Pierre ^a,b   Lounici, Karim ^c  

^a CREST   (France)

^b UNIVERSITÉ PARIS DESCARTES   (France)

^c GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Exponential weights; High dimensional regression; Pac bayesian inequalities; References; Sparsity oracle inequality

Indexed keywords

EID: 79960999147     PISSN: 19357524     EISSN: None     Source Type: Journal    
DOI: 10.1214/11-EJS601     Document Type: Article

References (46)

1. Akaike, H. (1973). Information Theory and an Extension of the Maximum Likelihood Principle. 2nd International Symposium on Information Theory. Budapest: Akademia Kiado. B. N. Petrov and F. Csaki, editors. 267-281.
2. Alquier, P. (2006). Transductive and Inductive Adaptative Inference for Regression and Density Estimation. PhD Thesis, Université Paris 6.
3. Alquier, P. (2008). PAC-Bayesian bounds for randomized empirical risk minimizers. Mathematical Methods of Statistics 17 (4) 279-304. MR2483458
4. Audibert, J.-Y. (2004). Aggregated Estimators and Empirical Complexity for Least Square Regression. Annales de l'Institut Henri Poincaré B: Probability and Statistics 40 (6) 685-736. MR2096215
5. Audibert, J.-Y. (2004). PAC-Bayesian Statistical Learning Theory. PhD Thesis, Université Paris 6.
6. Bach, F. (2008). Bolasso: model consistent Lasso estimation through the bootstrap. Proceedings of the 25th international conference on Machine learning, ACM, New-York.
7. Bickel, P. J., Ritov, Y. and Tsybakov, A. (2009). Simultaneous Analysis of Lasso and Dantzig Selector. The Annals of Statistics 37 (4) 1705-1732.MR2533469
8. Bunea, F., Tsybakov, A. and Wegkamp, M. (2007). Sparsity oracle inequalities for the Lasso. Electronic Journal of Statistics 1 169-194. MR2397610
9. Bunea, F., Tsybakov, A. and Wegkamp, M. (2007). Aggregation for Gaussian regression. The Annals of Statistics 35 1674-1697. MR2351101
10. Cai, T., Xu, G. and Zhang, J. (2009). On Recovery of Sparse Signals via ℓ1 Minimization. IEEE Transactions on Information Theory 55 3388-3397.MR2598028
11. Candés, E. and Tao, T. (2007) The Dantzig selector: statistical estimation when p is much larger than n. The Annals of Statistics 35. MR2382644
12. Catoni, O. (2003) A PAC-Bayesian approach to adaptative classification. Preprint Laboratoire de Probabilités et Modéles Al ́eatoires PMA-840.
13. Catoni, O. (2004) Statistical Learning Theory and Stochastic Optimiza- tion. Saint-Flour Summer School on Probability Theory 2001 (Jean Picard ed.), Lecture Notes in Mathematics, Springer. MR2163920
14. Catoni, O. (2007) PAC-Bayesian Supervised Classification (The Thermo- dynamics of Statistical Learning). IMS Lecture Notes - Monograph Series 56. MR2483528
15. Chen, S. S., Donoho, D. L. and Saunders, M. A. (2001). Atomic Decomposition by Basis Pursuit. SIAM Review 43 (1) 129-159 MR1639094
16. Dalalyan, A. and Tsybakov, A. (2007). Aggregation by exponential weighting and sharp oracle inequalities. Proceedings of the 20th annual con- ference on Learning theory, Springer-Verlag Berlin, Heidelberg. MR2397581
17. Dalalyan, A. and Tsybakov, A. (2008). Aggregation by exponential weighting, sharp oracle inequalities and sparsity. Machine Learning 72 (1- 2) 39-61.
18. Dalalyan, A. and Tsybakov, A. (2010). Mirror averaging with sparsity priors. In minor revision for Bernoulli, arXiv:1003.1189.
19. Dalalyan, A. and Tsybakov, A. (2010). Sparse Regression Learning by Aggregation and Langevin Monte-Carlo. Preprint arXiv:0903.1223.
20. Dembo, A. and Zeitouni, O. (1998). Large Deviation Techniques and Applications. Springer. MR1619036
21. Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least Angle Regression. The Annals of Statistics 32 (2) 407-499. MR2060166
22. Frank, L. and Friedman, J. (1993). A Statistical View on Some Chemometrics Regression Tools. Technometrics 16 199-511.
23. Ghosh, S. (2007). Adative Elastic Net: an Improvement of Elastic Net to achieve Oracle Properties. Preprint.
24. Juditsky, A., Rigollet, P. and Tsybakov, A. (2008). Learning by mirror averaging. The Annals of Statistics 36 (5) 2183-2206. MR2458184
25. Koltchinskii, V. (2010). Sparsity in Empirical Risk Minimization. An- nales de l'Institut Henri Poincaré B: Probability and Statistics, to appear.
26. Koltchinskii, V. (2009). The Dantzig selector and sparsity oracle inequalities. Bernoulli 15 (3) 799-828. MR2555200
27. Lehmann, E. L. and Casella, G. (1998). Theory of Point Estimation. Springer. MR1639875
28. Leung, G. and Barron, A. R. (2006). Information theory and mixing least-squares regressions. IEEE Transactions on Information Theory 52 (8) 3396-3410. MR2242356
29. Lounici, K. (2007). Generalized mirror averaging and d-convex aggregation. Mathematical Methods of Statistics 16 (3). MR2356820
30. Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electronic Journal of Statistics 2 90-102. MR2386087
31. Mallows, C. L. (1973). Some comments on cp. Technometrics 15 661-676.MR1365719
32. Marin, J.-M. and Robert, C. P. (2007). Bayesian Core: A practical approach to computational Bayesian analysis. Springer. MR2289769
33. Massart, P. (2007). Concentration Inequalities and Model Selection. Saint-Flour Summer School on Probability Theory 2003 (Jean Picard ed.), Lecture Notes in Mathematics, Springer. MR2319879
34. McAllester D. A. (1998). Some pac-bayesian theorems. Proceedings of the Eleventh Annual Conference on Computational Learning Theory. ACM, 230-234. MR1811587
35. Meinshausen, N. and Bühlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society B, to appear.
36. Rigollet, P. and Tsybakov, A. (2010). Exponential Screening and optimal rates of sparse estimation. The Annals of Statistics, to appear.
37. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6 461-464.MR0468014
38. Shawe-Taylor, J. and Williamson, R. (1997). A PAC analysis of a bayes estimator. Proceedings of the Tenth Annual Conference on Computational Learning Theory. ACM, 2-9.
39. Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society B 58 (1) 267-288. MR1379242
40. Tsybakov, A. (2003). Optimal rates of aggregation. Computationnal Learning theory and Kernel Machines, Lecture Notes in Artificial Intelligence n.2777, Springer, Heidelberg. 303-313.
41. Tsybakov, A. (2009). Introduction to nonparametric estimation. Spinger, New York.MR2724359
42. Van de Geer, S. A. and Bühlmann, P. (2009). On the conditions used to prove oracle results for the LASSO. Electronic Journal of Statistics 3 1360-1392. MR2576316
43. Yang, Y. (2004). Aggregating regression procedures to improve performance. Bernoulli, 10 (1) 25-47. MR2044592
44. Zhang, T. (2008). From epsilon-entropy to KL-entropy: analysis of minimum information complexity density estimation. The Annals of Statistics 34 2180-2210. MR2291500
45. Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101 1418-1429. MR2279469
46. Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society B 67 (2) 301-320. MR2137327