Sharp oracle inequalities for aggregation of affine estimators

Annals of Statistics

Volumn 40, Issue 4, 2012, Pages 2327-2355

  Dalalyan, Arnak S ^a   Salmon, Joseph ^b  

^a CREST   (France)

^b NONE   (France)

Author keywords

Aggregation; Exponentially weighted aggregation.; Minimax risk; Model selection; Oracle inequalities; Regression

Indexed keywords

EID: 84884954887     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-AOS1038     Document Type: Article

References (66)

1. ALQUIER, P. and LOUNICI, K. (2011). PAC-Bayesian bounds for sparse regression estimation with exponential weights. Electron. J. Stat. 5 127-145. MR2786484
2. AMIT, Y. and GEMAN, D. (1997). Shape quantization and recognition with randomized trees. Neural Comput. 9 1545-1588.
3. ARLOT, S. and BACH, F. (2009). Data-driven calibration of linear estimators with minimal penalties. In NIPS 46-54. MIT Press, Vancouver.
4. AUDIBERT, J.-Y. (2007). Progressive mixture rules are deviation suboptimal. In NIPS 41-48. MIT Press, Vancouver.
5. BARAUD, Y., GIRAUD, C. and HUET, S. (2010). Estimator selection in the Gaussian setting. Unpublished manuscript.
6. BARRON, A., BIRGÉ, L. and MASSART, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301-413. MR1679028
7. BREIMAN, L. (1996). Bagging predictors. Mach. Learn. 24 123-140.
8. BUADES, A., COLL, B. andM OREL, J. M. (2005). A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4 490-530. MR2162865
9. BUNEA, F., TSYBAKOV, A. B. andW EGKAMP, M. H. (2007). Aggregation for Gaussian regression. Ann. Statist. 35 1674-1697. MR2351101
10. CAI, T. T. (1999). Adaptive wavelet estimation: A block thresholding and oracle inequality approach. Ann. Statist. 27 898-924. MR1724035
11. CATONI, O. (2004). Statistical Learning Theory and Stochastic Optimization. Lecture Notes in Math. 1851. Springer, Berlin. MR2163920
12. CAVALIER, L. (2008). Nonparametric statistical inverse problems. Inverse Problems 24 19. MR2421941
13. CAVALIER, L., GOLUBEV, G. K., PICARD, D. and TSYBAKOV, A. B. (2002). Oracle inequalities for inverse problems. Ann. Statist. 30 843-874. MR1922543
14. CAVALIER, L. and TSYBAKOV, A. (2002). Sharp adaptation for inverse problems with random noise. Probab. Theory Related Fields 123 323-354. MR1918537
15. CAVALIER, L. and TSYBAKOV, A. B. (2001). Penalized blockwise Stein's method, monotone oracles and sharp adaptive estimation. Math. Methods Statist. 10 247-282. MR1867161
16. CESA-BIANCHI, N. and LUGOSI, G. (2006). Prediction, Learning, and Games. Cambridge Univ. Press, Cambridge. MR2409394
17. DAI, D., RIGOLLET, P. and ZHANG, T. (2012). Deviation optimal learning using greedy Q -aggregation. Ann. Statist. To appear. Available at arXiv:1203.2507.
18. DAI, D. and ZHANG, T. (2011). Greedy model averaging. In NIPS 1242-1250. MIT Press, Granada.
19. DALALYAN, A. S. and SALMON, J. (2011). Competing against the best nearest neighbor filter in regression. In ALT. Lecture Notes in Computer Science 6925 129-143. Springer, Berlin.
20. DALALYAN, A. S. and SALMON, J. (2012). Supplement to "Sharp oracle inequalities for aggregation of affine estimators." DOI:10.1214/12-AOS1038SUPP.
21. DALALYAN, A. S. and TSYBAKOV, A. B. (2007). Aggregation by exponential weighting and sharp oracle inequalities. In Learning Theory. Lecture Notes in Computer Science 4539 97-111. Springer, Berlin. MR2397581
22. DALALYAN, A. S. and TSYBAKOV, A. B. (2008). Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Mach. Learn. 72 39-61.
23. DALALYAN, A. S. and TSYBAKOV, A. B. (2012). Sparse regression learning by aggregation and Langevin Monte-Carlo. J. Comput. System Sci. 78 1423-1443. MR2926142
24. DALALYAN, A. S. and TSYBAKOV, A. B. (2012). Mirror averaging with sparsity priors. Bernoulli 18 914-944.
25. DONOHO, D. L. and JOHNSTONE, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 425-455. MR1311089
26. DONOHO, D. L. and JOHNSTONE, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. J. Amer. Statist. Assoc. 90 1200-1224. MR1379464
27. DONOHO, D. L., LIU, R. C. andM AC G IBBON, B. (1990).Minimax risk over hyperrectangles, and implications. Ann. Statist. 18 1416-1437. MR1062717
28. EFROMOVICH, S. and PINSKER, M. (1996). Sharp-optimal and adaptive estimation for heteroscedastic nonparametric regression. Statist. Sinica 6 925-942. MR1422411
29. EFROѣMOVICH, S. Y. and PINSKER, M. S. (1984). A self-training algorithm for nonparametric filtering. Avtomat. i Telemekh. 11 58-65. MR0797991
30. FREUND, Y. (1990). Boosting a weak learning algorithm by majority. In COLT 202-216. Morgan Kaufmann, Rochester.
31. G AÏFFAS, S. and LECUÉ, G. (2011). Hyper-sparse optimal aggregation. J. Mach. Learn. Res. 12 1813-1833. MR2819018
32. GEORGE, E. I. (1986). Minimax multiple shrinkage estimation. Ann. Statist. 14 188-205. MR0829562
33. GERCHINOVITZ, S. (2011). Sparsity regret bounds for individual sequences in online linear regression. J. Mach. Learn. Res. 19 377-396.
34. GIRAUD, C. (2008). Mixing least-squares estimators when the variance is unknown. Bernoulli 14 1089-1107. MR2543587
35. GOLDENSHLUGER, A. and LEPSKI, O. (2008). Universal pointwise selection rule in multivariate function estimation. Bernoulli 14 1150-1190. MR2543590
36. GOLUBEV, Y. (2010). On universal oracle inequalities related to high-dimensional linear models. Ann. Statist. 38 2751-2780. MR2722455
37. JUDITSKY, A. andN EMIROVSKI, A. (2000). Functional aggregation for nonparametric regression. Ann. Statist. 28 681-712. MR1792783
38. JUDITSKY, A. and NEMIROVSKI, A. (2009). Nonparametric denoising of signals with unknown local structure. I. Oracle inequalities. Appl. Comput. Harmon. Anal. 27 157-179. MR2543191
39. KIVINEN, J. andWARMUTH, M. K. (1999). Averaging expert predictions. In Computational Learning Theory (Nordkirchen, 1999). Lecture Notes in Computer Science 1572 153-167. Springer, Berlin. MR1724987
40. KNEIP, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835-866. MR1292543
41. LANCKRIET, G. R. G., CRISTIANINI, N., BARTLETT, P., E L GHAOUI, L. and JORDAN, M. I. (2003/04). Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res. 5 27-72. MR2247973
42. LANGFORD, J. and SHAWE-TAYLOR, J. (2002). PAC-Bayes & margins. In NIPS 423-430. MIT Press, Vancouver.
43. LECUÉ, G. andM ENDELSON, S. (2012). On the optimality of the aggregate with exponential weights for low temperatures. Bernoulli. To appear.
44. LEUNG, G. (2004). Information theory and mixing least squares regression. Ph.D. thesis, Yale Univ.
45. LEUNG, G. and BARRON, A. R. (2006). Information theory and mixing least-squares regressions. IEEE Trans. Inform. Theory 52 3396-3410. MR2242356
46. LOUNICI, K. (2007). Generalized mirror averaging and D -convex aggregation. Math. Methods Statist. 16 246-259. MR2356820
47. M C A LLESTER, D. A. (1998). Some PAC-Bayesian theorems. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory (Madison, WI, 1998) 230-234 (electronic). ACM, New York. MR1811587
48. NEMIROVSKI, A. (2000). Topics in non-parametric statistics. In Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math . 1738 85-277. Springer, Berlin. MR1775640
49. PINSKER, M. S. Optimal filtration of square-integrable signals in Gaussian noise. Probl. Peredachi Inf. 16 52-68. MR0624591
50. POLZEHL, J. and SPOKOINY, V. G. (2000). Adaptive weights smoothing with applications to image restoration. J. R. Stat. Soc. Ser. B Stat. Methodol. 62 335-354. MR1749543
51. RIGOLLET, P. (2012). Kullback-Leibler aggregation and misspecified generalized linear models. Ann. Statist. 40 639-665.
52. RIGOLLET, P. and TSYBAKOV, A. (2011). Exponential screening and optimal rates of sparse estimation. Ann. Statist. 39 731-771. MR2816337
53. RIGOLLET, P. and TSYBAKOV, A. B. (2007). Linear and convex aggregation of density estimators. Math. Methods Statist. 16 260-280. MR2356821
54. RIGOLLET, P. and TSYBAKOV, A. B. (2011). Sparse estimation by exponential weighting. Unpublished manuscript.
55. SALMON, J. andDALALYAN, A. S. (2011). Optimal aggregation of affine estimators. J. Mach. Learn. Res. 19 635-660.
56. SALMON, J. and L E PENNEC, E. (2009). NL-Means and aggregation procedures. In ICIP 2977-2980. IEEE, Caìro.
57. SEEGER, M. (2003). PAC-Bayesian generalisation error bounds for Gaussian process classification. J. Mach. Learn. Res. 3 233-269. MR1971338
58. SHAWE-TAYLOR, J. and CRISTIANINI, N. (2000). An Introduction to Support Vector Machines: And Other Kernel-Based Learning Methods. Cambridge Univ. Press, Cambridge.
59. STEIN, C. M. (1973). Estimation of the mean of a multivariate distribution. In Proc. Prague Symp. Asymptotic Statist. Charles Univ., Prague.
60. TSYBAKOV, A. B. (2003). Optimal rates of aggregation. In COLT 303-313. Springer, Washington, DC.
61. TSYBAKOV, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York. MR2724359
62. WANG, Z., PATERLINI, S., GAO, F. and YANG, Y. (2012). Adaptive minimax estimation over sparse lq -hulls. Technical report. Available at arXiv:1108.1961v4 [math.ST].
63. YANG, Y. (2000). Combining different procedures for adaptive regression. J. Multivariate Anal. 74 135-161. MR1790617
64. YANG, Y. (2003). Regression with multiple candidate models: Selecting or mixing? Statist. Sinica 13 783-809. MR1997174
65. YANG, Y. (2004). Aggregating regression procedures to improve performance. Bernoulli 10 25-47. MR2044592
66. YUAN, M. and LIN, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 49-67. MR2212574