Fast learning rates in statistical inference through aggregation

Annals of Statistics

Volumn 37, Issue 4, 2009, Pages 1591-1646

  Audibert, Jean Yves ^a,b  

^a UNIVERSITÉ PARIS EST   (France)

^b ECOLE NORMALE SUPÉRIEURE   (France)

Author keywords

Aggregation; Convex loss; Excess risk; Fast rates of convergence; Lower bounds in VC classes; Lq regression; Minimax lower bounds; Statistical learning

Indexed keywords

EID: 68649088331     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/08-AOS623     Document Type: Article

References (56)

1. ALQUIER, P. (2008). Iterative feature selection in least square regression estimation. Ann. Inst. H. Poincaré Probab. Statist. MR2451571
2. ANTOS, A. (2002). Lower bounds for the rate of convergence in nonparametric pattern recognition. Theoret. Comput. Sci. 284 3-24. MR1915064
3. ASSOUAD, P. (1983). Deux remarques sur l'estimation. C. R. Acad. Sci. Paris 296 1021-1024. MR0777600
4. AUDIBERT, J.-Y. (2004). Aggregated estimators and empirical complexity for least square regression. Ann. Inst. H. Poincaré Probab. Statist. 40 685-736. MR2096215
5. AUDIBERT, J.-Y. (2004). A better variance control for PAC-Bayesian classification. Preprint n. 905, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 and Paris 7. Available at http://www.proba.jussieu.fr/mathdoc/textes/PMA-905Bis.pdf.
6. AUDIBERT, J.-Y. (2004). Classification under polynomial entropy and margin assumptions and randomized estimators. Preprint n. 908, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 and Paris 7. Available at http://www.proba.jussieu.fr/mathdoc/preprints/ index.html#2004.
7. AUDIBERT, J.-Y. (2004). PAC-Bayesian statistical learning theory. Ph.D. thesis, Laboratoire de Probabilités et Modèles Aléatoires. Univ. Paris 6 and Paris 7. Available at http://certis.enpc. fr/~audibert/ThesePack.zip.
8. AUDIBERT, J.-Y. (2006). Fast learning rates in statistical inference through aggregation. Research Report 06-20, Certis-Ecole des Ponts. Available at http://arxiv.org/abs/math/0703854.
9. AUDIBERT, J.-Y. (2007). Progressive mixture rules are deviation suboptimal. Adv. Neural Inf. Process. Syst. 20.
10. AUDIBERT, J.-Y. and TSYBAKOV, A. B. (2007). Fast learning rates for plug-in classifiers. Ann. Statist. 35 608-633. MR2336861
11. BARRON, A. (1987). Are bayes rules consistent in information? In Open Problems in Communication and Computation (T. M. Cover and B. Gopinath, eds.) 85-91. Springer, New York. MR0922073
12. BARRON, A. and YANG, Y. (1999). Information-theoretic determination of minimax rates of convergence. Ann. Statist. 27 1564-1599. MR1742500
13. BARTLETT, P. L., BOUSQUET, O. and MENDELSON S. (2002). Localized rademacher complexities. In Proceedings of the 15th Annual Conference on Computational Learning Theory (K. Kivinen, ed.). Lecture Notes in Computer Science 2375 Springer, London. MR2040404
14. BIRGÉ, L. (1983). Approximation dans les espaces métriques et théorie de l'estimation. Z. Wahrsch. Verw. Gebiete 65 181-237. MR0722129
15. BIRGÉ, L. (2005). A new lower bound for multiple hypothesis testing. IEEE Trans. Inform. Theory 51 1611-1615. MR2241522
16. BLANCHARD, G. (1999). The progressive mixture estimator for regression trees. Ann. Inst. H. Poincaré Probab. Statist. 35 793-820. MR1725711
17. BOUCHERON, S., BOUSQUET, O. and LUGOSI, G. (2005). Theory of classification: Some recent advances. ESAIM Probab. Stat. 9 323-375. MR2182250
18. BRETAGNOLLE, J. and HUBER, C. (1979). Estimation des densités: Risque minimax. Z. Wahrsch. Verw. Gebiete 47 119-137. MR0523165
19. BUNEA, F. and NOBEL, A. (2005). Sequential procedures for aggregating arbitrary estimators of a conditional mean. Technical report. Available at http://stat.fsu.edu/~flori/ps/bnapril2005IEEE.pdf.
20. CATONI, O. (1997). A mixture approach to universal model selection. Preprint LMENS 97-30, Available at http://www.dma.ens.fr/edition/ preprints/Index.97.html.
21. CATONI, O. (1999). Universal aggregation rules with exact bias bound. Preprint n. 510. Available at http://www.proba.jussieu.fr/mathdoc/ preprints/index.html#1999.
22. CATONI, O. (2004). Statistical learning theory and stochastic optimization. In Ecole d'été de Probabilités de Saint-Flour XXXI-2001. Lecture Notes in Math. 1851. Springer, Berlin. MR2163920
23. CESA-BIANCHI, N., CONCONI, A. and GENTILE, C. (2004). On the generalization ability of on-line learning algorithms. IEEE Trans. Inform. Theory 50 2050-2057. MR2097190
24. CESA-BIANCHI, N., FREUND, Y., HAUSSLER, D., HELMBOLD, D. P., SCHAPIRE, R. E. and WARMUTH, M.K. (1997). How to use expert advice. J. ACM 44 427-485. MR1470152
25. CESA-BIANCHI, N. and LUGOSI, G. (1999). On prediction of individual sequences. Ann. Statist. 27 1865-1895. MR1765620
26. CESA-BIANCHI, N. and LUGOSI, G. (2006). Prediction, Learning and Games. Cambridge Univ. Press, Cambridge. MR2409394
27. CESA-BIANCHI, N., MANSOUR, Y. and STOLTZ, G. (2007). Improved second-order bounds for prediction with expert advice. Mach. Learn. 66 321-252.
28. CSISZAR, I. (1967). Information-type measures of difference of probability distributions and indirect observations. Stud. Math. Hung. 2 299-318. MR0219345
29. DEVROYE, L. (1982). Any discrimination rule can have an arbitrarily bad probability of error for finite sample size. IEEE Trans. Pattern Analysis and Machine Intelligence 4 154-157.
30. DEVROYE, L., GYÖRFI, L. and LUGOSI, G. (1996). A Probabilistic Theory of Pattern Recognition. Springer, New York. MR1383093
31. DEVROYE, L. and LUGOSI, G. (2000). Combinatorial Methods in Density Estimation. Springer, New York. MR1843146
32. DUDLEY, R. M. (1978). Central limit theorems for empirical measures. Ann. Probab. 6 899-929. MR0512411
33. HAUSSLER, D., KIVINEN, J. and WARMUTH, M. K. (1998). Sequential prediction of individual sequences under general loss functions. IEEE Trans. Inform. Theory 44 1906-1925. MR1664051
34. JUDITSKY, A., RIGOLLET, P. and TSYBAKOV, A. B. (2006). Learning by mirror averaging. Preprint n. 1034, Laboratoire de Probabilités et Modèles Aléatoires. Univ. Paris 6 and Paris 7. Available at http://arxiv.org/abs/math/0511468.
35. KIVINEN, J. and WARMUTH, M. K. (1999). Averaging expert predictions. 18 p. Available at www.cse.ucsc.edu/~manfred/pubs/C50.pdf. MR1724987
36. KIVINEN, J. and WARMUTH, M. K. (1999). Averaging expert predictions. In Lecture Notes in Computer Science 1572 153-167. Springer, Berlin. MR1724987
37. KOLTCHINSKII, V. (2006). Local rademacher complexities and oracle inequalities in risk minimization. Ann. Statist. 34. MR2329442
38. LECUÉ, G. (2007). Optimal rates of aggregation in classification under low noise assumption. Bernoulli 13 1000-1022. MR2364224
39. LECUÉ, G. (2007). Simultaneous adaptation to the margin and to complexity in classification. Ann. Statist. 35 1698-1721. MR2351102
40. LECUÉ, G. (2007). Suboptimality of penalized empirical risk minimization in classification. In Proceedings of the 20th annual conference on Computational Learning Theory (COLT). Lecture Notes in Computer Science 4539 142-156. Springer, Berlin. MR2397584
41. MAMMEN, E. and TSYBAKOV, A. B. (1999). Smooth discrimination analysis. Ann. Statist. 27 1808-1829. MR1765618
42. MASSART, P. (2000). Some applications of concentration inequalities to statistics. Ann. Fac. Sci. Toulouse Math. 9 245-303. MR1813803
43. MERHAV, N. and FEDER, M. (1998). Universal prediction. IEEE Trans. Inform. Theory 44 2124-2147. MR1658815
44. POLONIK, W. (1995). Measuring mass concentrations and estimating density contour clustersan excess mass approach. Ann. Statist. 23 855-881. MR1345204
45. TSYBAKOV, A. B. (2003). Optimal rates of aggregation. In Computational Learning Theory and Kernel Machines (B. Scholkopf and M. Warmuth, eds). Lecture Notes in Artificial Intelligence 2777 303-313. Springer, Heidelberg.
46. TSYBAKOV, A. B. (2004). Introduction à L'estimation non Paramétrique. Springer, Berlin. MR2013911
47. TSYBAKOV, A. B. (2004). Optimal aggregation of classifiers in statistical learning. Ann. Statist. 32 135-166. MR2051002
48. VAPNIK, V. (1982). Estimation of Dependences Based on Empirical Data. Springer, Berlin. MR0672244
49. VAPNIK, V. (1995). The Nature of Statistical Learning Theory, 2nd ed. Springer, New York. MR1367965
50. VAPNIK, V. and CHERVONENKIS, A. (1974). Theory of Pattern Recognition. Nauka, Moscow. MR0474638
51. VOVK, V. G. (1990). Aggregating strategies. In COLT '90: Proceedings of the Third Annual Workshop on Computational Learning Theory 371-386. Morgan Kaufmann, San Francisco, CA.
52. VOVK, V. G. (1998). A game of prediction with expert advice. J. Comput. System Sci. 153-173. MR1629690
53. YANG, Y. (2000). Combining different procedures for adaptive regression. J. Multivariate Anal. 74 135-161. MR1790617
54. YAROSHINSKY, R., EL-YANIV, R. and SEIDEN, S. S. (2004). How to better use expert advice. Mach. Learn. 55 271-309.
55. ZHANG, T. (2005). Data dependent concentration bounds for sequential prediction algorithms. In Proceedings of the 18th Annual Conference on Computational Learning Theory (COLT). Lecture Notes in Computer Science 173-187. Springer, Berlin. MR2203261
56. ZHANG, T. (2006). Information theoretical upper and lower bounds for statistical estimation. IEEE Trans. Inform. Theory 52 1307-1321. MR2241190