Deviation optimal learning using greedy q-aggregation

Annals of Statistics

Volumn 40, Issue 3, 2012, Pages 1878-1905

  Dai, Dong ^a   Rigollet, Philippe ^b   Zhang, Tong ^a  

^a RUTGERS UNIVERSITY   (United States)

^b Princeton University   (United States)

Author keywords

Deviation bounds; Deviation suboptimality; Exponential weights Oracle inequalities; Greedy algorithm; Lower bounds; Model averaging; Model selection; Regression

Indexed keywords

EID: 84872080947     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-AOS1025     Document Type: Article

References (27)

1. AUDIBERT, J.-Y. (2004). Aggregated estimators and empirical complexity for least square regression. Ann. Inst. Henri Poincaré Probab. Stat. 40 685-736. MR2096215
2. AUDIBERT, J.-Y. (2008). Progressive mixture rules are deviation suboptimal. In Advances in Neural Information Processing Systems 20 (J. C. Platt, D. Koller, Y. Singer and S. Roweis, eds.) 41-48. MIT Press, Cambridge, MA.
3. BARRON, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39 930-945. MR1237720
4. BECK, A. and TEBOULLE, M. (2009). A fast iterative shrinkage-Thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2 183-202. MR2486527
5. BOYD, S. and VANDENBERGHE, L. (2004). Convex Optimization. Cambridge Univ. Press, Cambridge. MR2061575
6. CATONI, O. (1999). Universal aggregation rules with exact bias bounds. Technical report, Laboratoire de Probabilités et Modeles Aléatoires, Paris.
7. CLARKSON, K. L. (2008). Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms 922-931. ACM, New York. MR2487663
8. DAI, D. and ZHANG, T. (2011). Greedy model averaging. In Advances in Neural Information Processing Systems 24 (J. Shawe-Taylor, R. S. Zemel, P. Bartlett, F. C. N. Pereira and K. Q. Weinberger, eds.) 1242-1250. MIT Press, Cambridge, MA.
9. DALALYAN, A. S. and SALMON, J. (2011). Sharp oracle inequalities for aggregation of affine estimators. Available at arXiv:1104.3969.
10. DALALYAN, A. S. and TSYBAKOV, A. B. (2007). Aggregation by exponential weighting and sharp oracle inequalities. In Learning Theory. Lecture Notes in Computer Science 4539 97-111. Springer, Berlin. MR2397581
11. DALALYAN, A. and TSYBAKOV, A. B. (2008). Aggregation by exponential weighting, sharp PACBayesian bounds and sparsity. Machine Learning 72 39-61.
12. FRANK, M. and WOLFE, P. (1956). An algorithm for quadratic programming. Naval Res. Logist. Quart. 3 95-110. MR0089102
13. GAÏFFAS, S. and LECUÉ, G. (2011). Hyper-sparse optimal aggregation. J. Mach. Learn. Res. 12 1813-1833. MR2819018
14. JAGGI, M. (2011). Convex optimization without projection steps. Available at arXiv:1108.1170v6.
15. JONES, L. K. (1992). A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20 608-613. MR1150368
16. JUDITSKY, A. and NEMIROVSKI, A. (2000). Functional aggregation for nonparametric regression. Ann. Statist. 28 681-712. MR1792783
17. JUDITSKY, A., RIGOLLET, P. and TSYBAKOV, A. B. (2008). Learning by mirror averaging. Ann. Statist. 36 2183-2206. MR2458184
18. LAURENT, B. and MASSART, P. (2000). Adaptive estimation of a quadratic functional by model selection. Ann. Statist. 28 1302-1338. MR1805785
19. LECUÉ, G. and MENDELSON, S. (2009). Aggregation via empirical risk minimization. Probab. Theory Related Fields 145 591-613. MR2529440
20. LECUÉ, G. and MENDELSON, S. (2012). On the optimality of the aggregate with exponential weights for low temperatures. Bernoulli. To appear.
21. NEMIROVSKI, A. (2000). Topics in non-parametric statistics. In Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math. 1738 85-277. Springer, Berlin. MR1775640
22. RIGOLLET, P. (2012). Kullback-Leibler aggregation and misspecified generalized linear models. Ann. Statist. 40 639-665.
23. RIGOLLET, P. and TSYBAKOV, A. (2011). Exponential screening and optimal rates of sparse estimation. Ann. Statist. 39 731-771. MR2816337
24. RIGOLLET, P. and TSYBAKOV, A. (2012). Sparse estimation by exponential weighting. Statist. Sci. To appear. Available at arXiv:1108.5116.
25. SHALEV-SHWARTZ, S., SREBRO, N. and ZHANG, T. (2010). Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM J. Optim. 20 2807-2832. MR2721156
26. TSYBAKOV, A. B. (2003). Optimal rates of aggregation. In COLT (B. Schölkopf andM. K.Warmuth, eds.). Lecture Notes in Computer Science 2777 303-313. Springer, Berlin.
27. YANG, Y. (1999). Model selection for nonparametric regression. Statist. Sinica 9 475-499. MR1707850