Adaptive minimax regression estimation over sparse lq-hulls

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1675-1711

  Wang, Zhan ^a   Paterlini, Sandra ^b   Gao, Fuchang ^c   Yang, Yuhong ^d  

^a Fulcrum Analytics   (United States)

^b EBS BUSINESS SCHOOL   (Germany)

^c UNIVERSITY OF IDAHO   (United States)

^d UNIVERSITY OF MINNESOTA   (United States)

Author keywords

High dimensional sparse learning; Minimax rate of convergence; Model selection; Optimal aggregation

Indexed keywords

REGRESSION ANALYSIS;

ARBITRARY CORRELATION; HIGH-DIMENSIONAL; MINIMAX RATES; MODEL MISSPECIFICATION; MODEL SELECTION; REGRESSION ESTIMATION; SPARSE REPRESENTATION;

OPTIMIZATION;

EID: 84902815139     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (55)

1. J.-Y. Audibert. No fast exponential deviation inequalities for the progressive mixture rule. Available at http://arxiv.org/abs/math.ST/0703848v1, 2007.
2. J.-Y. Audibert. Fast learning rates in statistical inference through aggregation. Annals of Statistics, 37:1591-1646, 2009.
3. J.-Y. Audibert and O. Catoni. Risk bounds in linear regression through PAC-Bayesian truncation. Preprint at arXiv:1010.0072, 2010.
4. Y. Baraud. Model selection for regression on a fixed design. Probability Theory Related Fields, 117:467-493, 2000.
5. Y. Baraud. Model selection for regression on a random design. ESAIM Probab. Statist., 6: 127-146, 2002.
6. A. R. Barron, L. Birgé, and P. Massart. Risk bounds for model selection via penalization. Probability Theory and Related Fields, 113:301-413, 1999.
7. A.R. Barron and T.M. Cover. Minimum complexity density estimation. IEEE Transactions on Information Theory, 37:1034-1054, 1991.
8. L. Birgé. On estimating a density using Hellinger distance and some other strange facts. Probab. Theory Related Fields, 97:113-150, 1986.
9. L. Birgé. Model selection for Gaussian regression with random design. Bernoulli, 10:1039-1051, 2004. (Pubitemid 44242747)
10. L. Birgé. Model selection via testing: an alternative to (penalized) maximum likelihood estimators. Ann. I. H. Poincaré, 42:273-325, 2006. (Pubitemid 43467656)
11. 2-loss. Probab. Theory Related Fields, 158:533-574, 2014.
12. L. Birgé and P. Massart. Gaussian model selection. Journal of European Math. Society, 3: 203-268, 2001.
13. F. Bunea and A. Nobel. Sequential procedures for aggregating arbitrary estimators of a conditional mean. IEEE Transactions on Information Theory, 54:1725-1735, 2008. (Pubitemid 351512998)
14. F. Bunea, A.B. Tsybakov, and M.H. Wegkamp. Aggregation for gaussian regression. Annals of Statistics, 35:1674-1697, 2007.
15. O. Catoni. The mixture approach to universal model selection. Preprint, LMENS-97-30, Ecole Normale Supèrieure, Paris, France, 1997.
16. O. Catoni. Universal aggregation rules with exact bias bounds. Preprint 510, Laboratoire de Probabilitès et Modèles Aléatoires, Univ. Paris 6 and Paris 7, France. Available at http://www.proba.jussieu.fr/ mathdoc/preprints/index.html#1999, 1999.
17. O. Catoni. Statistical Learning Theory and Stochastic Optimization, Volume 1851. Springer, New York, 2004.
18. O. Catoni. Challenging the empirical mean and empirical variance: A deviation study. Ann. I. H. Poincaré, 48:909-1244, 2012.
19. N. Cesa-Bianchi and G. Lugosi. Prediction, Learning and Games. Cambridge University Press, 2006.
20. A. S. Dalalyan and J. Salmon. Sharp oracle inequalities for aggregation of affine estimators. Annals of Statistics, 40:2327-2355, 2012.
21. A. S. Dalalyan and A.B. Tsybakov. Aggregation by exponential weighting and sharp oracle inequalities. Lecture Notes in Computer Science, 4539:97-111, 2007.
22. A. S. Dalalyan and A.B. Tsybakov. Sparse regression learning by aggregation and Langevin Monte Carlo. Journal of Computer and System Science, 78:1423-1443, 2012a.
23. A. S. Dalalyan and A.B. Tsybakov. Mirror averaging with sparsity priors. Bernoulli, 18: 914-944, 2012b.
24. D. L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52:1289-1306, 2006.
25. p-balls for-error. Probability Theory and Related Fields, 99:277-303, 1994.
26. D. E. Edmunds and H. Triebel. Function spaces, entropy numbers, and differential operators. Cambridge Tracts in Mathematics, 120, 1998.
27. C. Giraud. Mixing least-squares estimators when the variance is unknown. Bernoulli, 14: 1089-1107, 2008.
28. A. Goldenshluger. A universal procedure for aggregating estimators. Annals of Statistics, 37:542-568, 2009.
29. L. Györfi and Gy. Ottucsák. Sequential prediction of unbounded stationary time series. IEEE Transactions on Information Theory, 53:1866-1872, 2007. (Pubitemid 46744816)
30. L. Györfi, M. Kohler, A. Krzyzak, and H. Walk. A Distribution-Free Theory of Nonpara-metric Regression. Springer, New York, 2002.
31. A. Juditsky and A. Nemirovski. Functional aggregation for nonparametric estimation. Annals of Statistics, 28:681-719, 2000.
32. A. Juditsky, P. Rigollet, and A.B. Tsybakov. Learning by mirror averaging. Annals of Statistics, 36:2183-2206, 2008.
33. T. Kühn. A lower estimate for entropy numbers. Journal of Approximation Theory, 110: 120-124, 2001. (Pubitemid 33619665)
34. G. Leung and A.R. Barron. Information theory and mixing least-squares regressions. IEEE Transactions on Information Theory, 52:3396-3410, 2006. (Pubitemid 44145107)
35. K. Lounici. Generalized mirror averaging and d-convex aggregation. Mathematical methods of statistic, 16:246-259, 2007.
36. P. Massart. Concentration Inequalities and Model Selection, Volume 1896. Lecture Notes in Mathematics, Springer, Berlin/Heidelberg, 2007.
37. S. Negahban, P. Ravikumar, M.J. Wainwright, and B. Yu. A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers. Statistical Science, 27:538-557, 2012.
38. A. Nemirovski. Topics in non-parametric statistics. Lecture Notes in Mathematics, Springer, New York, ecole d'eté de probabilités de saint-flour 1998 edition, 2000.
39. G. Pisier. Remarques sur un résultat non publié de b. maure. Sém. d'Analyse Fonctionnelle 1980/1981, Exp. V., 1981.
40. G. Raskutti, M. Wainwright, and B. Yu. Restricted eigenvalue properties for correlated gaussian designs. Journal of Machine Learning Research, 11:2241-2259, 2010.
41. G. Raskutti, M. Wainwright, and B. Yu. Minimax rates of estimation for high-dimensional linear regression over-balls. IEEE Transactions on Information Theory, 57:6976-699, 2012.
42. P. Rigollet and A.B. Tsybakov. Exponential screening and optimal rates of sparse estimation. Annals of Statistics, 39:731-771, 2010.
43. A.B. Tsybakov. Optimal rates of aggregation. Annals of Statistics, 39:731-771, 2003.
44. S.A. van de Geer and P. Bühlmann. On the conditions used to prove oracle results for the lasso. Electron. J. Statist, 3:1360-1392, 2009.
45. q-hulls. Arxiv preprint arXiv:1108.1961, 2011.
46. M. Wegkamp. Model selection in nonparametric regression. Annals of Statistics, 31:252-273, 2003.
47. Y. Yang. Minimax optimal density estimation. Ph.D. Dissertation, Department of Statistics, Yale University, 1996.
48. Y. Yang. Model selection for nonparametric regression. Statistica Sinica, 9:475-499, 1999.
49. Y. Yang. Combining different procedures for adaptive regression. Journal of Multivariate Analysis, 74:135-161, 2000a.
50. Y. Yang. Mixing strategies for density estimation. Annals of Statistics, 28:75-87, 2000b.
51. Y. Yang. Adaptive regression by mixing. Journal of American Statistical Association, 96: 574-588, 2001.
52. Y. Yang. Aggregating regression procedures to improve performance. Bernoulli, 10:25-47, 2004. An older version of the paper is Preprint #1999-17 of Department of Statistics at Iowa State University. (Pubitemid 44242782)
53. Y. Yang and A.R. Barron. An asymptotic property of model selection criteria. IEEE Trans. Inform. Theory, 44:95-116, 1998. (Pubitemid 128737882)
54. Y. Yang and A.R. Barron. Information theoretic determination of minimax rates of convergence. Annals of Statistics, 27:1564-1599, 1999.
55. C.H. Zhang. Nearly unbiased variable selection under minimax concave penalty. Annals of Statistics, 38:894-942, 2010.