Data-driven calibration of penalties for least-squares regression

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 245-279

  Arlot, Sylvain ^a   Massart, Pascal ^b  

^a Centre National de la Recherche Scientifique   (France)

^b UNIVERSITÉ PARIS SACLAY   (France)

Author keywords

Data driven calibration; Heteroscedastic data; Model selection by penalization; Non parametric regression; Regressogram

Indexed keywords

DATA-DRIVEN CALIBRATION; HETEROSCEDASTIC DATA; MODEL SELECTION BY PENALIZATION; NON-PARAMETRIC REGRESSION; REGRESSOGRAM;

CALIBRATION; COMPUTATIONAL FLUID DYNAMICS; CURVE FITTING; HEURISTIC METHODS; TRELLIS CODES;

REGRESSION ANALYSIS;

EID: 61749091779     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (45)

1. Hirotugu Akaike. Statistical predictor identification. Ann. Inst. Statist. Math., 22:203-217, 1970.
2. Hirotugu Akaike. Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (Tsahkadsor, 1971), pages 267-281. Akadémiai Kiadó, Budapest, 1973.
3. David M. Allen. The relationship between variable selection and data augmentation and a method for prediction. Technometrics, 16:125-127, 1974.
4. Sylvain Arlot. Resampling and Model Selection. PhD thesis, University Paris-Sud 11, December 2007. oai:tel.archives-ouvertes.fr:tel- 00198803-v1.
5. Sylvain Arlot. Suboptimality of penalties proportional to the dimension for model selection in heteroscedastic regression, December 2008a. arXiv:0812.3141.
6. Sylvain Arlot. V-fold cross-validation improved: V-fold penalization, February 2008b. arXiv:0802.0566v2.
7. Sylvain Arlot. Model selection by resampling penalization, March 2008c. oai:hal.archives-ouvertes.fr:hal-00262478-v1.
8. Yannick Baraud. Model selection for regression on a fixed design. Probab. Theory Related Fields, 117(4):467-493, 2000.
9. Yannick Baraud. Model selection for regression on a random design. ESAIM Probab. Statist., 6: 127-146 (electronic), 2002.
10. Yannick Baraud, Christophe Giraud, and Sylvie Huet. Gaussian model selection with unknown variance, 2007. To appear in The Annals of Statistics. arXiv:math.ST/0701250.
11. Andrew Barron, Lucien Birgé, and Pascal Massart. Risk bounds for model selection via penalization. Probab. Theory Related Fields, 113(3):301-413, 1999.
12. Peter L. Bartlett, Stéphane Boucheron, and Gábor Lugosi. Model selection and error estimation. Machine Learning, 48:85-113, 2002.
13. Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. Ann. Statist., 33(4): 1497-1537, 2005.
14. Jean-Patrick Baudry. Clustering through model selection criteria, June 2007. Poster session at One Day Statistical Workshop in Lisieux. http://www.math.u-psud.fr/~baudry.
15. Lucien Birgé and Pascal Massart. Gaussian model selection. J. Eur. Math. Soc. (JEMS), 3(3): 203-268, 2001.
16. Lucien Birgé and Pascal Massart. Minimal penalties for Gaussian model selection. Probab. Theory Related Fields, 138(1-2):33-73, 2007.
17. Stéphane Boucheron and Pascal Massart. A poor man's wilks phenomenon, March 2008. Personal communication.
18. Prabir Burman. Estimation of equifrequency histograms. Statist. Probab. Lett., 56(3):227-238, 2002.
19. Imre Csiszár. Large-scale typicality of Markov sample paths and consistency of MDL order estimators. IEEE Trans. Inform. Theory, 48(6): 1616-1628, 2002.
20. Imre Csiszár and Paul C. Shields. The consistency of the BIC Markov order estimator. Ann. Statist., 28(6):1601-1619, 2000.
21. Luc Devroye and Gábor Lugosi. Combinatorial Methods in Density Estimation. Springer Series in Statistics. Springer-Verlag, New York, 2001.
22. Bradley Efron. Estimating the error rate of a prediction rule: improvement on cross-validation. J. Amer. Statist. Assoc., 78(382):316-331, 1983.
23. Seymour Geisser. The predictive sample reuse method with applications. J. Amer. Statist. Assoc., 70:320-328, 1975.
24. Edward I. George and Dean P. Foster. Calibration and empirical Bayes variable selection. Biometrika, 87(4):731-747, 2000.
25. Clifford M. Hurvich and Chih-Ling Tsai. Regression and time series model selection in small samples. Biometrika, 76(2):297-307, 1989.
26. Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. IEEE Trans. Inform. Theory, 47(5): 1902-1914, 2001.
27. Vladimir Koltchinskii. Local Rademacher complexities and oracle inequalities in risk minimization. Ann. Statist., 34(6):2593-2656, 2006.
28. Marc Lavielle. Using penalized contrasts for the change-point problem. Signal Proces., 85(8): 1501-1510, 2005.
29. Emilie Lebarbier. Detecting multiple change-points in the mean of a Gaussian process by model selection. Signal Proces., 85:717-736, 2005.
30. Guillaume Lecué. Méthodes d'Agrégation : Optimalité et Vitesses Rapides. PhD thesis, LPMA, University Paris VII, May 2007.
31. Vincent Lepez. Some Estimation Problems Related to Oil Reserves. PhD thesis, University Paris XI, 2002.
32. L, cross-validation and generalized cross-validation: discrete index set. Ann. Statist., 15(3):958-975, 1987.
33. Fernando Lozano. Model selection using Rademacher penalization. In Proceedings of the 2nd ICSC Symp. on Neural Computation (NC2000). Berlin, Germany. ICSC Academic Press, 2000.
34. p. Technometrics, 15:661-675, 1973.
35. Pascal Massart. Concentration Inequalities and Model Selection, volume 1896 of Lecture Notes in Mathematics. Springer, Berlin, 2007.
36. Cathy Maugis and Bertrand Michel. A non asymptotic penalized criterion for gaussian mixture model selection. Technical Report 6549, INRIA, 2008.
37. p-test in the projection estimation of a regression. Teor. Veroyatnost. i Primenen., 35(2):305-317, 1990.
38. Xiaotong Shen and Jianming Ye. Adaptive model selection. J. Amer. Statist. Assoc., 97(457):210-221, 2002.
39. Ritei Shibata. An optimal selection of regression variables. Biometrika, 68(1):45-54, 1981.
40. Charles J. Stone. An asymptotically optimal histogram selection rule. In Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer, Vol. II (Berkeley, Calif., 1983), Wadsworth Statist./Probab. Ser., pages 513-520, Belmont, CA, 1985. Wadsworth.
41. M. Stone. Cross-validatory choice and assessment of statistical predictions. J. Roy. Statist. Soc. Ser. B, 36:111-147, 1974.
42. Nariaki Sugiura. Further analysis of the data by Akaike's information criterion and the finite corrections. Comm. Statist. A - Theory Methods, 7(1): 13-26, 1978.
43. Alexandre B. Tsybakov. Optimal aggregation of classifiers in statistical learning. Ann. Statist., 32 (1):135-166, 2004.
44. Nicolas Verzelen. Gaussian Graphical Models and Model Selection. PhD thesis, University Paris XI, December 2008.
45. Fanny Villers. Tests et Sélection de Modèles pour l'Analyse de Données Protéomiques et Transcriptomiques. PhD thesis, University Paris XI, December 2007.