Risk bounds for model selection via penalization

Probability Theory and Related Fields

Volumn 113, Issue 3, 1999, Pages 301-413

  Barron, Andrew ^a   Birgé, Lucien ^b   Massart, Pascal ^c  

^a YALE UNIVERSITY   (United States)

^b University of Paris VII LPSM partially supported by ERC Starting Grant 680275 MALIG   (France)

^c UNIVERSITÉ PARIS SUD   (France)

Author keywords

Adaptive estimation; Empirical processes; Minimum contrast estimators; Model selection; Penalization; Sieves

Indexed keywords

EID: 0033459856     PISSN: 01788051     EISSN: None     Source Type: Journal    
DOI: 10.1007/s004400050210     Document Type: Article

References (105)

1. Akaike, H.: Information theory and an extension of the maximum likelihood principle. In Proceedings 2nd International Symposium on Information Theory, P.N. Petrov and F. Csaki (Eds.). Akademia Kiado, Budapest, 267-281 (1973)
2. Assouad, P.: Deux remarques sur l'estimation. C. R. Acad. Sc. Paris Sér. I Math. 296, 1021-1024 (1983)
3. Baraud, Y.: Model selection for regression on a fixed design. Technical Report #97.49, Université Paris-Sud (1997)
4. Barron, A.R.: Complexity regularization with applications to artificial neural networks. In Nonparametric Functional Estimation (G. Roussas, ed.). Kluwer, Dordrecht, 561-576 (1991)
5. Barron, A.R.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory 39, 930-945 (1993)
6. Barron, A.R.: Approximation and estimation bounds for artificial neural networks. Machine Learning. 14, 115-133 (1994)
7. Barron, A.R., Cover, T.M.: Minimum complexity density estimation. IEEE Transactions on Information Theory 37, 1034-1054 (1991)
8. Barron, A.R., Sheu C.-H.: Approximation of density functions by sequences of exponential families. Ann. Statist. 19, 1347-1369 (1991)
9. Berger, M., Gauduchon, P. Mazet, E.: Le Spectre d'une Variété Riemannienne. Lecture Notes in Mathematics 194. Springer, Berlin (1971)
10. Birgé, L.: Approximation dans les espaces métriques et théorie de l'estimation. Z. Wahrscheinlichkeitstheorie Verw. Geb. 65, 181-237 (1983)
11. Birgé, L.: On estimating a density using Hellinger distance and some other strange facts. Probab. Th. Rel. Fields 71, 271-291 (1986)
12. Birgé, L.: The Grenander estimator: a non asymptotic approach. Ann. Statist. 17, 1532-1549 (1989)
13. Birgé, L., Massart, P.: Rates of convergence for minimum contrast estimators. Probab. Theory Relat. Fields 97, 113-150 (1993)
14. Birgé, L., Massart, P.: Minimum contrast estimators on sieves. Technical Report #94.34, Université Paris-Sud (1994)
15. Birgé, L., Massart, P.: From model selection to adaptive estimation. In Festschrift for Lucien Le Cam: Research Papers in Probability and Statistics (D. Pollard, E. Torgersen and G. Yang, eds.), 55-87. Springer-Verlag, New York (1997)
16. Birgé, L., Massart, P.: Minimum contrast estimators on sieves: exponential bounds and rates of convergence. Bernoulli 4, 329-375 (1998)
17. P. Mat. Sbornik 73, 295-317 (1967)
18. Breiman, L.: Hinging hyperplanes for regression, classification, and function approximation. IEEE Transactions on Information Theory 39, 999-1013 (1993)
19. Bretagnolle, J., Huber, C.: Estimation des densités: risque minimax. Z. Wahrscheinlichkeitstheorie verw. Gebiete 47, 119-137 (1979)
20. Cencov, N.N.: Statistical Decision Rules and Optimal Inference. Translations of Mathematical Monographs 53, American Math. Society, Providence (1982)
21. Chavel, I.: Eigenvalues in Riemannian Geometry. Academic Press, Orlando (1984)
22. Chow, Y.-S., Grenander, U.: A sieve method for the spectral density Ann. Statist. 13, 998-1010 (1985)
23. Cirel'son, B.S., Ibragimov, I.A., Sudakov, V.N.: Norm of gaussian sample function. In Proceedings of the 3rd Japan-U.S.S.R. Symposium on Probability Theory, Springer Lecture Notes in Mathematics 550 pp. 20-41. Springer-Verlag, Berlin (1976)
24. Cox, D.D.: Approximation of least squares regression on nested subspaces. Ann. Statist. 16, 713-732 (1988)
25. Dahmen, W., DeVore, R.A., Scherer, K.: Multidimensional spline approximation. SIAM J. Numer. Anal. 17, 380-402 (1980)
26. Daniel, C., Wood, F.S.: Fitting Equations to Data. Wiley, New York (1971)
27. Daubechies, I.: Ten Lectures on Wavelets. S.I.A.M., Philadelphia (1992)
28. DeVore, R.A., Lorentz, G.G.: Constructive Approximation. Springer-Verlag, Berlin (1993)
29. Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425-455 (1994a)
30. Donoho, D.L., Johnstone, I.M.: Ideal denoising in an orthonormal basis chosen from a library of bases. C. R. Acad. Sc. Paris Sér. I Math. 319, 1317-1322 (1994b)
31. q-error. Probab. Theory Relat. Fields 99, 277-303 (1994c)
32. Donoho, D.L., Johnstone, I.M.: Adapting to unknown smoothness via wavelet shrinkage. JASA. 90, 1200-1224 (1995)
33. Donoho, D.L., Johnstone, I.M.: Minimax estimation via wavelet shrinkage. Ann. Statist. 26, 879-921 (1998)
34. Donoho, D.L., Johnstone, I.M., Kerkyacharian, G., Picard, D.: Wavelet shrinkage: Asymptopia? J. R. Statist. Soc. B 57, 301-369 (1995)
35. Donoho, D.L., Johnstone, I.M., Kerkyacharian, G., Picard, D.: Density estimation by wavelet thresholding. Ann. Statist. 24, 508-539 (1996)
36. Dudley, R.M.: Central limit theorems for empirical measures. Ann. Probab. 6 899-929 (1978)
37. Dudley, R.M.: A course on empirical processes. In Ecole d'Eté de Probabilités de Saint-Flour XII - 1982. Lecture Notes in Mathematics 1097, Springer, Berlin (1984)
38. Efroimovich, S.Yu.: Nonparametric estimation of a density of unknown smoothness. Theory Probab. Appl. 30, 557-568 (1985)
39. Efroimovich, S.Yu., Pinsker, M.S.: Estimation of square-integrable density on the basis of a sequence of observations. Probl. Inf. Transm. 17, 182-196 (1981)
40. Efroimovich, S.Yu., Pinsker, M.S.: Estimation of square-integrable probability density of a random variable. Probl. Inf. Transm. 18, 175-189 (1982)
41. Efroimovich, S.Yu., Pinsker, M.S.: Learning algorithm for nonparametric filtering. Automat. Remote Control 11, 1434-1440, translated from Avtomatika i Telemekhanika 11, 58-65 (1984)
42. Efroimovich, S.Yu., Pinsker, M.S.: Learning algorithm for nonparametric filtering. Automat. Remote Control 11, 1434-1440, translated from Avtomatika i Telemekhanika 11, 58-65 (1984)
43. Efroimovich, S.Yu., Pinsker, M.S.: Self-tuning algorithm for minimax nonparametric estimation of spectral density. Probl. Inf. Transm. 22, 209-221 (1986)
44. Friedman, J.: Multivariate adaptive regression splines (with discussion). Ann. Statist. 19, 1-141 (1991)
45. Gallager, R.G.: Information Theory and Reliable Communication. Wiley, New York (1968)
46. Girosi, F., Anzellotti, G.: Convergence rates of approximation by translates. Artificial Intelligence Lab Technical Report 1288, Massachusetts Institute of Technology (1992)
47. Goldenshluger, A. and Nemirovski, A.: On spatially adaptive estimation of nonparametric regression. Math. Meth. Stat. 6, 135-170 (1997)
48. 2. Probl. Inf. Transm. 26, 15-20 (1990)
49. 2. Probl. Inf. Transm. 28, 44-54 (1992)
50. Golubev, G.K., Nussbaum, M.: Adaptive spline estimates for nonparametric regression models. Theory Probab. Appl. 37, 521-529 (1992)
51. Grenander, U.: Abstract inference. Wiley, New York (1981)
52. Hall, P.: Large-sample optimality of least-squares cross-validation in density estimation. Ann. Statist. 11, 1156-1174 (1983)
53. Hall, P.: Cross-validation and the smoothing of orthogonal series density estimators. J. Mult. Analysis 21, 207-237 (1987)
54. Haussler, D.: Sphere packing numbers for subsets of the Boolean n-cube with bounded Vapnik-Chervonenkis dimension. J. Combinatorial Theory A 69, 217-232 (1991)
55. Haussler, D.: Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation 100, 78-150 (1992)
56. Hornik, K., Stinchcombe, M. B., White, H., Auer, P.: Degree of approximation results for feedforward networks approximating unknown mappings and their derivatives. Neural Computation 6, 1262-1275 (1994)
57. Huber, P.J.: The behavior of maximum likelihood estimates under non-standard conditions. Proc. 5th Berkeley Symp. Math. Statist. Prob. Vol. 1, 221-233 (1967)
58. Ibragimov, I.A., Has'minskii, R.Z.: On estimate of the density function. Zap. Nauchn. Semin. LOMI 98, 61-85 (1980)
59. Ibragimov, I.A., Has'minskii, R.Z.: On the non-parametric density estimates. Zap. Nauchn. Semin. LOMI 108, 73-89 (1981)
60. Johnstone, I., Kerkyacharian, G., Picard, D.: Estimation d'une densité de probabilité par méthode d'ondelettes. C. R. Acad. Sc. Paris Sér. I Math. 315, 211-216 (1992)
61. Jones, L.K.: A simple lemma on greedy approximation in Hubert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20, 608-613 (1992)
62. Juditsky, A.: Wavelet estimators: adapting to unknown smoothness. Math. Methods of Statist 6, 1-25 (1997)
63. Kahane, J.-P.: Some random series of functions, 2nd ed. Cambridge University Press, Cambridge (1985)
64. Korostelev, A.P., Tsybakov, A.B.: Estimation of the density support and its functionals. Probl. Inf. Transm. 29, 1-15 (1993a)
65. Korostelev, A.P., Tsybakov, A.B.: Minimax Theory of Image Reconstruction. Lecture Notes in Statistics 82, Springer-Verlag, New York (1993b)
66. Le Cam, L.M.: Convergence of estimates under dimensionality restrictions. Ann. Statist. 1, 38-53 (1973)
67. Le Cam, L.M.: Asymptotic Methods in Statistical Decision Theory. Springer, New York (1986)
68. Le Cam, L.M., Yang, G.L.: Asymptotics in Statistics: Some Basic Concepts. Springer, New York (1990)
69. Ledoux, M.: Isoperimetry and Gaussian analysis. In Lectures on Probability Theory and Statistics, Ecole d'Eté de Probabilités de Saint-Flour XXIV-1994 (P. Bernard, ed.), 165-294. Springer, Berlin (1996)
70. Lepskii, O.V.: Asymptotically minimax adaptive estimation I: Upper bounds. Optimally adaptive estimates. Theory Probab. Appl. 36, 682-697 (1991)
71. Lepskii, O.V.: Asymptotically minimax adaptive estimation II: Statistical model without optimal adaptation. Adaptive estimators. Theory Probab. Appl. 37, 433-468 (1992)
72. Lepskii, O.V., Mammen, E., Spokoiny, V.G.: Optimal spatial adaptation to inhomogeneous smoothness: An approach based on kernel estimates with variable bandwidth selectors. Ann. Statist. 25, 929-947 (1997)
73. Lepskii, O.V., Spokoiny, V.G.: Local adaptation to inhomogeneous smoothness: resolution level. Math. Methods Statist. 4, 239-258 (1995)
74. L, cross-validation, and generalized cross-validation: Discrete index set. Ann. Statist. 15, 958-975 (1987)
75. McGaffrey, D.F., Gallant, A.R.: (1994). Convergence rates for single hidden layer feedforward networks. Neural Networks 7, 147-158 (1987)
76. Makovoz, Y.: Random approximants and neural networks. J. Approx. Th. 85, 98-109 (1996)
77. P. Technometrics 15, 661-675 (1973)
78. Meyer, Y: Ondelettes et Opérateurs I. Hermann, Paris (1990)
79. Modha, D.S., Masry, E.: Minimum complexity regression estimation with weakly dependent observations. IEEE Transactions on Information Theory 42, 2133-2145 (1996)
80. Nemirovskii, A.S.: Nonparametric estimation of smooth regression function. Izv. Akad. Nauk. SSSR Tekhn. Kibernet. 3. 50-60 (1985) (in Russian); Soviet. J. Comput. Systems Sci. 23, 1-11 (1986) (in English)
81. Nemirovskii, A.S.: Nonparametric estimation of smooth regression function. Izv. Akad. Nauk. SSSR Tekhn. Kibernet. 3. 50-60 (1985) (in Russian); Soviet. J. Comput. Systems Sci. 23, 1-11 (1986) (in English)
82. Pisier, G.: The Volume of Convex Bodies and Banach Space Geometry. Cambridge University Press, Cambridge (1989)
83. Pollard, D.: New ways to prove central limit theorems. Econometric Theory 1, 295-314 (1985)
84. P-test for the orthogonal series estimation of regression. Theory Probab. Appl. 35, 293-306 (1990)
85. Polyak, B.T., Tsybakov, A.B.: A family of asymptotically optimal methods for choosing the estimate order in orthogonal series regression. Theory Probab. Appl. 37, 471-481 (1993)
86. Rissanen, J.: Modeling by shortest data description. Automatica 14, 465-471 (1978)
87. Rissanen, J.: A universal prior for integers and estimation by minimum description length. Annals of Statistics 11, 416-431 (1983)
88. Schumaker, L.L.: Spline Functions: Basic Theory. Wiley, New York (1981)
89. Schwarz, G.: Estimating the dimension of a model. Ann. Statist. 6, 461-464 (1978)
90. Shen, X., Wong, W.H.: Convergence rates of sieve estimates. Ann. Statist. 22, 580-615 (1994)
91. Shibata, R.: An optimal selection of regression variables. Biometrika 68, 45-54 (1981)
92. Silverman, B.W.: On the estimation of a probability density function by the maximum penalized likelihood method. Ann. Statist. 10, 795-810 (1982)
93. Stein, E.M., Weiss, G.: Introduction to Fourier Analysis on Euclidean Spaces, Princeton University Press, Princeton (1971)
94. Stone, C.J.: An asymptotically optimal window selection rule for kernel density estimates. Ann. Statist. 12, 1285-1297 (1984)
95. Stone, C.J.: Large-sample inference for log-spline models. Ann. Statist. 18, 717-741 (1990)
96. Stone, C.J.: The use of polynomial splines and their tensor products in multivariate function estimation. Ann. Statist. 22, 118-184 (1994)
97. Talagrand, M.: Sharper bounds for empirical processes. Ann. Probab. 22, 28-76 (1994)
98. Talagrand, M.: New concentration inequalities in product spaces. Invent. Math. 126, 505-563 (1996)
99. Van de Geer, S.: Estimating a regression function. Ann. Statist. 18, 907-924 (1990)
100. Van de Geer, S.: The method of sieves and minimum contrast estimators. Math. Methods Statist. 4, 20-38 (1995)
101. Vapnik, V.: Estimation of Dependences Based on Empirical Data. Springer, New York (1982)
102. Wahba, G.: Spline Models for Observational Data. S.I.A.M., Philadelphia (1990)
103. Whittaker, E.T., Watson, G.N.: A Course of Modern Analysis. Cambridge University Press, London (1927)
104. Yang, Y., Barron, A.R.: An asymptotic property of model selection criteria. IEEE Transactions on Information Theory 44, 95-116 (1998)
105. Yukich, J.E., Stinchcombe, M.B., White, H.: Sup norm approximation bounds for networks through probabilistic methods. IEEE Transactions on Information Theory 41, 1021-1027 (1995)