Information-theoretic determination of minimax rates of convergence

Annals of Statistics

Volumn 27, Issue 5, 1999, Pages 1564-1599

  Yang, Yuhong ^a   Barron, Andrew ^b  

^a IOWA STATE UNIVERSITY   (United States)

^b YALE UNIVERSITY   (United States)

Author keywords

Density estimation; Kullback Leibler distance; Metric entropy; Minimax risk

Indexed keywords

EID: 0033233737     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (70)

1. BALL, K. and PAJOR, A. (1990). The entropy of convex bodies with "few" extreme points. In Geometry of Banach Spaces 26-32. Cambridge Univ. Press.
2. BARRON, A. R. (1987). Are Bayes rules consistent in information? In Open Problems in Communication and Computation (T. M. Cover and B. Gopinath, eds.) 85-91. Springer, New York.
3. BARRON, A. R. (1991). Neural net approximation. In Proceedings of the Yale Workshop on Adaptive Learning Systems (K. Narendra, ed.) Yale University.
4. BARRON, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39 930-945.
5. BARRON, A. R. (1994). Approximation and estimation bounds for artificial neural networks. Machine Learning 14 115-133.
6. BARRON, A. R., BIRGÉ, L. and MASSART, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301-413.
7. BARRON, A. R. and COVER, T. M. (1991). Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 1034-1054.
8. BARRON, A. R. and HENGARTNER, N. (1998). Information theory and superefficiency. Ann. Statist. 26 1800-1825.
9. BARBON, A. R. and SHEU, C.-H. (1991). Approximation of density functions by sequences of exponential families. Ann. Statist. 19 1347-1369.
10. BICKEL, P. J. and RITOV, Y. (1988). Estimating integrated squared density derivatives: sharp best order of convergence estimates. Sankhyā Ser. A 50 381-393.
11. BIRGÉ, L. (1983). Approximation dans les espaces metriques et theorie de l'estimation. Z. Wahrsch. Verw. Gebiete 65 181-237.
12. BIRGÉ, L. (1986). On estimating a density using Hellinger distance and some other strange facts. Probab. Theory Related Fields 71 271-291.
13. BIRGÉ, L. and MASSART, P. (1993). Rates of convergence for minimum contrast estimators. Probab. Theory Related Fields 97 113-150.
14. BIRGÉ, L. and MASSART, P. (1994). Minimum contrast estimators on sieves. Technical report, Univ. Paris-Sud.
15. BIRGÉ, L. and MASSART, P. (1995). Estimation of integral functionals of a density. Ann. Statist. 23 11-29.
16. BIRGÉ, L. and MASSART, P. (1996). From model selection to adaptive estimation. In Research Papers in Probability and Statistics: Festschrift in Honor of Lucien Le Gam (D. Pollard, E. Torgersen and G. Yang, eds.) 55-87. Springer, New York.
17. BIRMAN, M. S. and SOLOMJAK, M. (1974). Quantitative analysis in Sobolev embedding theorems and application to spectral theory. Tenth Math. School Kiev 5-189.
18. BRETAGNOLLE, J. and HUBER, C. (1979). Estimation des densites: risque minimax. Z. Wahrsch. Verw. Gebiete 47 119-137.
19. CARL, B. (1981). Entropy numbers of embedding maps between Besov spaces with an application to eigenvalue problems. Proc. Roy. Soc. Edinburgh 90A 63-70.
20. CENCOV, N. N. (1972). Statistical Decision Rules and Optimal Inference. Nauka, Moscow; English translation in Amer. Math. Soc. Transl. 53 (1982).
21. CLARKE, B. and BARRON, A. R. (1990). Information-theoretic asymptotics of Bayes methods. IEEE Trans. Inform. Theory 36 453-471.
22. CLARKE, B. and BARRON, A. R. (1994). Jeffrey's prior is asymptotically least favorable under entropy risk. J. Statist. Plann. Inference 41 37-60.
23. COVER, T. M. and THOMAS, J. A. (1991). Elements of Information Theory. Wiley, New York.
24. COX, D. D (1988). Approximation of least squares regression on nested subspaces. Ann. Statist. 16 713-732.
25. DAVISSON, L. (1973). Universal noiseless coding. IEEE Trans. Inform. Theory 19 783-795.
26. DAVISSON, L. and LEON-GARCIA, A. (1980). A source matching approach to finding minimax codes. IEEE Trans. Inform. Theory 26 166-174.
27. DEVORE, R. A. and LORENTZ, G. G. (1993). Constructive Approximation. Springer, New York.
28. DEVROYE, L. (1987). A Course in Density Estimation. Birkhäuser, Boston.
29. DONOHO, D. L. (1993). Unconditional bases are optimal bases for data compression and for statistical estimation. Appl. Comput. Harmon. Anal. 1 100-115.
30. DONOHO, D. L. (1996). Unconditional bases and bit-level compression. Technical report 498, Dept. Statistics, Stanford Univ.
31. DONOHO, D. L., JOHNSTONE, I. M., KERKYACHARIAN, G. and PICARD, D. (1996). Density estimation by wavelet thresholding. Ann. Statist. 24 508-539.
32. DONOHO, D. L. and LIU, R. C. (1991). Geometrizing rates of convergence II. Ann. Statist. 19 633-667.
33. DUDLEY, R. M. (1987). Universal Donsker classes and metric entropy. Ann. Probab. 15 1306-1326.
34. EDMUNDS, D. E. and TRIEBEL, H. (1987). Entropy numbers and approximation numbers in function spaces. Proc. London Math. Soc. 58 137-152.
35. EFROIMOVICH, S. YU. and PINSKER, M. S. (1982). Estimation of square-integrable probability density of a random variable. Problemy Peredachi Informatsii 18 19-38.
36. FANO, R. M. (1961). Transmission of Information: A Statistical Theory of Communication. MIT Press.
37. FARRELL, R. (1972). On the best obtainable asymptotic rates of convergence in estimation of a density function at a point. Ann. Math. Statist. 43 170-180.
38. HASMINSKII, R. Z. (1978). A lower bound on the risks of nonparametric estimates of densities in the uniform metric. Theory Probab. Appl. 23 794-796.
39. HASMINSKII, R. Z. and IBRAGIMOV, I. A. (1990). On density estimation in the view of Kolmogorov's ideas in approximation theory. Ann. Statist. 18 999-1010.
40. HAUSSLER, D. (1997). A general minimax result for relative entropy. IEEE Trans. Inform. Theory 40 1276-1280.
41. HAUSSLER, D. and UPPER, M. (1997). Mutual information, metric entropy and cumulative relative entropy risk. Ann. Statist. 25 2451-2492.
42. IBRAGIMOV, I. A. and HASMINSKII, R. Z. (1977). On the estimation of an infinite-dimensional parameter in Gaussian white noise. Soviet Math. Dokl. 18 1307-1309.
43. IBRAGIMOV, I. A. and HASMINSKII, R. Z. (1978). On the capacity in communication by smooth signals. Soviet Math. Dokl. 19 1043-1047.
44. JONES, L. K. (1992). A simple lemma on greedy approximation in Hubert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20 608-613.
45. KOLMOGOROV, A. N. and TIHOMIROV, V. M. (1959). ∈-entropy and ∈-capacity of sets in function spaces. Uspekhi Mat. Nauk 14 3-86.
46. KOO, J. Y. and KIM, W. C. (1996). Wavelet density estimation by approximation of log-densities. Statist. Probab. Lett 26 271-278.
47. LE CAM, L. M. (1973). Convergence of estimates under dimensionality restrictions. Ann. Statist. 1, 38-53.
48. LE CAM, L. M. (1986). Asymptotic Methods in Statistical Decision Theory. Springer, New York.
49. LORENTZ, G. G. (1966). Metric entropy and approximation. Bull. Amer. Math. Soc. 72 903-937.
50. LORENTZ, G. G., GOLITSCHEK, M. V. and MAKOVOZ, Y. (1996). Constructive Approximation: Advanced Problems. Springer, New York.
51. MAKOVOZ, Y. (1996). Random approximants and neural networks. J. Approx. Theory 85 98-109.
52. MITJAGIN, B. S. (1961). The approximation dimension and bases in nuclear spaces. Uspekhi Mat. Nauk 16 63-132.
53. NEMIROVSKII, A. (1985). Nonparametric estimation of smooth regression functions. J. Comput. System. Sci. 23 1-11.
54. NEMIROVSKII, A., POLYAK, B. T. and TSYBAKOV, A. B. (1985). Rates of convergence of nonparamet-ric estimates of maximum-likelihood type. Probl. Peredachi Inf. 21 17-33.
55. POLLARD, D. (1993). Hypercubes and minimax rates of convergence. Preprint.
56. RlSSANEN, J. ( 1984). Universal coding, information, prediction, and estimation. IEEE Trans. In-form. Theory 30 629-636.
57. RISSANEN, J., SPEED, T. and Yu, B. (1992). Density estimation by stochastic complexity. IEEE Trans. Inform. Theory 38 315-323.
58. α(B) in the L2 metric. Dokl. Akad. Nauk SSSR 131 30-33.
59. STONE, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040-1053.
60. TEMLYAKOV, V. N. (1989). Estimation of the asymptotic characteristics of classes of functions with bounded mixed derivative or difference. Trudy Mat. Inst. Steklov 189 162-197.
61. TRIEBEL, H. (1975). Interpolation properties of ∈-entropy and diameters. Geometric characteristics of embedding for function spaces of Sobolev-Besov type. Mat. Sb. 98 27-41.
62. VAN DE GEER, S. (1990). Hellinger consistency of certain nonparametric maximum likelihood estimates. Ann. Statist. 21 14-44.
63. WONG, W. H. and SHEN, X. (1995). Probability inequalities for likelihood ratios and convergence rates of sieve MLEs. Ann. Statist. 23 339-362.
64. YANG, Y. (1999). Model selection for nonparametric regression. Statist. Sinica 9 475-500.
65. YANG, Y. (1999). Minimax nonparametric classification I: rates of convergence. IEEE Trans. Inform. Theory 45 2271-2284.
66. YANG, Y. and BARRON, A. R. (1997). Information-theoretic determination of minimax rates of convergence. Technical Report 28, Dept. Statistics, Iowa State Univ.
67. YANG, Y. and BARRON, A. R. (1998). An asymptotic property of model selection criteria. IEEE Trans. Inform. Theory 44 95-116.
68. YATRACOS, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov's entropy. Ann. Statist. 13 768-774.
69. YATRACOS, Y. G. (1988). A lower bound on the error in nonparametric regression type problems. Ann. Statist. 16 1180-1187.
70. Yu, B. (1996). Assouad, Fano, and Le Cam. In Research Papers in Probability and Statistics: Festschrift in Honor of Lucien Le Cam (D. Pollard, E. Torgersen and G. Yang, eds.) 423-435. Springer, New York.