Mutual information, metric entropy and cumulative relative entropy risk

Annals of Statistics

Volumn 25, Issue 6, 1997, Pages 2451-2492

  Haussler, David ^a   Opper, Manfred ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF WÜRZBURG   (Germany)

Author keywords

Bayes risk; Density estimation; Hellinger distance; Kullback leibler distance; Metric entropy; Minimax risk; Mutual information; Relative entropy

Indexed keywords

EID: 0031326925     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/aos/1030741081     Document Type: Article

References (57)

1. AMARI, S. (1982). Differential geometry of curved exponential families - curvatures and information loss. Ann. Statist. 10 357-385.
2. AMARI, S. and MURATA, N. (1993). Statistical theory of learning curves under entropic loss. Neural Comput. 5 140-153.
3. BARRON, A. (1985). The strong ergodic theorem for densities: generalized Shannon - McMilan-Breiman theorem. Ann. Probab. 13 1292-1303.
4. BARRON, A. (1987). Are Bayes rules consistent in information? In Open Problems in Communication and Computation (T. M. Cover and B. Gopinath, eds.) 85-91. Springer-Verlag, New York.
5. BARRON, A. (1987). The exponential convergence of posterior probabilities with implications for Bayes estimators of density functions. Technical Report 7, Dept. Statistics, Univ. Illinois Urbana-Champaign.
6. BARRON, A., CLARKE, B. and HAUSSLER, D. (1993). Information bounds for the risk of Bayesian predictions and the redundancy of universal codes. Proceedings of the International Symposium on Information Theory. IEEE Press, New York.
7. BARRON, A. and COVER, T. (1988). A bound on the financial value of information. IEEE Trans. Inform. Theory 34 1097-1100.
8. BARRON, A., GYÖRFI, L. and VAN DER MEULEN, E. (1992). Distribution estimation consistent in total variation and in two types of information divergence. IEEE Trans. Inform. Theory 38 1437-1454.
9. BARRON, A. and YANG, Y. (1995). Information theoretic lower bounds on convergence rates of nonparametric estimators. Unpublished manuscript.
10. BIRGÉ, L. (1983). Approximation dans les espaces métriques et théorie de l'estimation. Z. Wahrsch. Verw. Gebiete 65 181-237.
11. BIRGÉ, L. (1986). On estimating a density using Hellinger distance and some other strange facts. Probab. Theory Related Fields 71 271-291.
12. BIRGÉ, L. and MASSART, P. (1993). Rates of convergence for minimum contrast estimators. Probab. Theory Related Fields 97 113-150.
13. CAMERON, R. H. and MARTIN, W. T. (1944). Transformation of Wiener integrals under translations. Ann. Math. 45 386-396.
14. CLARKE, B. (1989). Asymptotic cumulative risk and Bayes risk under entropy loss with applications. Ph.D. thesis, Dept. Statistics, Univ. Illinois.
15. CLARKE, B. and BARRON, A. (1990). Information-theoretic asymptotics of Bayes methods. IEEE Trans. Inform. Theory 36 453-471.
16. CLARKE, B. and BARRON, A. (1994). Jefferys' prior is asymptotically least favorable under entropy risk. J. Statist. Plann. Inference 41 37-60.
17. CLEMENTS, G. F. (1963). Entropy of several sets of real-valued functions. Pacific J. Math. 13 1085-1095.
18. COVER, T. and THOMAS, J. (1991). Elements of Information Theory. Wiley, New York.
19. DAVISSON, L. and LEON-GARCIA, A. (1980). A source matching approach to finding minimax codes. IEEE Trans. Inform. Theory 26 166-174.
20. 1 View. Wiley, New York.
21. DIACONIS, P. and FREEDMAN, D. (1986). On the consistency of Bayes estimates. Ann. Statist. 14 1-26.
22. DUDLEY, R. M. (1984). A course on empirical processes. Lecture Notes in Math. 1097 2-142. Springer, New York.
23. EFROIMOVICH, S. Y. (1980). Information contained in a sequence of observations. Problems Inform. Transmission 15 178-189.
24. FEDER, M., FREUND, Y. and MANSOUR, Y. (1995). Optimal universal learning and prediction of probabilistic concepts. In Proceedings of the IEEE Information Theory Conference 233. IEEE, New York.
25. GALLAGER, R. (1979). Source coding with side information and universal coding. Technical Report LIDS-P-937, Laboratory for Information and Decision Systems, MIT.
26. GHOSH, J., GHOSAL, S. and SAMANTA, T. (1994). Stability and convergence of the posterior in non-regular problems. In Statistical Decision Theory and Related Topics. V (S. Gupta and J. O. Berger, eds.). Springer, New York.
27. GINÉ, E. and ZINN, J. (1984). Some limit theorems for empirical processes. Ann. Probab. 12 929-989.
28. GYÖRFI, L., PÁLI, I. and VAN DER MEULEN, E. (1994). There is no universal source code for an infinite alphabet. IEEE Trans. Inform. Theory 40 267-271.
29. HASMINSKII, R. and IBRAGIMOV, I. (1990). On density estimation in the view of Kolmogorov's ideas in approximation theory. Ann. Statist. 18 999-1010.
30. HAUSSLER, D. (1997). A general minimax result for relative entropy. IEEE Trans. Inform. Theory 40 1276-1280.
31. HAUSSLER, D. and BARRON, A. (1992). How well do Bayes methods work for on-line prediction of {+ 1, - 1} values? In Proceedings of the Third NEC Symposium on Computation and Cognition 74-100. SIAM, Philadelphia.
32. HAUSSLER, D., KEARNS, M. and SCHAPIRE, R. E. (1994). Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension. Machine Learning 14 83-113.
33. HAUSSLER, D. and OPPER, M. (1995). General bounds on the mutual information between a parameter and n conditionally independent observations. In Proceedings of the Seventh Annual ACM Workshop on Computational Learning Theory 402-411. ACM Press, New York.
34. HAUSSLER, D. and OPPER, M. (1996). Mutual information, metric entropy, and risk in estimation of probability distributions. Technical Report UCSC-CRL-96-27, Comput. Res. Lab., Univ. California, Santa Cruz.
35. IBRAGIMOV, I. and HASMINSKII, R. (1972). On the information in a sample about a parameter. In Second International Symposium on Information Theory 295-309. IEEE, New York.
36. IZENMAN, A. J. (1991). Recent developments in nonparametric density estimation. J. Amer. Statist. Assoc. 86 205-224.
37. KOLMOGOROV, A. N. and TIKHOMIROV, V. M. (1961). ε-entropy and ε-capacity of sets in functional spaces. Amer. Math. Soc. Trans. Ser. 2 17 277-364.
38. KOMAKI, F. (1994). On asymptotic properties of predictive distributions. Technical Report METR 94-21, Dept. Math. Engrg. Phys., Univ. Tokyo.
39. LE CAM, L. (1955). An extension of Wald's theory of statistical decision functions. Ann. Math. Statist. 26 69-81.
40. LECAM, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer, New York.
41. MEIR, R. and MERHAV, N. (1995). On the stochastic complexity of learning realizable and unrealizable rules. Machine Learning 19 241-261.
42. MERHAV, N. and FEDER, M. (1995). A strong version of the redundancy-capacity theorem of universal coding. IEEE Trans. Inform. Theory 41 714-722.
43. OPPER, M. and HAUSSLER, D. (1991). Calculation of the learning curve of Bayes optimal classification algorithm for learning a perceptron with noise. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory 75-87. Morgan Kaufmann, San Mateo, CA.
44. OPPER, M. and HAUSSLER, D. (1995). Bounds for predictive errors in the statistical mechanics of in supervised learning. Phys. Rev. Lett. 75 3772-3775.
45. PINSKER, M. S. (1964). Information and Information Stability of Random Variables and Processes. Holden-Day, Oakland, CA.
46. POLLARD, D. (1990). Empirical Processes: Theory and Applications. IMS, Hayward, CA.
47. RENYI, A. (1960). On measures of entropy and information. Proc. Fourth Berkeley Symp. Math. Statist. Probab. 1 547-561. Univ. California Press, Berkeley.
48. RENYI, A. (1964). On the amount of information concerning an unknown parameter in a sequence of observations. Publ. Math. Inst. Hungar. Acad. Sci. 9 617-625.
49. RISSANEN, J. (1986). Stochastic complexity and modeling. Ann. Statist. 14 1080-1100.
50. RISSANEN, J., SPEED, T. and YU, B. (1992). Density estimation by stochastic complexity. IEEE Trans. Inform. Theory 38 315-323.
51. SYMANZIK, K. (1965). Proof and refinements of an inequality of Feynman. J. Math. Phys. 6 1155-1165.
52. VAN DEGEER, S. (1993). Hellinger-consistency of certain nonparametric maximum likelihood estimators. Ann. Statist. 21 14-44.
53. WONG, W. and SHEN, X. (1995). Probability inequalities for likelihood ratios and convergence rates for sieve MLE's. Ann. Statist. 23 339-362.
54. YAMANISHI, K. (1995). A loss bound model for on-line stochastic prediction algorithms. Inform. Comput. 119 39-54.
55. YU, B. (1996). Lower bounds on expected redundancy for nonparametric classes. IEEE Trans. Inform. Theory 42 272-275.
56. ZHU, H. and ROHWER, R. (1995). Information geometric measurements of generalization. Technical Report NCRG 4350, Neural Computing Research Group, Aston Univ., England.
57. HAUSSLER, D. and OPPER, M. (1997). Metric entropy and minimax risk in classification. In Lecture Notes in Comp. Sci.: Studies in Logic and Comp. Sci. (J. Mycielski, G. Rozenberg and A. Salomaa, eds.) 1261 212-235. Springer-Verlag, New York.