A general minimax result for relative entropy

IEEE Transactions on Information Theory

Volumn 43, Issue 4, 1997, Pages 1276-1280

  Haussler, David ^a,b  

^a IEEE   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Bayes risk; Channel capacity; Computational learning theory; Density estimation; Kullback leibler divergence; Minimax redundancy; Minimax risk; Minimax theorem; Relative entropy; Source coding

Indexed keywords

CHANNEL CAPACITY; PROBABILITY; RANDOM PROCESSES; SET THEORY; SIGNAL ENCODING;

BAYES RISK; KULLBACK LEIBLER DIVERGENCE; RELATIVE ENTROPY;

INFORMATION THEORY;

EID: 0031190750     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.605594     Document Type: Article

References (19)

1. A. Barren and T. Cover, "A bound on the financial value of information," IEEE Trans. Inform. Theory, vol. 34, pp. 1097-1100, 1988.
2. L. L. Cam, "An extension of Wald's theory of statistical decision functions," Ann. Math. Statist., vol. 26, pp. 69-81, 1955.
3. B. Clarke and A. Barron, "Information-theoretic asymptotics of Bayes methods," IEEE Trans. Inform. Theory, vol. 36, pp. 453-471, 1990.
4. _, "Jefferys" prior is asymptotically least favorable under entropy risk," J. Statist. Planning Infer., vol. 41, pp. 37-60, 1994.
5. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
6. L. Davisson, "Universal noisless coding," IEEE Trans. Inform. Theory, vol. IT-19, pp. 783-795, 1973.
7. L. Davisson and A. Leon-Garcia, "A source matching approach to finding minimax codes," IEEE Trans. Inform. Theory, vol. IT-26, pp. 166-174, 1980.
8. R. M. Dudley, Real Analysis and Probability. Wadsworth, 1989.
9. S. Y. Efroimovich, "Information contained in a sequence of observations," Probl. Inform. Transm., vol. 15, pp. 178-189, 1980.
10. T. Ferguson, Mathematical Statistics: A Decision Theoretic Approach. New York: Academic, 1967.
11. R. Gallager, "Source coding with side information and universal coding," Tech. Rep. LIDS-P-937, MIT Lab. Inform. Decision Syst., Cambridge, MA, 1979.
12. D. Haussler and A. Barron, "How well do Bayes methods work for on-line prediction of {+1,-1} values? in Proc. 3rd NEC Symp. on Computation and Cognition, 1992.
13. D. Haussler, M. Kearns, and R. E. Schapire, "Bounds on the sample complexity of Bayesian learning using information theory and the Vapnik-Chervonenkis dimension," Mach. Learn., vol. 14, no. 1, pp. 83-113, 1994.
14. D. Haussler and M. Opper, "Mutual information, metric entropy, and risk in estimation of probability distributions," Tech. Rep. UCSC-CRL-96-27, Univ. of Calif., Comp. Res. Lab, Santa Cruz, CA, 1996. Shortened version to be published in Ann. Statist.
15. I. Ibragimov and R. Hasminskii, "On the information in a sample about a parameter," in Proc. 2nd Int. Symp. on Information Theory, 1972, pp. 295-309.
16. N. Merhav and M. Feder, "A strong version of the redundancy-capacity theorem of universal coding," IEEE Trans. Inform. Theory, vol. 41, pp. 714-722, May 1995.
17. M. S. Pinsker, Information and Information Stability of Random Variables and Processes. San Francisco, CA: Holden Day, 1964. (Translated.)
18. E. Posner, "Random coding strategies for minimum entropy," IEEE Trans. Inform. Theory, vol. IT-21, pp. 388-391, 1975.
19. J. Krob and H. Scholl, "A minimax result for the Kullback Leibler Bayes risk," Econ. Quality Contr., to be published in 1997.