On divergences and informations in statistics and information theory

IEEE Transactions on Information Theory

Volumn 52, Issue 10, 2006, Pages 4394-4412

  Liese, Friedrich ^b   Vajda, Igor ^a,c  

^a IEEE   (Czech Republic)

^b UNIVERSITY OF ROSTOCK   (Germany)

^c INSTITUTE OF INFORMATION THEORY AND AUTOMATION   (Czech Republic)

Author keywords

Arimoto divergence; Arimoto entropy; Arimoto information; Deficiency; Discrimination information; f divergence; Minimum f divergence estimators; Minimum f divergence tests; Shannon divergence; Shannon information; Statistical information; Sufficiency

Indexed keywords

ARIMOTO DIVERGENCE; ARIMOTO ENTROPY; ARIMOTO INFORMATION; DISCRIMINATION INFORMATION; F-DIVERGENCE; SHANNON DIVERGENCE; STATISTICAL INFORMATION;

DECISION MAKING; FUNCTIONS; INFORMATION THEORY; MAXIMUM LIKELIHOOD ESTIMATION; STATISTICAL METHODS;

INFORMATION ANALYSIS;

EID: 33947426775     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2006.881731     Document Type: Article

References (54)

1. M. S. Ali and D. Silvey, "A general class of coefficients of divergence of one distribution from another," J. Roy. Statist. Soc., ser. B, no. 28, pp. 131-140, 1966.
2. S. Arimoto, "Information-theoretical considerations on estimation problems," Info. Contr., vol. 19, pp. 181-194, 1971.
3. A. R. Barron, L. Györfi, and E. C. van der Meulen, "Distribution estimates consistent in total variation and two types of information divergence," IEEE Trans. Inf. Theory, vol. 38, no. 5, pp. 1437-1454, Sep. 1990.
4. A. Berlinet, I. Vajda, and E. C. van der Meulen, "About the asymptotic accuracy of Barron density estimates," IEEE Trans. Inf. Theory, vol. 44, no. 3, pp. 999-1009, May 1990.
5. A. Bhattacharyya, "On some analogues to the amount of information and their uses in statistical estimation," Sankhya, vol. 8, pp. 1-14, 1946.
6. R. E. Blahut, Principles and Practice of Information Theory. Reading, MA: Adison-Wesley, 1987.
7. A. Buzo, A. H. Gray Jr., R. M. Gray, and J. D. Markel, "Speech coding based upon vector quantization," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-28, no. 5, pp. 562-574, Oct. 1980.
8. H. Chernoff, "A measure of asymptotic efficiency for test of a hypothesis based on the sum of observations," Ann. Math. Statist., vol. 23, pp. 493-507, 1952.
9. B. S. Clarke and A. R. Barron, "Information-theoretic asymptotics of Bayes methods," IEEE Trans. Inf. Theory, vol. 36, no. 3, pp. 453-471, May 1990.
10. T. Cover and J. Thomas, Elements of Information Theory. New Ycrk: Wiley, 1991.
11. I. Csiszdr, "Eine Informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität on Markoffschen Ketten," Publ. Math. Inst. Hungar. AcaJ. Sei., ser. A, vol. 8, pp. 84-108, 1963.
12. --, "Information-type measures of difference of probability distributions and indirect observations," Studia Sci. Math. Hungar., vol. 2, pp. 299-318, 1967.
13. I. Csiszár and J. Körner, Information Theory. Coding Theorems for Discrete Memoryless Systems. Budapest, Hungary: Akademiai Kaidó, 1981.
14. I. Csiszar, "Generalized cutoff rates and Renyi information measures," IEEE Trans. Infm. Theory, vol. 41, no. 1, pp. 26-34, Jan. 1995.
15. M. H. De Groot, "Uncertainty, information and sequential experiments," Ann. Math. Statist., vol. 33, pp. 404-419, 1962.
16. _, Optimal Statistical Decisions. New York: McGraw Hill, 1970.
17. D. M. Endres and J. E. Schindelin, "A new metric for probability distributions," IEEE Trans. Inf. Theory, vol. 49, no. 7, pp. 1858-1860, Jul. 2003.
18. A.A.Fedotov, P.Harremoës, and F.Topsøe, "Refinements of Pinsker's inequality," IEEE Trans. Inf. Theory, vol. 49, no. 6, pp. 1491-1498, Jun. 2003.
19. A. Feinstein, Information Theory. New York: McGraw-Hill, 1958.
20. D. Feldman and F. Österreicher, "A note on f-divergences," Studia Sci. Math. Hungar., vol. 24, pp. 191-200, 1989.
21. I. M. Gel'fand, A. N. Kolmogorov, and A. M. Yaglom, "On the general definition ofthe amount of information," Dokl. Akad. Nauk. SSSR, vol. 11, pp. 745-748, 1956.
22. C. Guttenbrunner, "On applications of the representation of f-divergences as averaged minimal Bayesian risk," in Trans. 11th Prague Conf. Inf. Theory, Statist. Dec. Funct., Random Processes, Prague, 1992, vol. A, pp. 449-456.
23. L. Györfi, G. Morvai, and I. Vajda, "Information-theoretic methods in testing the goodness of fit," in Proc. IEEE Int. Symp. Information Theory, Sorrento, Italy, Jun. 2000, p. 28.
24. P. Harremoës and F. Topsøe, "Inequalities between entropy and the index of coincidence derived from information diagrams," IEEE Trans. Inf. Theory, vol. 47, no. 7, pp. 2944-2960, Nov. 2001.
25. H. Hewitt and K. Stromberg, Real and Abstract Analysis. Berlin, Germany: Springer, 1965.
26. L. K. Jones and C. L. Byrne, "Generalized entropy criteria for inverse problems, with applications to data comprassion, pattern classification and cluster analysis," IEEE Trans. Inf. Theory, vol. 36, no. 1, pp. 23-30, Jan. 1990.
27. T. Kailath, "The divergence and Bhattacharyya distance measures in signal selection," IEEE Trans. Commun. Technol., vol. COM-15, no. 1, pp. 52-60, Feb. 1967.
28. S. Kakutani, "On equivalence of infinite product measures," Ann. Math., vol. 49, pp. 214-224, 1948.
29. O. Kallenberg, Foundations of Modern Probability. New York: Spinger, 1997.
30. S. Kullback and R. Leibler, "On information and sufficiency," Ann. Math. Statist., vol. 22, pp. 79-86, 1951.
31. L. LeCam, Asymptotic Methods in Statistical Decision Theory. Berlin: Springer, 1986.
32. E. L. Lehmann, Theory of Point Estimation. New York: Wiley, 1983.
33. F. Liese and I. Vajda, Convex Statistical Distances. Leipzig, Germany: Teubner, 1987.
34. J. Lin, "Divergence measures based on the Shannon entropy," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 145-151, Jan. 1991.
35. M. Longo, T. D. Lookabaugh, and R. M. Gray, "Quantization for decentralized hypothesis testing under communication constraints," IEEE Trans. Inf. Theory, vol. 36, no. 2, pp. 241-255, Mar. 1991.
36. D. Morales, L. Pardo, and I. Vajda, "Uncertainty of discrete stochastic systems: General theory and statistical inference," IEEE Trans. Syst. Man Cybern., Part A, vol. 26, no. 6, pp. 681-697, Nov. 1996.
37. _, "Minimum divergence estimators based on grouped data," Ann. Inst. Statist. Math., vol. 53, pp. 277-288, 2001.
38. M. L. Menéndez, D. Morales, L. Pardo, and I. Vajda, "Some new statistics for testing hypotheses in parametric models," J. Multivar. Analysis, vol. 62, pp. 137-168.
39. D. Mussmann, "Sufficiency and f-divergences," Studia Sci. Math. Hungar., vol. 14, pp. 37-41, 1979.
40. F. Österreicher, "On a class of perimeter-type distances of probability distributions," Kybernetika, vol. 32, pp. 389-393, 1996.
41. F. Österreicher and I. Vajda, "Statistical information and discrimination," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 1036-1039, May 1993.
42. _, "A new class of metric divergences on probability spaces and its applicability in statistics," Ann. Inst. Statist. Math., vol. 55, no. 3, pp. 639-653, 2003.
43. H. V. Poor, "Robust decision design using a distance criterion," IEEE Trans. Inf. Theory, vol. IT-26, no. 5, pp. 575-587, Sep. 1980.
44. A. Rényi, "On measures of entropy and information," in Proc. 4th Berkeley Symp. Probability Theory and Mathematical Statist., Berkeley, CA, 1961, pp. 547-561.
45. M. R. C. Read and N. A. C. Cressie, Goodness-of-Fit Statistics for Discrete Multivariate Data. Berlin, Germany: Springer, 1988.
46. C. E. Shanon, "A mathematical theory of communication," Bell. Syst. Tech. J., vol. 27, pp. 379-423, 1948.
47. H. Strasser, Mathematical Theory of Statistics. Berlin, Germany: De Gruyter, 1985.
48. F. Topsøe, "Information-theoretical optimization techniques," Kybernetika, vol. 15, pp: 7-17, 1979.
49. _, "Some inequalities for information divergence and related measures of discrimination," IEEE Trans. Inf. Theory, vol. 46, no. 3, pp. 1602-1609, Apr. 2000.
50. E. Torgersen, Comparison of Statistical Experiments. Cambridge, U.K.: Cambridge Univ. Press, 1991.
51. I. Vajda, "On the f-divergence and singularity of probability measures," Periodica Math. Hungar., vol. 2, pp. 223-234, 1972.
52. α"-divergence and generalized Fisher's information," in Trans. 6th Prague Conf. Information Theory, Prague, Czechoslovakia, 1973, pp. 873-886.
53. _, Theory of Statistical Inference and Information. Boston, MA: Kluwer, 1989.
54. _, "On convergence of information contained in quantized observations," IEEE Trans. Inf. Theory, vol. 48, no. 8, pp. 2163-2172, Aug. 2002.