Axiomatic characterizations of information measures

Entropy

Volumn 10, Issue 3, 2008, Pages 261-273

  Csiszár, Imre ^a  

^a INSTITUTE OF EXPERIMENTAL MEDICINE   (Hungary)

Author keywords

Bregman distance; f divergence; f entropy; Functional equation; Kullback I divergence; Maximum entropy; Proper score; R nyi information measures; Shannon entropy; Transitive inference rule

Indexed keywords

EID: 54749100076     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e10030261     Document Type: Article

References (60)

1. Aczél, J.; Daróczy, Z. On Measures of Information and Their Characterizations; Academic Press: New York, 1975.
2. Aczél, J.; Daróczy, Z. A mixed theory of information I. RAIRO Inform. Theory 1978, 12, 149-155.
3. Aczél, J.; Forte, B.; Ng, C.T. Why Shannon and Hartley entropies are "natural". Adv. Appl. Probab. 1974, 6, 131-146.
4. Ahlswede, R.; Cai, N. An interpretation of identification entropy. IEEE Trans. Inf. Theory 2006, 52, 4198-4207.
5. Ali, S.M.; Silvey, S.D. A general class of coefficients of divergence of one distribution from another. J. Roy. Statist. Soc. B 1966, 25, 131-142.
6. Arimoto, S. Information-theoretic considerations on estimation problems. Information and Control 1971, 19, 181-194.
7. Arimoto, S. Information measures and capacity of order a for discrete memoryless channels. In Topics in Information Theory, Colloq. Math. Soc. J. Bolyai 16; Csiszár, I.; Elias, P., Eds.; North Holland: Amsterdam, 1977; pp. 41-52.
8. Ben-Bassat, M. f-entropies, probability of error, and feature selection. Information and Control 1978, 39, 227-242.
9. Bennett, C.; Brassard, G.; Crépeau, C.; Maurer, U. Generalized privacy amplification. IEEE Trans. Inf. Theory 1995, 47, 1915-1923.
10. Bhattacharyya, A. On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc. 1943, 35, 99-109.
11. Bregman, L.M. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comp. Math. and Math. Phys. 1967, 7, 200-217.
12. Campbell, L.L. A coding theorem and Rényi's entropy. Information and Control 1965, 8, 423-429.
13. Chaundry, T.W.; McLeod, J.B. On a functional equation. Edinburgh Mat. Notes 1960, 43, 7-8.
14. Csiszár, I. Eine informationstheorische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität von Markoffschen Ketten. Publ. Math. Inst. Hungar. Acad. Sci. 1963, 5, 85-107.
15. Csiszár, I. Information-type measures of difference of probability distributions and indirect observations. Studia Sei. Math. Hungar. 1967, 2, 299-318.
16. Csiszár, I. A class of measures of informativity of observation channels. Periodica Math. Hungar. 1972, 2, 191-213.
17. Csiszár, I. Information measures: a critical survey. In Trans. 7th Prague Conference on Inf. Theory, etc.; Academia: Prague, 1977, pp. 73-86.
18. Csiszár, I. Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems. Ann. Statist. 1991, 19, 2032-2066.
19. Csiszár, I. Generalized cutoff rates and Rényi information measures. IEEE Trans. Inf. Theory 1995, 41, 26-34.
20. Daróczy, Z. Über die gemeinsame Charakterisierung der zu den nicht vollständigen Verteilungen gehörigen Entropien von Shannon und von Rényi. Z. Wahrscheinlichkeitsth. Verw. Gebiete 1963, 7, 381-388.
21. Daróczy, Z. Über Mittelwerte und Entropien vollständiger Wahrscheinlichkeitsverteilungen. Acta Math. Acad. Sci. Hungar. 1964, 15, 203-210.
22. Daróczy, Z. Generalized information functions. Information and Control 1970, 16, 36-51.
23. Daróczy, Z. On the measurable solutions of a functional equation. Acta Math. Acad. Sci. Hungar. 1971, 34, 11-14.
24. Daróczy, Z.; Maksa, Gy. Nonnegative information functions. In Analytic Function Methods in Probability and Statistics, Colloq. Math. Soc. J. Bolyai 21; Gyires, B., Ed.; North Holland: Amsterdam, 1979; pp. 65-76.
25. Diderrich, G. The role of boundedness in characterizing Shannon entropy. Information and Control 1975, 29, 149-161.
26. Ebanks, B.; Sahoo, P.; Sander, W. Characterizations of Information Measures; World Scientific: Singapore, 1998.
27. Faddeev, D.K. On the concept of entropy of a finite probability scheme (in Russian). Uspehi Mat. Nauk 1956, 11, 227-231.
28. Fischer, P. On the inequality Σ;Pif(Pi) ≥ Σpif(qi). Metrika 1972, 18, 199-208.
29. Forte, B. Why Shannon's entropy. In Conv. Inform. Teor., Rome 1973, Symposia Math. 15; Academic Press: New York, 1975; pp. 137-152.
30. Grünwald, P.; Dawid, P. Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. Ann. Statist. 2004, 32, 1367-1433.
31. Havrda, J.; Charvát, F. Quantification method of classification processes. Concept of structural a-entropy. Kybernetika 1967, 3, 30-35.
32. Ingarden, R.S.; Urbanik, K. Information without probability. Colloq. Math. 1962, 9, 131-150.
33. Jaynes, E.T. Information theory and statistical mechanics. Phys. Rev. 1957, 106, 620-630.
34. Jones, L.K.; Byrne, C.L. General entropy criteria for inverse problems, with applications to data compression, pattern classification and cluster analysis. IEEE Trans. Inf. Theory 1990, 36, 23-30.
35. Kampé de Fériet, J.; Forte, B. Information et probabilité. C. R. Acad. Sci. Paris A 1967, 265, 110-114, 142-146, and 350-353.
36. Kannappan, Pl.; Ng, C.T. Measurable solutions of functional equations related to information theory. Proc. Amer. Math. Soc. 1973, 38, 303-310.
37. Kannappan, Pl. and Ng, C.T. A functional equation and its applications in information theory. Ann. Polon. Math. 1974, 30, 105-112.
38. Kolmogorov, A.N. A new invariant for transitive dynamical systems (in Russian). Dokl. Akad. NaukSSSR 1958, 119, 861-864.
39. Kullback, S. Information Theory and Statistics; Wiley: New York, 1959.
40. Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Statist. 1951, 22, 79-86.
41. Lee, P.M. On the axioms of information theory. Ann. Math. Statist. 1964, 35, 415-418.
42. Linhard, J.; Nielsen, V. Studies in statistical dynamics. Kong.Danske Vid. Selskab Mat-fys. Med. 1971, 38, 9, 1-42.
43. Maksa, Gy. On the bounded solutions of a functional equation. Acta Math. Acad. Sci. Hungar. 1981, 37, 445-440.
44. Matúš, F. Infinitely many information inequalities. In IEEE ISIT07 Nice, Symposium Proceedings; pp. 41-44.
45. Neumann, J. Thermodynamik quantenmechanischer Gesamtheiten. Gött. Nachr. 1927, 1, 273-291.
46. Paris, J.; Vencovská, A. A note on the inevitability of maximum entropy. Int'l J. Inexact Reasoning 1990, 4, 183-223.
47. Pippenger, N. What are the laws of information theory? In Special Problems on Communication and Computation Conference; Palo Alto, CA, Sep. 3-5, 1986.
48. Rényi, A. On measures of entropy and information. In Proc. 4th Berkeley Symp. Math. Statist. Probability, 1960; Univ. Calif. Press: Berkeley 1961; Vol. 1, pp. 547-561.
49. Rényi, A. On the foundations of information theory. Rev. Inst. Internat. Stat. 1965, 33, 1-4.
50. Sanov, I.N. On the probability of large deviations of random variables (in Russian). Mat. Sbornik 1957, 42, 11-44.
51. Schützenberger, M.P. Contribution aux applications statistiques de la théorie de l'information. Publ. Inst. Statist. Univ. Paris 1954, 3, 3-117.
52. Shannon, C.E. A mathematical theory of communication. Bell System Tech. J. 1948, 27, 379-423 and 623-656.
53. Shore, J.E.; Johnson, R.W. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans. Inf. Theory 1980, 26, 26-37.
54. Sibson, R. Information radius. Z. Wahrscheinlichkeitsth. Verw. Gebiete 1969, 14, 149-161.
55. Tsallis,C Possible generalizations of the Boltzmann-Gibbs statistics. J. Statist. Phys. 1988, 52, 479-487.
56. Tverberg, H. A new derivation of the information function. Math. Scand. 1958, 6, 297-298.
57. Vajda, I. Bounds on the minimal error probability for testing a finite or countable number of hypotheses (in Russian). Probl. Inform. Transmission 1968, 4, 9-17.
58. Wald, A. Sequential Analysis; Wiley: New York, 1947.
59. Yeung, R.W. A First Course in Information Theory; Kluwer: New York, 2002.
60. Zhang, Z.; Yeung, R.W. On characterizations of entropy function via information inequalities. IEEE Trans. Inf. Theory 1998, 44, 1440-1452.