Some inequalities for information divergence and related measures of discrimination

IEEE Transactions on Information Theory

Volumn 46, Issue 4, 2000, Pages 1602-1609

  Topsøe, Flamming ^a  

^a UNIVERSITY OF COPENHAGEN   (Denmark)

Author keywords

[No Author keywords available]

Indexed keywords

COMMUNICATION CHANNELS (INFORMATION THEORY); DATA COMMUNICATION SYSTEMS; POLYNOMIALS; PROBABILITY; PROBABILITY DISTRIBUTIONS; THEOREM PROVING;

CAPACITORY DISCRIMINATION; HELLINGER DISTANCE; INFORMATION DIVERGENCE; PINSKER INEQUALITY; TRIANGULAR DISCRIMINATION;

INFORMATION THEORY;

EID: 0034224418     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.850703     Document Type: Article

References (42)

1. J. Abrahams, On the selection of measures of distance between probability distributions, in Proc. IEEE Int. Symp. Information Theory, 1981. New York: Elsevier, 1982, pp. 109-113.
2. R. Ahlswede and I. Wegener, Search Problems. Chichester, U.K.: Wiley, 1987. German original by Teubner, 1979.
3. S. M. Ali and S. D. Silvey, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Ser. B, vol. 28, pp. 131-142, 1996.
4. C. Berg, J. P. R. Christensen, and P. Ressel, Harmonic Analysis on Semigroups. New York: Springer, 1984.
5. L. Birgé, Approximation dans les espaces métriques et théorie de l'estimation.' Z. Wahrscheinlichkeitstheorie verw.Geb., vol. 65, pp. 181-237, 1983.
6. L. Birge and P. Massart, Rates of convergence for minimum contrast estimator, Probab. Theory Relat. Fields, vol. 97, pp. 113-150, 1993.
7. I. Csiszár, A note on Jensen's inequality, Studio Sei. Math. Hungar., vol. 1, pp. 185-188, 1966.
8. _, Information-type measures of difference of probability distributions and indirect observation, Studio Sei. Math. Hungar., vol. 2, pp. 299-318, 1967.
9. _, On topological properties of f-divergences, Studio Sci. Math. Hungar., vol. 2, pp. 329-339, 1967.
10. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
11. I. Csiszär, Sanov property, generalized I-projection and a conditional limit theorem, Ann. Probab., vol. 12, pp. 768-793, 1984.
12. D. Dacunha-Castelle, Ecole d'Ete de Probabilites de Saint-Flour VII-1977. Berlin, Heidelberg, New York: Springer, 1978.
13. A. Dembo, Information inequalities and concentration of measure, Ann. Probab., vol. 25, pp. 927-939, 1997.
14. D. Feldman and F. osterreicher, A note on /-divergences, Studio Sei. Math. Hungar, vol. 24, pp. 191-200, 1989.
15. L. Györfi and I. Vajda, A class of modified Pearson and Neyman statistics, Statist. Decisions, to be published.
16. P. Harremoës and F. Topsøe, Instances of exact prediction and universal coding. In preparation.
17. _, Inequalities which connect entropy and index of coincidence. Unpublished.
18. P. Kafka, F. Osterreicher, and I. Vincze, On powers of f-divergences defining a distance, Studio Sei. Math. Hungar., vol. 26, pp. 415-422, 1991.
19. N. S. Kambo and S. Kotz, On exponential bounds for binomial probabilities, Ann. Inst. Statist. Math., vol. 18, pp. 277-287, 1966.
20. O. Krafft, A note on exponential bounds for binomial probabilities, Ann. Inst. Statist. Math., vol. 21, pp. 219-220, 1969.
21. O. Krafft and N. Schmilz, A note on Hoefding's inequality, J. Amer. Statist. Assoc., vol. 64, pp. 907-912, 1969.
22. C. Kraft, Some conditions for consistency and uniform consistency of statistical procedures, Univ. Calif. Publ. in Statistics, vol. 1, pp. 125-142, 1955.
23. S. Kullback, Information Theory and Statistics. New York: Wiley, 1959.
24. S. Kullback and R. Leibler, On information and sufficiency, Ann. Math. Statist., vol. 22, pp. 79-86, 1951.
25. L. LeCam, Asymptotic Methods in Statistical Decision Theory. New York: Springer, 1986.
26. J. Lin, Divergence measures based on the Shannon entropy, IEEE Trans. Inform. Theory, vol. 37, pp. 145-151, Jan. 1991.
27. J. Lin and S. K. M. Wong, A new directed divergence measure and its characterization, Int. J. Gen. Syst., vol. 17, pp. 73-81, 1990.
28. K. Marton, Bounding d-distance by informational divergence: A method to prove measure concentration, Ann. Probab., vol. 24, pp. 857-866, 1966.
29. _, A simple proof of the blowing-up lemma, IEEE Trans. Inform. Theory, vol. IT-32, pp. 445-446, May 1986.
30. M. Menendez, D. Morales, L. Pardo, and I. Vajda, Two approaches to grouping of data and related disparaty statistics, Commun. Statist.-Theory and Methods, vol. 27, pp. 609-633, 1998.
31. F. Österreicher and I. Vajda, Statistical information and discrimination, IEEE Trans. Inform. Theory, vol. 39, pp. 1036-1039, May 1993.
32. J. Pecaric, On Directed Divergence Measures, unpublished, 1999, private communication.
33. M. S. Pinsker, Information and Information Stability of Random Variables and Processes. San Francisco, CA: Holden-Day, 1964. Russian original 1960.
34. B. Ya. Ryabko, Comments on 'A source matching approach to finding minimax codes', IEEE Trans. Inform. Theory, vol. IT-27, pp. 780-781, Nov. 1981. Including also the ensuing Editor's Note.
35. B. Ya. Ryabko and F. Topsøe, On asymptotically optimal methods of prediction and adaptive coding for Markov sources, submitted for publication.
36. F. Topsøe, Information theoretical optimization techniques, Kybernetika, vol. 15, pp. 8-27, 1979.
37. _, (1999) Some inequalities for information divergence and related measures of discrimination. [Online]. Available: http://melba. vu.edu.au/̃rgmia/v2n1.html
38. G. T. Toussaint, Sharper lower bounds for discrimination information in terms of variation, IEEE Trans. Inform. Theory, vol. IT-21, pp. 99-100, Jan. 1975.
39. I. Vajda, Note on discrimination information and variation, IEEE Trans. Inform. Theory, vol. IT-16, pp. 771-773, Nov. 1970.
40. I. Vincze, On the concept and measure of information contained in an observation, in Contributions to Probability, J. Gani and V K. Rohatgi, Eds. New York: Academic, 1981, pp. 207-214.
41. A. K. C. Wong and M. You, Entropy and distance of random graphs with application to structural pattern recognition, IEEE Trans. Pattern Anal. Machine Intell, vol. PAMI-7, pp. 599-609, 1985.
42. Y. Yang and A. Barron, Information-theoretic determination of minimax rates of convergence, submitted for publication.