Divergence Measures Based on the Shannon Entropy

IEEE Transactions on Information Theory

Volumn 37, Issue 1, 1991, Pages 145-151

  Lin, Jianhua ^a  

^a BRANDEIS UNIVERSITY   (United States)

Author keywords

discrimination in; dissimilarity measure; Divergence; entropy; formation; probability of error bounds

Indexed keywords

PROBABILITY;

DIVERGENCE MEASURES; SHANNON ENTROPY;

INFORMATION THEORY;

EID: 0025952277     PISSN: 00189448     EISSN: 15579654     Source Type: Journal    
DOI: 10.1109/18.61115     Document Type: Article

References (32)

1. J. Aczel and Z. Daroczy, On Measures of Information and Their Characterizations. New York: Academic, 1975.
2. S. M. Ali and S. D. Silvey, “A general class of coefficients of divergence of one distribution from another,” J. Roy. Statist. Soc., Ser. B, vol. 28, pp. 131–142, 1966.
3. M. Ben Bassat, “f-entropies, probability of error, and feature selection,” Inform. Contr., vol. 39, pp. 227–242, 1978.
4. C. H. Chen, Statistical Pattern Recognition. Rochelle Park, NJ: Hayden Book Co., 1973, Ch. 4.
5. C. H. Chen, On information and distance measures, error bounds, and feature selection,” Inform. Sci., vol. 10, pp. 159–173, 1976.
6. C. K. Chow and C. N. Liu, “Approximating discrete probability distributions with dependence trees,” IEEE Trans. Inform. Theory, vol. IT-14, no. 3, pp. 462–467, May 1968.
7. I. Csiszár, “Information-type measures of difference of probability distributions and indirect observations,” Studia Sci. Math. Hungar., vol. 2, pp. 299–318, 1967.
8. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
9. S. Guiasu, Information Theory with Applications. New York: Mc-Graw-Hill, 1977.
10. D. V. Gokhale and S. Kullback, Information in Contingency Tables. New York: Marcel Dekker, 1978.
11. M. E. Heilman and J. Raviv, “Probability of error, equivocation, and the Chernoff bound,” IEEE Trans. Inform. Theory, vol. IT-16, no. 4, pp. 368–372, July 1970.
12. R. W. Johnson, “Axiomatic characterization of the directed divergences and their linear combinations,” IEEE Trans. Inform. Theory, vol. IT-25, no. 6, pp. 709–716, Nov. 1979.
13. T. T. Kadota and L. A. Shepp, On the best finite set of linear observables for discriminating two gaussians signals,” IEEE Trans. Inform. Theory, vol. IT-13, no. 2, pp. 278–284, Apr. 1967.
14. T. Kailath, “The divergence and Bhattacharyya distance measures in signal selection,” IEEE Transactions Commun. Technol., vol. COM-15, no. 1, 52–60, Feb. 1967.
15. J. N. Kapur, “A comparative assessment of various measures of directed divergence,” Advances Manag. Stud., vol. 3, no. 1, pp. 1–16, Jan. 1984.
16. D. Kazakos and T. Cotsidas, “A decision theory approach to the approximation of discrete probability densities,” IEEE Trans. Pattern Anal. Machine Intell, vol. PAMI-2, vol. 1, pp. 61–67, Jan. 1980.
17. S. Kullback, Information Theory and Statistics. New York: Dover Publications, 1968.
18. S. Kullback, A lower bound for discrimination information in terms of variation,” IEEE Trans. Inform. Theory, vol. IT-13, pp. 126–127, Jan. 1967.
19. S. Kullback and R. A. Leibler, “On information and sufficiency,” Ann. Math. Statist., vol. 22, pp. 79–86, 1951.
20. U. Kumar, V. Kumar, and J. N. Kapur, “Some normalized measures of directed divergence,” Int. J. Gen. Syst., vol. 13, pp. 5–16, 1986.
21. J. Lin and S. K. M. Wong, “Approximation of discrete probability distributions based on a new divergence measure,” Congressus Numerantium, vol. 61, 75–80, 1988.
22. H. Jeffreys, “An invariant form for the prior probability in estimation problems,” Proc. Roy. Soc. Lon., Ser. A, vol. 186, 1946, pp. 453–461.
23. C. R. Rao and T. K. Nayak, “Cross entropy, dissimilarity measures, and characterizations of quadratic entropy,” IEEE Trans. Inform. Theory, vol. IT-31, no. 5, pp. 589–593, Sept. 1985.
24. C. R. Rao, “Diversity and dissimilarity coefficients: A unified approach,” Theoretical Population Biol., vol. 21, pp. 24–43, 1982.
25. C. R. Rao, Diversity: Its measurement, decomposition, apportionment and analysis,” Sankhya: Indian J. Statist., Ser. A, vol. 44, pt. 1. pp. 1–22, Feb. 1982.
26. G. T. Toussaint, “On some measures of information and their application to pattern recognition,” in Proc. Conf. Measures of Information and Their Applications, Indian Inst. Technol., Bombay, Aug. 1974.
27. G. T. Toussaint, Sharper lower bounds for discrimination information in terms of variation,” IEEE Trans. Inform. Theory, vol. IT-21, no. 1, pp. 99–100, Jan. 1975.
28. I. Vajda, “Note on discrimination information and variation,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 771–773, Nov. 1970.
29. I. Vajda, “On the f-divergence and singularity of probability measures,” Periodica Mathem. Himgarica, vol. 2, pp. 223–234, 1972.
30. I. Vajda, Theory of Statistical Inference and Information. Dordrecht-Boston: Kluwer, 1989.
31. J. W. Van Ness, “Dimensionality and the classification performance with independent coordinates,” IEEE Trans. Syst. Man Cybern., vol. SMC-7, pp. 560–564, July 1977.
32. A. K. C. Wong and M. You, “Entropy and distance of random graphs with application to structural pattern recognition,” IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-7, no. 5, pp. 599–609, Sept. 1985.