Information inequalities for joint distributions, with interpretations and applications

IEEE Transactions on Information Theory

Volumn 56, Issue 6, 2010, Pages 2699-2713

  Madiman, Mokshay ^a   Tetali, Prasad ^b  

^a YALE UNIVERSITY   (United States)

^b Georgia Institute of Technology   (United States)

Author keywords

Entropy inequality; Entropy based counting; Inequality for minors; Submodularity

Indexed keywords

ARBITRARY GRAPHS; CHAIN RULES; COMBINATORICS; ENTROPY INEQUALITY; ERROR SOURCES; INDEPENDENT SET; INEQUALITY FOR MINORS; JOINT DISTRIBUTIONS; JOINT ENTROPY; MATRIX THEORY; NEW RESULTS; NUMBER OF ZEROS; RELATIVE ENTROPY; SUBMODULAR FUNCTIONS; SUBMODULARITY; UPPER AND LOWER BOUNDS;

RANDOM VARIABLES;

ENTROPY;

EID: 77957602848     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2010.2046253     Document Type: Article

References (51)

1. N. Alon and J. H. Spencer, The Probabilistic Method, second ed. New York: Wiley , 2000.
2. S. Artstein, K. M. Ball, F. Barthe, and A. Naor, "Solution of Shannon's problem on the monotonicity of entropy," J. Amer. Math. Soc., vol.17, no.4, pp. 975-982, 2004
3. R. Bellman, Introduction to Matrix Analysis. New York: McGraw-Hill, 1960.
4. S. G. Bobkov and M. Ledoux, "On modified logarithmic Sobolev inequalities for Bernoulli and Poisson measures," J. Funct. Anal., vol.156, no.2, pp. 347-365, 1998
5. O. N. Bondareva, "Some applications of the methods of linear programming to the theory of cooperative games (in Russian)," Problemy Kibernetiki, vol.10, pp. 119-139, 1963.
6. G. Brightwell and P. Tetali, "The number of linear extensions of the boolean lattice," Order, vol.20, pp. 333-345, 2003.
7. S. Chaiken, "A combinatorial proof of the all minors matrix tree theorem," SIAM J. Algebr. Discr. Methods, vol.3, no.3, pp. 319-329, 1982
8. T. H. Chan, "Balanced information inequalities," IEEE Trans. Inf. Theory, vol.49, pp. 3261-3267, 2003.
9. F. R. K. Chung, R. L. Graham, P. Frankl, and J. B. Shearer, "Some intersection theorems for ordered sets and graphs," J. Combinator. Theory, Ser. A, vol.43, pp. 23-37, 1986
10. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
11. A. Dembo, T. M. Cover, and J. A. Thomas, "Information-theoretic inequalities," IEEE Trans. Inf. Theory, vol.37, pp. 1501-1518, 1991.
12. R. Dougherty, C. Freiling, and K. Zeger, "Networks, matroids and non-Shannon information inequalities," IEEE Trans. Inf. Theory, vol.53, no.6, pp. 1949-1969, Jun. 2007.
13. J. Edmonds, "Submodular functions, matroids and certain polyhedra," in Proc. Int. Conf. Combinator. Struct. Appl., 1970.
14. U. Feige, "On maximizing welfare when utility functions are subadditive," presented at the 38th ACMSymp. Theory Comput., Seattle,WA, May 2006.
15. E. Friedgut, "Hypergraphs, entropy, and inequalities," Amer. Math. Month., vol.111, no.9, pp. 749-760, Nov. 2004.
16. E. Friedgut and J. Kahn, "On the number of copies of one hypergraph in another," Israel J. Math., vol.105, pp. 251-256, 1998.
17. S. Fujishige, "Polymatroidal dependence structure of a set of random variables," Inf. Contr., vol.39, pp. 55-72, 1978.
18. S. Fujishige, Submodular Functions and Optimization, Volume 58 of Annals of Discrete Mathematics, 2nd ed. Amsterdam, The Netherlands: Elsevier, 2005.
19. Z. Füredi, "Scrambling permutations and entropy of hypergraphs," Random Structures Algorithms, vol.8, no.2, pp. 97-104, 1996.
20. D. Galvin, Pers. commun., 2006.
21. D. Galvin and P. Tetali, "On weighted graph homomorphisms," DIMACS-AMS Special Volume, vol.63, pp. 13-28, 2004.
22. L. Gross, "Logarithmic Sobolev inequalities," Amer. J. Math., vol.97, no.4, pp. 1061-1083, 1975.
23. T. S. Han, "Nonnegative entropy measures of multivariate symmetric correlations," Inf. Contr., vol.36, no.2, pp. 133-156, 1978.
24. B. Ibinson, N. Linden, and A. Winter, "All inequalities for the relative entropy," in Proc. IEEE Intl. Symp. Inform. Theory, Seattle, 2006, pp. 237-241.
25. J. Kahn, "An entropy approach to the hard-core model on bipartite graphs," Combinator., Probabil. Comput., vol.10, pp. 219-237, 2001. (Pubitemid 33692953)
26. J. Kahn, "Entropy, independent sets and antichains: A new approach to Dedekind's problem," Proc. Amer. Math. Soc., vol.130, no.2, pp. 371-378, 2001
27. J. Kahn, Pers. Commun., 2006.
28. J. L. Kelley, "Measures on Boolean algebras," Pacific J. Math., vol.9, pp. 1165-1177, 1959.
29. G. Kirchhoff, "Über die auflösung der gleichungen, auf welche man bei der untersuchung der linearen verteilung galvanischer ströme geführt wird," Ann. Phys. Chem., vol.72, pp. 497-508.
30. I. Kontoyiannis and M. Madiman, "Measure concentration for compound Poisson distributions," Elect. Commun. Probab., vol.11, pp. 45-57, 2006.
31. B. Lehmann, D. Lehmann, and N. Nisan, "Combinatorial auctions with decreasing marginal utilities," in Proc. 3rd ACM Conf. Electron. Commerce, Tampa, FL, 2001, pp. 18-28.
32. M. Lewin, "A generalization of the matrix-tree theorem," Math. Z., vol.181, no.1, pp. 55-70, 1982.
33. M. Madiman and A. R. Barron, "Generalized entropy power inequalities and monotonicity properties of information," IEEE Trans. Inf. Theory, vol.53, pp. 2317-2329, Jul. 2007.
34. M. Madiman, A. Marcus, and P. Tetali, Entropy and Set Cardinality Inequalities for Partition-Determined Functions, With Applications to Sumsets, 2008 [Online]. Available: http://arxiv.org/abs/0901.0055
35. M. Madiman and P. Tetali, "Sandwich bounds for joint entropy," presented at the IEEE Int. Symp. Inf. Theory Nice, France, Jun. 2007.
36. M. Madiman, "Cores of cooperative games in information theory," EURASIP J. Wireless Commun. Netw., no.318704, 2008.
37. M. Madiman, Determinant and Trace Inequalities for Sums of Positive-Definite Matrices 2008, unpublished manuscript.
38. P. Massart, "Some applications of concentration inequalities to statistics," Ann. Facultédes Sci. Toulouse, vol.IX, no.2, pp. 245-303, 2000.
39. F. Matúš, "Two constructions on limits of entropy functions," IEEE Trans. Inf. Theory, vol.53, no.1, pp. 320-330, Jan. 2007.
40. J. M. Ollagnier and D. Pinchon, "Filtre moyennant et valeurs moyennes des capacités invariantes," Bull. Soc. Math. France, vol.110, no.3, pp. 259-277, 1982
41. H. Narayanan, "Submodular functions and electrical networks," in Annals of Discrete Mathematics. Amsterdam, The Netherlands: North-Holland, 1997, vol.54.
42. J. Nayak, E. Tuncel, and K. Rose, "Zero-error source-channel codingwith side information," IEEE Trans. Inf. Theory, vol.52, pp. 4626-4629, 2006.
43. J. Radhakrishnan, "Entropy and counting," in Computational Mathematics, Modelling and Algorithms, J. C.Misra, Ed. New Delhi, India: Narosa, 2003.
44. E. R. Scheinerman and D. H. Ullman, Fractional Graph Theory. New York: Wiley, 1997.
45. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J., vol.27, pp. 379-423, 1948.
46. L. S. Shapley, "On balanced sets and cores," Naval Res. Logist. Quarter., vol.14, pp. 453-560, 1967.
47. L. S. Shapley, "Cores of convex games," Int. J. Game Theory, vol.1, no.1, pp. 11-26, 1971.
48. A. J. Stam, "Some inequalities satisfied by the quantities of information of Fisher and Shannon," Inf. Contr., vol.2, pp. 101-112, 1959.
49. A. C. D. van Enter, R. Fernández, and A. D. Sokal, "Regularity properties and pathologies of position-space renormalization-group transformations: Scope and limitations of Gibbsian theory," J. Statist. Phys., vol.72, no.5-6, pp. 879-1167, 1993.
50. S. Verdú and T.Weissman, "Erasure entropy," in Proc. IEEE Int. Symp. Inf. Theory, Seattle, WA, 2006.
51. J. Zhang and R. W. Yeung, "On characterization of entropy function via information inequalities," IEEE Trans. Inf. Theory, vol.44, pp. 1440-1452, 1998.