Monotonic convergence in an information-theoretic law of small numbers

IEEE Transactions on Information Theory

Volumn 55, Issue 12, 2009, Pages 5412-5422

  Yu, Yaming ^a  

^a 5226 Donald Bren Hall   (United States)

Author keywords

Binomial thinning; Convex order; Logarithmic Sobolev inequality; Majorization; Maximum entropy; Poisson approximation; Relative entropy; Schur concavity; Ultra log concavity

Indexed keywords

BINOMIAL THINNING; CONVEX ORDER; LOGARITHMIC SOBOLEV INEQUALITY; MAJORIZATION; MAXIMUM ENTROPY; POISSON APPROXIMATIONS; RELATIVE ENTROPY; SCHUR-CONCAVITY; ULTRA-LOG-CONCAVITY;

INFORMATION THEORY; OPTIMIZATION; POISSON DISTRIBUTION;

ENTROPY;

EID: 77951599486     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2009.2032727     Document Type: Article

References (42)

1. S. Artstein, K. M. Ball, F. Barthe, and A. Naor, "On the rate of convergence in the entropic central limit theorem," Prob. Theory Rel. Fields, vol.129, no.3, pp. 381-390, 2004.
2. S. Artstein, K. M. Ball, F. Barthe, and A. Naor, "Solution of Shannon's problem on the monotonicity of entropy," J. Amer. Math. Soc., vol.17, no.4, pp. 975-982, 2004.
3. A. D. Barbour, L. Holst, and S. Janson, Poisson Approximation, Oxford Studies in Probability. Oxford, U.K.: Clarendon, 1992, vol.2.
4. A. R. Barron, "Entropy and the central limit theorem," Ann. Probab., vol.14, pp. 336-342, 1986.
5. N. M. Blachman, "The convolution inequality for entropy powers," IEEE Trans. Inf. Theory, vol.11, pp. 267-271, 1965.
6. S. Bobkov and M. Ledoux, "On modified logarithmic Sobolev inequalities for Bernoulli and Poisson measures," J. Funct. Anal., vol.156, no.2, pp. 347-365, 1998.
7. F. Brenti, "Unimodal, log-concave, and Pólya frequency sequences in combinatorics," Mem. Amer. Math. Soc., vol.81, no.413, 1989.
8. D. Chafaï, "Entropies, convexity, and functional inequalities: On φ-entropies and φ-Sobolev inequalities," J. Math. Kyoto Univ., vol.44, no.2, pp. 325-363, 2004.
9. D. Chafaï, "Binomial-Poisson entropic inequalities and the M/M/∞ queue," ESAIM Probab. Statist., vol.10, pp. 317-339, 2006.
10. T. Cover and J. Thomas, Elements of Information Theory, 2nd ed. New York: Wiley, 2006.
11. H. Davenport and G. Pólya, "On the product of two power series," Canad. J. Math., vol.1, pp. 1-5, 1949.
12. W. Feller, An Introduction to Probability Theory and Its Applications. New York: Wiley, 1966, vol.2.
13. G. H. Hardy, J. E. Littlewood, and G. Polya, Inequalities. Cambridge, U.K: Cambridge Univ. Press, 1964.
14. P. Harremoës, "Binomial and Poisson distributions as maximum entropy distributions," IEEE Trans. Inf. Theory, vol.47, pp. 2039-2041, Jul. 2001.
15. P. Harremoës, O. Johnson, and I. Kontoyiannis, "Thinning and the law of small numbers," in Proc. IEEE Int. Symp. Information Theory, Nice, France, Jun. 2007.
16. P. Harremoës, O. Johnson, and I. Kontoyiannis, "Thinning and information projections," in Proc. IEEE Int. Symp. Information Theory, Toronto, ON, Canada, Jul. 2008.
17. P. Harremoës, O. Johnson, and I. Kontoyiannis, Thinning, entropy and the law of thin numbers 2009 [Online]. Available: http://arxiv.org/abs/0906. 0690
18. P. Harremoës and P. S. Ruzankin, "Rate of convergence to Poisson law in terms of information divergence," IEEE Trans. Inf. Theory, vol.50, pp. 2145-2149, 2004.
19. W. Hoeffding, "On the distribution of the number of successes in independent trials," Ann. Math. Statist., vol.27, pp. 713-721, 1956.
20. O. Johnson, "Log-concavity and the maximum entropy property of the Poisson distribution," Stoch. Processes Applicat., vol.117, no.6, pp. 791-802, 2007.
21. O. Johnson and A. R. Barron, "Fisher information inequalities and the central limit theorem," Prob. Theory Rel. Fields, vol.129, no.3, pp. 391-409, 2004.
22. O. Johnson, I. Kontoyiannis, and M. Madiman, On the Entropy and Log-Concavity of Compound Poisson Measures 2008 [Online]. Available: http://arxiv.org/abs/0805.4112v1
23. S. Karlin and Y. Rinott, "Entropy inequalities for classes of probability distributions I. The univariate case," Adv. Appl. Prob., vol.13, pp. 93-112, 1981.
24. I. Kontoyiannis, P. Harremoës, and O. T. Johnson, "Entropy and the law of small numbers," IEEE Trans. Inf. Theory, vol.51, pp. 466-472, Feb. 2005.
25. E. H. Lieb, "Proof of an entropy conjecture of Wehrl," Commun. Math. Phys., vol.62, no.1, pp. 35-41, Aug. 1978.
26. T. M. Liggett, "Ultra logconcave sequences and negative dependence," J. Combin. Theory Ser. A, vol.79, no.2, pp. 315-325, 1997.
27. M. Madiman and A. Barron, "Generalized entropy power inequalities and monotonicity properties of information," IEEE Trans. Inf. Theory, vol.53, pp. 2317-2329, Jul. 2007.
28. M. Madiman, O. Johnson, and I. Kontoyiannis, "Fisher information, compound Poisson approximation and the Poisson channel," in Proc. IEEE Int. Symp. Information Theory, Nice, France, Jun. 2007.
29. A. W. Marshall and I. Olkin, Inequalities: Theory of Majorization and Its Applications. New York: Academic, 1979.
30. A. Rényi, "A characterization of Poisson processes," Magyar Tud. Akad. Mat. Kutató Int. Közl., vol.1, pp. 519-527, 1956.
31. M. Shaked and J. G. Shanthikumar, Stochastic Orders. New York: Springer, 2007.
32. D. Shlyakhtenko, "A free analogue of Shannon's problem on monotonicity of entropy," Adv. Math., vol.208, no.2, pp. 824-833, Jan. 2007. (Pubitemid 44688709)
33. D. Shlyakhtenko, "Shannon's monotonicity problem for free and classical entropy," Proc. Nat. Acad. Sci. USA, vol.104, no.39, pp. 15254-15258, Sep. 2007. (Pubitemid 47502905)
34. A. J. Stam, "Some inequalities satisfied by the quantities of information of Fisher and Shannon," Inf. Contr., vol.2, no.2, pp. 101-112, Jun. 1959.
35. A. M. Tulino and S. Verdú, "Monotonic decrease of the non-Gaussianness of the sum of independent random variables: A simple proof," IEEE Trans. Inf. Theory, vol.52, pp. 4295-4297, Sep. 2006.
36. L. Wu, "A new modified logarithmic Sobolev inequality for Poisson point processes and several applications," Probab. Theory Rel. Fields, vol.118, no.3, pp. 427-438, 2000.
37. Y. Yu, "On the maximum entropy properties of the binomial distribution," IEEE Trans. Inf. Theory, vol.54, pp. 3351-3353, Jul. 2008.
38. Y. Yu, "On an inequality of Karlin and Rinott concerning weighted sums of i.i.d. random variables," Adv. Appl. Prob., vol.40, no.4, pp. 1223-1226, 2008.
39. Y. Yu, "On the entropy of compound distributions on nonnegative integers," IEEE Trans. Inf. Theory, vol.55, pp. 3645-3650, Aug. 2009.
40. Y. Yu, "Complete monotonicity of the entropy in the central limit theorem for gamma and inverse Gaussian distributions," Stat. Prob. Lett., vol.79, pp. 270-274, 2009.
41. Y. Yu, "Relative log-concavity and a pair of triangle inequalities," Bernoulli J., to be published.
42. Y. Yu and O. Johnson, "Concavity of entropy under thinning," in Proc. IEEE Int. Symp. Information Theory, Seoul, Korea, Jul. 2009.