The complexity of approximating the entropy

SIAM Journal on Computing

Volumn 35, Issue 1, 2006, Pages 132-150

  Batu, Tuǧkan ^a   Dasgupta, Sanjoy ^b   Kumar, Ravi ^c   Rubinfeld, Ronitt ^d  

^a SIMON FRASER UNIVERSITY   (Canada)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c IBM ALMADEN RESEARCH CENTER   (United States)

^d MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Entropy estimation; Properties of distributions; Property testing; Sublinear algorithms

Indexed keywords

ENTROPY ESTIMATION; PROPERTIES OF DISTRIBUTIONS; PROPERTY TESTING; SUBLINEAR ALGORITHMS;

ALGORITHMS; APPROXIMATION THEORY; COMPUTATIONAL COMPLEXITY; MATHEMATICAL MODELS; PROBABILITY;

ENTROPY;

EID: 33644591301     PISSN: 00975397     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0097539702403645     Document Type: Article

References (16)

1. A. ANTOS AND I. KONTOYIANNIS, Estimating the entropy of discrete distributions, in Proceedings of the IEEE International Symposium on Information Theory (ISIT). IEEE, Los Alamitos, CA, 2001, p. 45.
2. T. BATU, E. FISCHER, L. FORTNOW, R. KUMAR, R. RUBINFELD, AND P. WHITE, Testing random variables for independence and identity, in Proceedings of the 42nd Annual Symposium on Foundations of Computer Science, IEEE, Los Alamitos, CA, 2001, pp. 442-451.
3. T. BATU, L. FORTNOW, R. RUBINFELD, W. D. SMITH, AND P. WHITE, Testing that distributions are close, in Proceedings of the 41st Annual Symposium on Foundations of Computer Science, Redondo Beach, CA, IEEE, Los Alamitos, CA, 2000, pp. 259-269.
4. H. CAI, S. R. KULKARNI, AND S. VERDÚ Universal entropy estimation via block sorting, IEEE Trans. Inform. Theory, 50 (2004), pp. 1551-1561.
5. O. GOLDREICH AND D. RON, On Testing Expansion in Bounded-Degree Graphs, Tech. report TROO-020, Electronic Colloquium on Computational Complexity (ECCC), http://eccc.unitrier.de/eccc-reports/2000/TR00-020/index.html (2000).
6. O. GOLDREICH AND S. VADHAN, Comparing entropies in statistical zero knowledge with applications to the structure of SZK, in Proceedings of the 14th Annual IEEE Conference on Computational Complexity (CCC-99), May 4-6, 1999, IEEE, Los Alamitos, CA, pp. 54-73.
7. P. GRASSBERGER, Entropy estimates from insufficient samplings, E-print Physics/0307138, http://arxiv.org/PS.cache/physics/pdf/0307/0307138.pdf (2003).
8. B. HARRIS, The statistical estimation of entropy in the non-parametric case, in Topics in Information Theory, Colloq. Math. Soc., János Bolyai, 16, North-Holland, Amsterdam, 1977, pp. 323-355.
9. M. KEARNS, Y. MANSOUR, D. RON, R. RUBINFELD, R. E. SCHAPIRE, AND L. SELLIE, On the learnability of discrete distributions, in Proceedings of the 26th Annual Symposium on the Theory of Computing, ACM, New York, 1994, pp. 273-282.
10. S-K. MA, Calculation of entropy from data of motion, J. Statist. Phys., 26 (1981), pp. 221-240.
11. L. PANINSKI, Estimation of entropy and mutual information, Neural Computation, 15 (2003), pp. 1191-1253.
12. L. PANINSKI, Estimating entropy on m bins given fewer than m samples, IEEE Trans. Inform. Theory, 50 (2004), pp. 2200-2203.
13. D. RON, Private communication, 2004.
14. S. P. STRONG, R. KOBERLE, R. R. DE RUYTER VAN STEVENINCK, AND W. BIALEK, Entropy and information in neural spike trains, Phys. Rev. Lett., 80 (1998), pp. 197-200.
15. D. WOLPERT AND D. R. WOLF, Estimating functions of probability distributions from a finite set of samples. Part I. Bayes estimators and the Shannon entropy, Phys. Rev. E, 52 (1995), pp. 6841-6854.
16. A. J. WYNER AND D. FOSTER, On the lower limits of entropy estimation, IEEE Trans. Inform. Theory, submitted.