Estimating the unseen: An n/log(n)-sample estimator for entropy and support size, shown optimal via new CLTs

Proceedings of the Annual ACM Symposium on Theory of Computing

Volumn , Issue , 2011, Pages 685-694

  Valiant, Gregory ^a   Valiant, Paul ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

central limit theorem; distinct elements problem; entropy; lower bounds; property estimation; Stein's method; support size

Indexed keywords

ADDITIVES; ENTROPY; IMAGE CODING;

CENTRAL LIMIT THEOREM; DISTINCT ELEMENTS; LOWER BOUNDS; PROPERTY ESTIMATION; STEIN'S METHOD;

COMPUTATION THEORY;

EID: 79959740921     PISSN: 07378017     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1993636.1993727     Document Type: Conference Paper

References (44)

1. J. Acharya, A. Orlitsky, and S. Pan. The maximum likelihood probability of unique-singleton, ternary, and length-7 patterns. In IEEE Symp. on Information Theory, 2009.
2. N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. J. Comput. System Sci., 58:137-147, 1999.
3. Z. Bar-Yossef, T. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting distinct elements in a data stream. In Proc. 6th Workshop on Rand. and Approx. Techniques.
4. Z. Bar-Yossef, R. Kumar, and D. Sivakumar. Sampling algorithms: Lower bounds and applications. In STOC, 2001.
5. A. Barbour and L. Chen. An Introduction to Stein's Method. Singapore University Press, 2005.
6. T. Batu. Testing properties of distributions. Ph.D. thesis, Cornell University, 2001.
7. T. Batu, S. Dasgupta, R. Kumar, and R. Rubinfeld. The complexity of approximating the entropy. In STOC, 2002.
8. T. Batu, S. Dasgupta, R. Kumar, and R. Rubinfeld. The complexity of approximating the entropy. SIAM Journal on Computing, 2005.
9. T. Batu, L. Fortnow, R. Rubinfeld, W. Smith, and P. White. Testing that distributions are close. In FOCS, 2000.
10. K. Beyer, P. J. Haas, B. Reinwald, Y. Sismanis, and R. Gemulla. On synopses for distinct-value estimation under multiset operations. In ACM SIGMOD Int. Conf. on Management of Data, 2007.
11. R. Bhattacharya and S. Holmes. An exposition of Götze's estimation of the rate of convergence in the multivariate central limit theorem. Stanford Department of Statistics Technical Report, 2010-02, March 2010.
12. J. Bunge. Bibliography of references on the problem of estimating support size, available at http://www.stat.cornell.edu/~bunge/bibliography.html.
13. J. Bunge and M. Fitzpatrick. Estimating the number of species: A review. Journal of the American Statistical Association, 88(421):364-373, 1993.
14. A. Chakrabarti, G. Cormode, and A. McGregor. A near-optimal algorithm for computing the entropy of a stream. In SODA, 2007.
15. M. Charikar, S. Chaudhuri, R. Motwani, and V. Narasayya. Towards estimation error guarantees for distinct values. In PODS, 2000.
16. S. Chatterjee. Personal communication. May 2010.
17. S. Chatterjee and E. Meckes. Multivariate normal approximation using exchangeable pairs. ALEA Lat. Am. J. Probab. Math. Stat., 4:257-283, 2008.
18. C. Daskalakis and C. H. Papadimitriou. Computing equilibria in anonymous games. In FOCS, 2007.
19. C. Daskalakis and C. H. Papadimitriou. Discretized multinomial distributions and Nash equilibria in anonymous games. In FOCS, 2008.
20. R. Fisher, A. Corbet, and C. Williams. The relation between the number of species and the number of individuals in a random sample of an animal population. Journal of the British Ecological Society, 1943.
21. I. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(16):237-264, 1953.
22. F. Götze. On the rate of convergence in the multivariate CLT. Annals of Probability, 19(2):724-739, 1991.
23. S. Guha, A. McGregor, and S. Venkatasubramanian. Streaming and sublinear approximation of entropy and information distances. In SODA, 2006.
24. L. Gurvits. On Newton(like) inequalities for multivariate homogeneous polynomials. http://www.optimization-online.org/DB-FILE/2008/06/1998.pdf, 2008.
25. N. Harvey, J. Nelson, and K. Onak. Sketching and streaming entropy via approximation theory. In FOCS, 2008.
26. P. Indyk and D. Woodruff. Tight lower bounds for the distinct elements problem. In FOCS, 2003.
27. D. Kane, J. Nelson, and D. Woodruff. An optimal algorithm for the distinct elements problem. In PODS, 2010.
28. N. Karmarkar. A new polynomial time algorithm for linear programming. Combinatorica, 4(4):373-395, 1984.
29. D. A. McAllester and R. Schapire. On the convergence rate of Good-Turing estimators. In COLT, 2000.
30. A. Orlitsky, N. Santhanam, K.Viswanathan, and J. Zhang. On modeling profiles instead of values. Uncertainity in Artificial Intelligence, 2004.
31. A. Orlitsky, N. Santhanam, and J. Zhang. Always Good Turing: Asymptotically optimal probability estimation. In FOCS, 2003.
32. A. Orlitsky, N. Santhanam, and J. Zhang. Always Good Turing: Asymptotically optimal probability estimation. Science, 302(5644):427-431, October 2003. (Pubitemid 37296264)
33. L. Paninski. Estimation of entropy and mutual information. Neural Comput., 15(6):1191-1253, 2003. (Pubitemid 37049793)
34. L. Paninski. Estimating entropy on m bins given fewer than m samples. IEEE Trans. on Information Theory, 50(9):2200-2203, 2004.
35. S. Raskhodnikova, D. Ron, A. Shpilka, and A. Smith. Strong lower bounds for approximating distribution support size and the distinct elements problem. SIAM J. Comput., 39(3):813-842, 2009.
36. G. Reinert and A. Röllin. Multivariate normal approximation with Stein's method of exchangeable pairs under a general linearity condition. Annals of Probability, 37(6):2150-2173, 2009.
37. B. Roos. On the rate of multivariate Poisson convergence. Journal of Multivariate Analysis, 69:120-134, 1999.
38. C. Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. Proc. Sixth Berkeley Symp. on Mathematical Statistics and Probability, 2, 1972.
39. G. Valiant and P. Valiant. A CLT and tight lower bounds for estimating entropy. http://www.eccc.uni-trier.de/report/2010/179/, 2010.
40. G. Valiant and P. Valiant. Estimating the unseen: a sublinear-sample canonical estimator of distributions. http://www.eccc.uni-trier.de/report/2010/ 180/, 2010.
41. P. Valiant. Testing symmetric properties of distributions. In STOC, 2008.
42. A. Wagner, P. Viswanath, and S. Kulkami. Strong consistency of the Good-Turing estimator. In IEEE Symp. on Information Theory, 2006.
43. A. Wagner, P. Viswanath, and S. Kulkami. A better Good-Turing estimator for sequence probabilities. In IEEE Symp. on Information Theory, 2007.
44. D. Woodruff. The average-case complexity of counting distinct elements. In The 12th Int. Conf. on Database Theory, 2009.