Estimating the unseen: Improved estimators for entropy and other properties

Advances in Neural Information Processing Systems

Volumn , Issue , 2013, Pages

  Valiant, Gregory ^a   Valiant, Paul ^b  

^a STANFORD UNIVERSITY   (United States)

^b BROWN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ENTROPY; PROBABILITY DISTRIBUTIONS;

CONSTANT FACTORS; DATA ANALYSIS SYSTEM; DISTANCE METRICS; DISTINCT ELEMENTS; DISTRIBUTIONAL PROPERTY; NATURAL DISTRIBUTION; TOTAL PROBABILITIES; WORST-CASE INSTANCES;

ESTIMATION;

EID: 84898993766     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. G. Valiant and P. Valiant. Estimating the unseen: an n/ log(n)-sample estimator for entropy and support size, shown optimal via new CLTs. In Symposium on Theory of Computing (STOC), 2011.
2. G. Valiant and P. Valiant. The power of linear estimators. In IEEE Symposium on Foundations of Computer Science (FOCS), 2011.
3. M. R. Nelson et al. An abundance of rare functional variants in 202 drug target genes sequenced in 14,002 people. Science, 337(6090):100-104, 2012.
4. J. A. Tennessen et al. Evolution and functional impact of rare coding variation from deep sequencing of human exomes. Science, 337(6090):64-69, 2012.
5. A. Keinan and A. G. Clark. Recent explosive human population growth has resulted in an excess of rare genetic variants. Science, 336(6082):740-743, 2012.
6. F. Olken and D. Rotem. Random sampling from database files: a survey. In Proceedings of the Fifth International Workshop on Statistical and Scientific Data Management, 1990.
7. P. J. Haas, J. F. Naughton, S. Seshadri, and A. N. Swami. Selectivity and cost estimation for joins based on random sampling. Journal of Computer and System Sciences, 52(3):550-569, 1996.
8. R.A. Fisher, A. Corbet, and C.B. Williams. The relation between the number of species and the number of individuals in a random sample of an animal population. Journal of the British Ecological Society, 12(1):42-58, 1943.
9. I. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(16):237-264, 1953.
10. D. A. McAllester and R.E. Schapire. On the convergence rate of Good-Turing estimators. In Conference on Learning Theory (COLT), 2000.
11. A. Orlitsky, N.P. Santhanam, and J. Zhang. Always Good Turing: Asymptotically optimal probability estimation. Science, 302(5644):427-431, October 2003.
12. A. Orlitsky, N. Santhanam, K.Viswanathan, and J. Zhang. On modeling profiles instead of values. Uncertainity in Artificial Intelligence, 2004.
13. J. Acharya, A. Orlitsky, and S. Pan. The maximum likelihood probability of unique-singleton, ternary, and length-7 patterns. In IEEE Symp. on Information Theory, 2009.
14. J. Acharya, H. Das, A. Orlitsky, and S. Pan. Competitive closeness testing. In COLT, 2011.
15. L. Paninski. Estimation of entropy and mutual information. Neural Comp., 15(6):1191-1253, 2003.
16. J. Bunge and M. Fitzpatrick. Estimating the number of species: A review. Journal of the American Statistical Association, 88(421):364-373, 1993.
17. J. Bunge. Bibliography of references on the problem of estimating support size, available at http://www.stat.cornell.edu/~bunge/bibliography.html.
18. Z. Bar-Yossef, R. Kumar, and D. Sivakumar. Sampling algorithms: lower bounds and applications. In STOC, 2001.
19. T. Batu Testing Properties of Distributions Ph.D. thesis, Cornell, 2001.
20. M. Charikar, S. Chaudhuri, R. Motwani, and V.R. Narasayya. Towards estimation error guarantees for distinct values. In SODA, 2000.
21. T. Batu, L. Fortnow, R. Rubinfeld,W.D. Smith, and P. White. Testing that distributions are close. In IEEE Symposium on Foundations of Computer Science (FOCS), 2000.
22. V.Q. Vu, B. Yu, and R.E. Kass. Coverage-adjusted entropy estimation. Statistics in Medicine, 26(21):4039-4060, 2007.
23. G. Miller. Note on the bias of information estimates. Information Theory in Psychology II-B, ed H Quastler (Glencoe, IL: Free Press):pp 95-100, 1955.
24. S. Panzeri and A Treves. Analytical estimates of limited sampling biases in different information measures. Network: Computation in Neural Systems, 7:87-107, 1996.
25. S. Zahl. Jackknifing an index of diversity. Ecology, 58:907-913, 1977.
26. B. Efron and C. Stein. The jacknife estimate of variance. Annals of Statistics, 9:586-596, 1981.
27. A. Chao and T.J. Shen. Nonparametric estimation of shannons index of diversity when there are unseen species in sample. Environmental and Ecological Statistics, 10:429-443, 2003.
28. D.G. Horvitz and D.J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663-685, 1952.
29. P. Valiant. Testing Symmetric Properties of Distributions. SIAM J. Comput., 40(6):1927-1968,2011.