Probability estimation in the rare-events regime

IEEE Transactions on Information Theory

Volumn 57, Issue 6, 2011, Pages 3207-3229

  Wagner, Aaron B ^a   Viswanath, Pramod ^b   Kulkarni, Sanjeev R ^c  

^a Cornell University   (United States)

^b University of Illinois at Urbana Champaign   (United States)

^c Princeton University   (United States)

Author keywords

Classification; entropy estimation; large alphabets; large number of rare events (LNRE); natural language; probability estimation

Indexed keywords

CLASSIFICATION; ENTROPY ESTIMATION; LARGE ALPHABETS; LARGE NUMBER OF RARE EVENTS (LNRE); NATURAL LANGUAGE; PROBABILITY ESTIMATION;

MAXIMUM LIKELIHOOD ESTIMATION; PROBABILITY DISTRIBUTIONS;

ESTIMATION;

EID: 79957659601     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2011.2137210     Document Type: Article

References (50)

1. R. W. Lucky, Silicon Dreams: Information, Man, and Machine. New York: St. Martin's Press, 1989.
2. T. M. Cover and R. C. King, "A convergent gambling estimate of the entropy of English," IEEE Trans. Inf. Theory, vol. IT-24, no. 4, pp. 413-421, Jul. 1978.
3. C. E. Shannon, "Prediction and entropy of printed English," Bell Syst. Tech. J., vol. 30, pp. 50-64, Jan. 1951.
4. R. H. Baayen,Word Frequency Distributions. Dordrecht, The Netherlands: Kluwer, 2001.
5. G. K. Zipf, Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. New York: Hafner, 1965.
6. K. W. Church and W. A. Gale, "A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams," Comput. Speech Lang., vol. 5, no. 1, pp. 19-54, Jan. 1991.
7. W. A. Gale and G. Sampson, "Good-Turing frequency estimation without tears," J. Quant. Ling., vol. 2, pp. 217-237, 1995.
8. S. Naranan and V. Balasubrahmanyan, "Models for power law relations in linguistics and information science," J. Quant. Ling., vol. 5, pp. 35-61, 1998.
9. I. J. Good, "The population frequencies of species and the estimation of population parameters," Biometrika, vol. 40, no. 3/4, pp. 237-264, 1953.
10. E. V. Khmaladze, The Statistical Analysis of a Large Number of Rare Events, Centrum Wiskunde & Informatica (CWI), Tech. Rep. MS-R8804, 1988.
11. E. V. Khmaladze and R. Ya Chitashvili, "Statistical analysis of a large number of rare events and related problems," in Proc. A. Razmadze Math. Inst., 1989, vol. 92, pp. 196-245, Russian.
12. C. A. J. Klaassen and R. M. Mnatsakanov, "Consistent estimation of the structural distribution function," Scand. J. Statist., vol. 27, pp. 733-746, 2000.
13. G. I. Ivchenko and Y. I. Medvedev, "Decomposable statistics and hypothesis testing. The case of small samples.," Theor. Probab. Appl., vol. 23, no. 4, pp. 764-775, 1978.
14. T. Batu, S. Dasgupta, R. Kumar, and R. Rubinfeld, "The complexity of approximating the entropy," SIAM J. Comput., vol. 35, no. 1, pp. 132-150, 2005.
15. G. A. Miller, "Note on the bias of entropy estimates," in Information Theory in Psychology II-B, H. Quastler, Ed. Glencoe, IL: Free Press, 1955, pp. 95-100.
16. P. Grassberger, "Finite sample corrections to entropy and dimension estimates," Phys. Lett. A, vol. 128, no. 6-7, pp. 369-373, 1988.
17. P. Grassberger, Entropy Estimates From Insufficient Samplings [Online]. Available: arXiv:physics/0307138
18. T. Schürmann, "Bias analysis in entropy estimation," J. Phys. A: Math. Gen., vol. 37, pp. L295-L301, 2004. (Pubitemid 38904275)
19. J. A. Bonachela, H. Hinrichson, and M. A. Mũoz, "Entropy estimates of small data sets," J. Phys. A: Math. Theor., vol. 41, 2008, 202001.
20. S.-K. Ma, "Calculation of entropy from data of motion," J. Stat. Phys., vol. 26, no. 2, pp. 221-240, Oct. 1981.
21. L. Paninski, "Estimation of entropy and mutual information," Neural Comput., vol. 15, no. 6, pp. 1191-1253, June 2003. (Pubitemid 37049793)
22. S. P. Strong, R. Koberle, R. R. de Ruyter van Steveninck, and W. Bialek, "Entropy and information in neural spike trains," Phys. Rev. Lett., vol. 80, no. 1, pp. 197-200, 1998. (Pubitemid 128621483)
23. A. Antos and I. Kontoyiannis, "Convergence properties of functional estimates for discrete distributions," Random Structures Alg., vol. 19, no. 3-4, pp. 163-193, 2001.
24. H. Cai, S. R. Kulkarni, and S. Verdú, "Universal entropy estimation via block sorting," IEEE Trans. Inf. Theory, vol. 50, no. 7, pp. 1551-1561, Jul. 2004.
25. H. Cai, S. R. Kulkarni, and S. Verdú, "Universal divergence estimation for finite-alphabet sources," IEEE Trans. Inf. Theory, vol. 52, no. 8, pp. 3456-3475, Aug. 2006. (Pubitemid 44145110)
26. L. Paninski, "Estimating entropy on m bins given fewer than samples," IEEE Trans. Inf. Theory, vol. 50, no. 9, pp. 2200-2203, Sep. 2004.
27. A. Orlitsky, N. P. Santhanam, and J. Zhang, "Always Good Turing: Asymptotically optimal probability estimation," Science, vol. 302, pp. 427-431, Oct. 2003. (Pubitemid 37296264)
28. N. L. Johnson and S. Kotz, Urn Models and Their Application: An Approach to Modern Discrete Probability Theory. New York:Wiley, 1977.
29. V. F. Kolchin, B. A. Sevast'yanov, and V. P. Chistyakov, Random Allocations. Washington, DC: V. H. Winston & Sons, 1978.
30. F. N. David and D. E. Barton, Combinatorial Chance. New York: Hafner, 1962.
31. P. Dupuis, C. Nuzman, and P. Whiting, "Large deviation asymptotics for occupancy problems," Ann. Probab., vol. 32, no. 3B, pp. 2765-2818, 2004. (Pubitemid 39258787)
32. H. E. Robbins, "Estimating the total probability of the unobserved outcomes of an experiment," Ann. Math. Statist., vol. 39, no. 1, pp. 256-7, 1968.
33. B. H. Juang and S. H. Lo, "On the bias of the Turing-Good estimate of probabilities," IEEE Trans. Signal Process., vol. 42, no. 2, pp. 496-8, Feb. 1994.
34. W. W. Esty, "The efficiency of Good's nonparametric coverage estimator," Ann. Statist., vol. 14, no. 3, pp. 1257-1260, 1986.
35. C. X. Mao and B. G. Lindsay, "A Poisson model for the coverage problem with a genomic application," Biometrika, vol. 89, no. 3, pp. 669-681, 2002. (Pubitemid 41311973)
36. D. McAllester and R. E. Schapire, "On the convergence rate of Good-Turing estimators," in Proc. 13th Annu. Conf. Comput. Learning Theory, San Francisco, CA, 2000, pp. 1-6, Morgan Kaufmann.
37. E. Drukh and Y. Mansour, "Concentration bounds for unigram language models," J. Mach. Learn. Res., vol. 6, pp. 1231-1264, 2005.
38. D. McAllester and L. Ortiz, "Concentration inequalities for the missing mass and for histogram rule error," J. Mach. Learn. Res., vol. 4, pp. 895-911, 2003.
39. C. Budianu, S. Ben-David, and L. Tong, "Estimation of the number of operating sensors in large-scale sensor networks with mobile access," IEEE Trans. Signal Process., vol. 54, no. 5, pp. 1703-1715, May 2006.
40. B. Efron and R. Thisted, "Estimating the number of unseen species: How many words did Shakespeare know?," Biometrika, vol. 63, no. 3, pp. 435-47, 1976.
41. A. B. Wagner, P. Viswanath, and S. R. Kulkarni, "Strong consistency of the Good-Turing estimator," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jul. 2006, pp. 2526-2530. (Pubitemid 351244558)
42. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. Hoboken, NJ: Wiley, 2006.
43. M. Gutman, "Asymptotically optimal classification for multiple tests with empirically observed statistics," IEEE Trans. Inf. Theory, vol. 35, no. 2, pp. 401-408, Mar. 1989.
44. J. Ziv, "On classification with empirically observed statistics and universal data compression," IEEE Trans. Inf. Theory, vol. 34, no. 2, pp. 278-286, Mar. 1988. (Pubitemid 18639980)
45. O. Kallenberg, Foundations of Modern Probability, 2nd ed. New York: Springer-Verlag, 2002.
46. C. McDiarmid, "On the method of bounded differences," in Surveys in Combinatorics, ser. London Math. Soc. Lecture Note Ser., J. Siemons, Ed. Cambridge, U.K,: Cambridge Univ. Press, 1989, vol. 141, pp. 148-188.
47. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed. New York: Springer-Verlag, 1998.
48. P. Billingsley, Probability and Measure, 3rd ed. Hoboken, NJ:Wiley, 1995.
49. R. Durrett, Probability: Theory and Examples, 2nd ed. Belmont, CA: Duxbury, 1996.
50. A. Orlitsky, N. P. Santhanam, and J. Zhang, "Universal compression of memoryless sources over unknown alphabets," IEEE Trans. Inf. Theory, vol. 50, no. 7, pp. 1469-1481, Jul. 2004.