Always Good-Turing: Asymptotically optimal probability estimation

Annual Symposium on Foundations of Computer Science - Proceedings

Volumn , Issue , 2003, Pages 179-188

  Orlitsky, Alon ^a,b   Santhanam, Narayana P ^a   Zhang, Junan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ESTIMATION; INFORMATION THEORY; LEARNING SYSTEMS; OPTIMIZATION; PROBABILITY DISTRIBUTIONS; ARTIFICIAL INTELLIGENCE; COMPUTER SCIENCE; MILITARY OPERATIONS; PROBABILITY;

GOOD-TURING ESTIMATOR; OPTIMAL PROBABILITY ESTIMATION;

COMPUTATION THEORY; PROBABILITY DISTRIBUTIONS;

ASYMPTOTICALLY OPTIMAL; GOOD-TURING; LONG SEQUENCES; PER-SYMBOL; PROBABILITY ESTIMATION; PROBABILITY ESTIMATOR; WORLD WAR II;

EID: 0345412700     PISSN: 02725428     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (33)

1. J. Åberg, Y. Shtarkov, and B. Smeets. Multialphabet coding with separate alphabet description. In Proceedings of compression and complexity of sequences, 1997.
2. N. Cesa-Bianchi and G. Lugosi. Minimax regret under log loss for general classes of experts. In Proceedings of the Twelfth annual conference on computational learning theory, pages 12-18, 1999.
3. S. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. In Proceedings of the Thirty-Fourth Annual Meeting of the Association for Computational Linguistics, pages 310-318, San Francisco, 1996. Morgan Kaufmann Publishers.
4. K. Church and W. Gale. Probability scoring for spelling correction. Statistics and Computing, 1:93-103, 1991.
5. B. Clarke and A. Barron. Jeffreys' prior is asymptotically least favorable under entropy risk. Journal of Statistical planning and Inference, 41(1):37-60, 1994.
6. T. Cover. Universal portfolios. Mathematical Finance, 1(1):1-29, Jan 1991.
7. I. Csisźar and P. Shields. Redundancy rates for renewal and other processes. IEEE Transactions on Information Theory, 42(6):2065-2072, Nov 1996.
8. L. Davison. Universal noiseless coding. IEEE Transactions on Information Theory, 19(6):783-795, Nov 1973.
9. W. Gale. Good Turing smoothing without tears. Technical report, AT&T Bell Laboratories, Murray Hill, NJ 1994.
10. W. Gale and K. Church. What is wrong with adding one? In N. Oostdijk and P. de Haan, editors, Corpus based research into language. Rodopi, Amsterdam, 1994.
11. W. Gale, K. Church, and D. Yarowsky. A method for disambiguating word senses. Computers and Humanities, 26:415-419, 1993.
12. I. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(3/4):237-264, Dec 1953.
13. I. Good. Turing's anticipation of Empirical Bayes in connection with the cryptanalysis of the Naval Enigma. Journal of Statistics Computation and Simulation, 66:101-111, 2000.
14. G. Hardy and S. Ramanujan. Asymptotic formulae in combinatory analysis. Proceedings of London Mathematics Society, 17(2):75-115, 1918.
15. W. Hayman. A generalization of Stirling's formula. Journal für die reine und angewandte Mathematik, 196:67-95, 1956.
16. A. Hodges. Alan Turing: The Enigma. Walker & Co., 2000.
17. N. Jevtić, A. Orlitsky, and N. Santhanam. Universal compression of unknown alphabets. In Proceedings of IEEE Symposium on Information Theory, 2002.
18. R. Krichevsky and V. Trofimov. The performance of universal coding. IEEE Transactions on Information Theory, 27:199-207, 1981.
19. P. Laplace. Philosophical essays on probabilities. Springer Verlag, New York, Translated by A. Dale from the 5th (1825) edition, 1995.
20. N. Littlestone and M. Warmuth. The weighted majority algorithm. In IEEE Symposium on Foundations of Computer Science, 1992.
21. D. McAllester and R. Schapire. On the convergence rate of Good Turing estimators. In Proceedings of the Thirteenth annual conference on computational learning theory, 2000.
22. N. Merhav and M. Feder. Universal prediction. IEEE Transactions on Information Theory, 44(6):2124-2147, Oct 1998.
23. A. Nadas. On Turing's formula for word probabilities. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-33(6):1414-1416, Dec 1985.
24. A. Nadas. Good, Jelinek, Mercer, and Robins on Turing's estimate of probabilities. American Journal of Mathematical and Management Sciences, 11:229-308, 1991.
25. A. Orlitsky and N. Santhanam. Performance of universal codes over infinite alphabets. In Proceedings of the Data Compression Conference, Mar 2003.
26. A. Orlitsky, N. Santhanam, and J. Zhang. Universal compression of memoryless sources over unknown alphabets. Submitted to IEEE Transactions on Information Theory.
27. A. Orlitsky and K. Viswanathan. The most likely alphabet. In preparation.
28. J. Shtarkov. Coding of discrete sources with unknown statistics. In I. Csiszar and P. Elias, editors, Topics in Information Theory (Coll. Math. Soc. J. Bolyai, no. 16), pages 559-574. Amsterdam, The Netherlands: North Holland, 1977.
29. Y. Shtarkov. Universal sequential coding of single messages. Problems of Information Transmission, 23(3):3-17, Jul-Sep. 1987.
30. F. Song and W. Croft. A general language model for information retrieval (poster abstract). In Research and Development in Information Retrieval, pages 279-280, 1999.
31. V. Vovk. A game of prediction with expert advice. Journal of Computer and System Sciences, 56(2):153-173, 1998.
32. H. Wilf. Generatingfunctionology, pages 91. Academic Press, 1990.
33. I. Witten and T. Bell. The zero frequency problem. IEEE Transactions on Information Theory, 37(4):1085-1094, 1991.