Knowing the unseen: Estimating vocabulary size over unseen samples

ACL-IJCNLP 2009 - Joint Conf. of the 47th Annual Meeting of the Association for Computational Linguistics and 4th Int. Joint Conf. on Natural Language Processing of the AFNLP, Proceedings of the Conf.

Volumn , Issue , 2009, Pages 108-117

  Bhat, Suma ^a   Sproat, Richard ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EMPIRICAL STUDIES; GOOD-TURING; LANGUAGE MODEL; LINGUISTIC PHENOMENA; NON-PARAMETRIC; NONPARAMETRIC ESTIMATORS; SAMPLE SIZES; STATE OF THE ART; VOCABULARY SIZE; WORD LENGTH;

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; SAMPLING;

ESTIMATION;

EID: 84859899929     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (8)

1. R. H. Baayen. 2001. Word Frequency Distributions, Kluwer Academic Publishers.
2. Marco Baroni and Stefan Evert. 2001. "Testing the extrapolation quality of word frequency models", Proceedings of Corpus Linguistics , volume 1 of The Corpus Linguistics Conference Series, P. Danielsson and M. Wagenmakers (eds.).
3. J. Bunge and M. Fitzpatrick. 1993. "Estimating the number of species: a review", Journal of the American Statistical Association, Vol. 88(421), pp. 364-373.
4. A. Gandolfi and C. C. A. Sastri. 2004. "Nonparametric Estimations about Species not Observed in a Random Sample", Milan Journal of Mathematics, Vol. 72, pp. 81-105.
5. E. V. Khmaladze. 1987. "The statistical analysis of large number of rare events", Technical Report, Department of Mathematics and Statistics., CWI, Amsterdam, MS-R8804.
6. E. V. Khmaladze and R. J. Chitashvili. 1989. "Statistical analysis of large number of rate events and related problems", Probability theory and mathematical statistics (Russian), Vol. 92, pp. 196-245.
7. P. Santhanam, A. Orlitsky, and K. Viswanathan, "New tricks for old dogs: Large alphabet probability estimation", in Proc. 2007 IEEE Information Theory Workshop, Sept. 2007, pp. 638-643.
8. A. B. Wagner, P. Viswanath and S. R. Kulkarni. 2006. "Strong Consistency of the Good-Turing estimator", IEEE Symposium on Information Theory, 2006.