Concentration bounds for unigram language models

Journal of Machine Learning Research

Volumn 6, Issue , 2005, Pages

  Drukh, Evgeny ^a   Mansour, Yishay ^a  

^a TEL AVIV UNIVERSITY   (Israel)

Author keywords

Chernoff bounds; Good Turing estimators; Leave one out estimation; Logarithmic loss

Indexed keywords

LEARNING ALGORITHMS; OPTICAL CHARACTER RECOGNITION; PROBABILITY; RANDOM PROCESSES; SPEECH RECOGNITION; TECHNICAL PRESENTATIONS;

CHERNOFF BOUNDS; GOOD-TURNING ESTIMATORS; LEAVE-ONE-OUT ESTIMATION; LOGARITHMIC LOSS; UNIGRAM MODELS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 23744441936     PISSN: 15337928     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (20)

1. D. Angluin and L. G. Valiant. Fast probabilistic algorithms for Hamiltonian circuits and matchings. Journal of Computer and System Sciences, 18:155-193, 1979.
2. S. F. Chen. Building Probabilistic Models for Natural Language. PhD thesis, Harvard University, 1996.
3. S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard University, 1998.
4. K. W. Church and W. A. Gale. A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams. Computer Speech and Language, 5:19-54, 1991.
5. J. R. Curran and M. Osborne. A very very large corpus doesn't always yield reliable estimates. In Proceedings of the Sixth Conference on Natural Language Learning, pages 126-131, 2002.
6. D. P. Dubhashi and D. Ranjan. Balls and bins: A study in negative dependence. Random Structures and Algorithms, 13(2):99-124, 1998.
7. P. Flajolet. Singularity analysis and asymptotics of Bernoulli sums. Theoretical Computer Science, 215:371-381, 1999.
8. I. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(16):237-264, 1953.
9. I. J. Good. Turing's anticipation of empirical Bayes in connection with the cryptanalysis of the naval Enigma. Journal of Statistical Computation and Simulation, 66(2):101-112, 2000.
10. W. Hoeffding. On the distribution of the number of successes in independent trials. Annals of Mathematical Statistics, 27:713-721, 1956.
11. W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.
12. S. B. Holden. PAC-like upper bounds for the sample complexity of leave-one-out cross-validation. In Proceesings of the Ninth Annual ACM Workshop on Computational Learning Theory, pages 41-50, 1996.
13. S. M. Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, 35(3):400-401, 1987.
14. M. Kearns and D. Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Computation, 11(6):1427-1453, 1999.
15. S. Kutin. Algorithmic Stability and Ensemble-Based Learning. PhD thesis, University of Chicago, 2002.
16. D. McAllester and L. Ortiz. Concentration inequalities for the missing mass and for histogram rule error. Journal of Machine Learning Research, Special Issue on Learning Theory, 4(Oct):895-911, 2003.
17. D. McAllester and R. E. Schapire. On the convergence rate of Good-Turing estimators. In Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, pages 1-6, 2000.
18. D. McAllester and R. E. Schapire. Learning theory and language modeling. In Seventeenth International Joint Conference on Artificial Intelligence, 2001.
19. C. McDiarmid. On the method of bounded differences. In Surveys in Combinatorics, pages 148-188. London Math. Soc. Lectures Notes 141, Cambridge University Press, 1989.
20. A. Orlitsky, N. P. Santhanam, and J. Zhang. Always Good Turing: Asymptotically optimal probability estimation. Science, 302(Oct):427-431, 2003.