A survey of smoothing techniques for ME models

IEEE Transactions on Speech and Audio Processing

Volumn 8, Issue 1, 2000, Pages 37-50

  Chen, Stanley F ^a,b   Rosenfeld, Ronald ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b Carnegie Mellon University ^*  (United States)

Author keywords

Exponential models; Language modeling; Maximum entropy; Minimum divergence; N gram models; Smoothing

Indexed keywords

ALGORITHMS; FUZZY SETS; LEARNING SYSTEMS; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; SPEECH ANALYSIS;

MAXIMUM ENTROPY (ME) MODELING;

SPEECH PROCESSING;

EID: 0033887568     PISSN: 10636676     EISSN: None     Source Type: Journal    
DOI: 10.1109/89.817452     Document Type: Article

References (43)

1. S. Della Pietra and V. Della Pietra, "Statistical modeling by ME," Int. Rep., IBM, 1993.
2. A. Berger, S. Della Pietra, and V. Della Pietra, "A ME approach to natural language processing," Comput. Linguist., vol. 22, pp. 39-71, 1996.
3. S. Della Pietra, V. Della Pietra, and J. Lafferty, "Inducing features of random fields," IEEE Trans. Pattern Anal. Machine Intell., vol. 19, pp. 380-393, Apr. 1997.
4. A. Ratnaparkhi, "ME models for natural language ambiguity resolution," Ph.D. dissertation, Univ. Pennsylvania, Philadelphia, PA, 1998.
5. R. Rosenfeld, "A ME approach to adaptive statistical language modeling," Comput. Speech Lang., vol. 10, pp. 187-228, 1996.
6. R. Lau, "Adaptive statistical language modeling," M.S. thesis, Dept. Elect. Eng. Comput. Sci., Mass. Inst. Technol., Cambridge, MA, 1994.
7. "Adaptive statistical language modeling: A ME approach," Ph.D. dissertation, Carnegie Mellon Univ., Pittsburgh, PA, Apr. 1994.
8. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling," Comput. Speech Lang., vol. 13, pp. 359-393, 1999.
9. E. Jaynes, "Information theory and statistical mechanics," Phys. Rev., vol. 106, pp. 620-630, 1957.
10. J. Darroch and D. Ratcliff, "Generalized iterative scaling for log-linear models," Ann. Math. Stat., vol. 43, pp. 1470-1480, 1972.
11. L. R. Bahl, F. Jelinek, and R. L. Mercer, "A maximum likelihood approach to continuous speech recognition," IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-5, pp. 179-190, Mar. 1983.
12. P. F. Brown et al., "A statistical approach to machine translation," Comput. Linguist., vol. 16, no. 2, pp. 79-85, Jun. 1990.
13. M. Kernighan, K. Church, and W. Gale, "A spelling correction program based on a noisy channel model," in Proc. 13th Int. Conf. Computational Linguistics, Aug. 1990, pp. 205-210.
14. F. Jelinek and R. L. Mercer, "Interpolated estimation of Markov source parameters from sparse data," in Proc. Workshop on Pattern Recognition in Practice, May 1980, pp. 381-397.
15. P. F. Brown et al., "An estimate of an upper bound for the entropy of English," Comput. Linguist., vol. 18, no. 1, pp. 31-40, Mar. 1992.
16. T. C. Bell, J. G. Cleary, and I. H. Witten, Text Compression. Englewood Cliffs, NJ: Prentice-Hall, 1990.
17. S. M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-35, pp. 400-401, Mar. 1987.
18. R. Kneser and H. Ney, "Improved backing-off for m-gram language modeling," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, May 1995, pp. 181-184.
19. H. Ney, U. Essen, and R. Kneser, "On structuring probabilistic dependences in stochastic language modeling," Comput. Speech Lang., vol. 8, pp. 1-38, 1994.
20. I. Good, "The population frequencies of species and the estimation of population parameters," Biometrika, vol. 40, pp. 237-264, 1953.
21. K. W. Church and W. A. Gale, "A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams," Comput. Speech Lang., vol. 5, pp. 19-54, 1991.
22. S. F. Chen, D. Beeferman, and R. Rosenfeld, "Evaluation metrics for language models," in Proc. DARPA Broadcast News Transcription and Understanding Workshop, 1998, pp. 275-280.
23. P. Brown et al., "A maximum penalized entropy construction of conditional log-linear language and translation models using learned features and a generalized Csiszar algorithm," Int. Rep., IBM, 1992.
24. R. Lau, R. Rosenfeld, and S. Roukos, "Adaptive language modeling using the ME principle," in Proc. ARPA Workshop on Human Language Technology, 1993, pp. 108-113.
25. D. B. Paul and J. M. Baker, "The design for the Wall Street Journal-based CSR corpus," in Proc. DARPA Speech and Natural Language Workshop, Feb. 1992, pp. 357-362.
26. S. F. Chen and R. Rosenfeld, "A Gaussian prior for smoothing ME models," Comput. Sci. Dept., Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-99-108, 1999.
27. J. Lafferty, private communication, 1997.
28. W. Newman, "Extension to the ME method," IEEE Trans. Inform. Theory; vol. IT-23, pp. 89-93, Jan. 1977.
29. S. Khudanpur, "A method of ME estimation with relaxed constraints," Proc. 1995 Johns Hopkins Univ. Language Modeling Workshop, pp. 1-17, 1995.
30. D. J. C. MacKay and L. C. Peto, "A hierarchical Dirichlet language model," Natural Lang. Eng., vol. 1, pp. 1-19, 1995.
31. A. Nádas, "Estimation of probabilities in the language model of the IBM speech recognition system," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-32, pp. 859-861, Aug. 1984.
32. J. Lafferty and B. Suhm, "Cluster expansions and iterative scaling for ME language models," in ME and Bayesian Methods, K. Hanson and R. Silver, Eds. Boston, MA: Kluwer, 1995, pp. 195-202.
33. H. Kucera and W. Francis, Computational Analysis of Present-Day American English. Providence, RI: Brown Univ. Press, 1967.
34. R. M. Stern, "Specification of the 1995 ARPA Hub 3 evaluation: Unlimited vocabulary NAB news baseline," in Proc. DARPA Speech Recognition Workshop, Feb. 1996, pp. 5-7.
35. A. Rudnicky, "Hub 4: Business Broadcast News," in Proc. DARPA Speech Recognition Workshop, Feb. 1996, pp. 8-11.
36. J. Godfrey, E. Holliman, and J. McDaniel, "SWITCHBOARD: Telephone speech corpus for research and development," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, vol. I, Mar. 1992, pp. 517-520.
37. W. Press, B. Flannery, S. Teukolsky, and W. Vetterling, Numerical Recipes in C. Cambridge, MA: Cambridge Univ. Press, 1988.
38. M. Siegler, A. Berger, M. Witbrock, and A. Hauptmann, "Experiments in spoken document retrieval at CMU," in Proc. TREC-7: The 7th Text Retrieval Conf., 1998, pp. 319-325.
39. K. Seymore et al., "The 1997 CMU Sphinx-3 English Broadcast News transcription system," in Proc. DARPA Speech Recognition Workshop, 1998, pp. 55-59.
40. S. Kullback, Information Theory and Statistics, New York: Wiley, 1959.
41. D. Beeferman, A. Berger, and J. Lafferty, "A model of lexical attraction and repulsion," in Proc. 35th Annual Meet. Assoc. Computational Linguistics, 1997, pp. 373-380.
42. K. Nigam, J. Lafferty, and A. McCallum, "Using ME for text classification," in Proc. IJCAI-99 Workshop on Machine Learning for Information Filtering, 1999, pp. 61-67.
43. A. Berger and R. Miller, "Just-in-time language modeling," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. II, 1998, pp. 705-708.