A maximum entropy approach to adaptive statistical language modelling

Computer Speech and Language

Volumn 10, Issue 3, 1996, Pages 187-228

  Rosenfeld, Ronald ^a  

^a Carnegie Mellon University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; CONSTRAINT THEORY; CONVERGENCE OF NUMERICAL METHODS; ITERATIVE METHODS; MATHEMATICAL MODELS; PROBABILITY; SPEECH RECOGNITION;

ADAPTIVE STATISTICAL LANGUAGE MODELING; MAXIMUM ENTROPY APPROACH; TRIGGER PAIRS;

COMPUTATIONAL LINGUISTICS;

EID: 0030181951     PISSN: 08852308     EISSN: None     Source Type: Journal    
DOI: 10.1006/csla.1996.0011     Document Type: Article

References (49)

1. Abramson, N. (1963). Information Theory and Coding. New York: McGraw-Hill.
2. Bahl, L., Jelinek, F. & Mercer, R. (1983). A maximum likelihood approach to continuous speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-5, 179-190.
3. Brown, P., DellaPietra, S., DellaPietra, V., Mercer, R., Nadas, A. & Roukos, S. (1994). A maximum penalized entropy construction of conditional log-linear language and translation models using learned features and a generalized csiszar algorithm. Unpublished IBM research report.
4. Brown, P. F., DellaPietra, V. J., deSouza, P. V., Lai, J. C. & Mercer, R. L. (1990). Class-based N-gram models of natural language. Proceedings of the IBM Natural Language ITL, March, Paris, France.
5. Chase, L., Rosenfeld, R. & Ward, W. (1994). Error-responsive modifications to speech recognizers: negative N-grams. Proceedings of the International Conference on Spoken Language Processing, September, Yokohama, Japan.
6. Church, K. & Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational Linguistics 16, 22-29.
7. Cover, T. M. & King, R. C. (1978). A convergent gambling estimate of the entropy of English. IEEE Transactions on Information Theory IT-24, 413-421.
8. Darroch, J. N. & Ratcliff, D. (1972). Generalized iterative scaling for log-linear models. The Annals of Mathematical Statistics 43, 1470-1480.
9. DellaPietra, S., DellaPietra, V., Mercer, R. & Roukos, S. (1992). Adaptive language modelling using minimum discriminant estimation. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pp. I-633-636, March, San Francisco, CA. Also published in Proceedings of the DARPA Workshop on Speech and Natural Language. San Mateo, CA: Morgan Kaufmann, pp. 103-106.
10. DellaPietra, S., DellaPietra, V., Mercer, R. & Roukos, S. (1992). Adaptive language modelling using minimum discriminant estimation. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pp. I-633-636, March, San Francisco, CA. Also published in Proceedings of the DARPA Workshop on Speech and Natural Language. San Mateo, CA: Morgan Kaufmann, pp. 103-106.
11. Dempster, A. P., Laird, N. M. & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39, 1-38.
12. Derouault, A.-M. & Merialdo, B. (1986). Natural language modeling for phoneme-to-text transcription. IEEE Transactions on Pattern Analysis and Machine Translation PAMI-8, 742-749.
13. Good, I. J. (1953). The population frequencies of species and the estimation of population parameters. Biometrika 40, 237-264.
14. Good, I. J. (1963). Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables. Annals of Mathematical Statistics 34, 911-934.
15. Huang, X., Alleva, F., Hwang, M.-Y. & Rosenfeld, R. (1993a). An overview of the SPHINX-II speech recognition system. Proceedings of the ARPA Human Language Technology Workshop, published as Human Language Technology, pp. 81-86. San Francisco, CA: Morgan Kaufmann.
16. Huang, X., Alleva, F., Hon, H.-W., Hwang, M.-Y., Lee, K.-F. & Rosenfeld, R. (1993b). The SPHINX-II speech recognition system: an overview. Computer, Speech and Language 2, 137-148.
17. Hwang, M.-Y., Rosenfeld, R., Thayer, E., Mosur, R., Chase, L., Weide, R., Huang, X. & Alleva, F. (1994). Improved Acoustic and Adaptive Language Models for Continuous Speech Recognition. In Proceedings of the ARPA Spoken Language Technologies Workshop, March.
18. Jaynes, E. T. (1957). Information theory and statistical mechanics. Physics Reviews 106, 620-630.
19. Jelinek, F. (1989). Self-organized language modelling for speech recognition. In Readings in Speech Recognition (Waibel, A. & Lee, K.-F., eds). San Francisco, CA: Morgan Kaufmann.
20. Jelinek, F. (1991). Up from trigrams! Eurospeech.
21. Jelinek, F., Mercer, R. L., Bahl, L. R. & Baker, J. K. (1977). Perplexity-a measure of difficulty of speech recognition tasks. 94th Meeting of the Acoustic Society of America, December, Miami Beach, FL.
22. Jelinek, F. & Mercer, R. (1980). Interpolated estimation of Markov source parameters from sparse data. In Pattern Recognition in Practice (Gelsema, E. S. & Kanal, L. N., eds), pp. 381-402. Amsterdam: North Holland.
23. Jelinek, F., Merialdo, B., Roukos, S. & Strauss, M. (1991). A dynamic language model for speech recognition. In Proceedings of the DARPA Workshop on Speech and Natural Language, pp. 293-295, February.
24. Katz, S. M. (1987). Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing ASSP-35, 400-401.
25. Kneser, R. & Ney, H. (1991). Forming word classes by statistical clustering for statistical language modelling. Proceedings of the 1st QUALICO Conference, September, Trier, Germany.
26. Kneser, R. & Ney, H. (1995). Improved smoothing for M-gram language modeling. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, May, Detroit, MI.
27. Kubala, F. & members of the CSR Corpus Coordinating Committee (CCCC) (1994). The hub and spoke paradigm for CSR evaluation. Proceedings of the ARPA Workshop on Human Language Technology, pp. 40-44. San Francisco, CA: Morgan Kaufmann.
28. Kuhn, R. (1988). Speech recognition and the frequency of recently used words: a modified Markov model for natural language. 12th International Conference on Computational Linguistics [COLING 88], pp. 348-350, August, Budapest.
29. Kuhn, R. & De Mori, R. (1990). A cache-based natural language model for speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-12, 570-583.
30. Kuhn, R. & De Mori, R. (1992). Correction to a cache-based natural language model for speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-14, 691-692.
31. Kullback, S. (1959). Information Theory in Statistics. New York, NY: Wiley.
32. Kupiec, J. (1989). Probabilistic models of short and long distance word dependencies in running text. Proceedings of the DARPA Workshop on Speech and Natural Language, pp. 290-295, February.
33. Lau, R. (1993). Maximum likelihood maximum entropy trigger language model. Bachelor's Thesis, Massachusetts Institute of Technology, MA.
34. Lau, R., Rosenfeld, R. & Roukos, S. (1993a). Trigger-based language models: a maximum entropy approach. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pp. II45-48, April, Minneapolis, MN.
35. Lau, R., Rosenfeld, R. & Roukos, S. (1993b). Adaptive language modeling using the maximum entropy principle. Proceedings of the ARPA Human Language Technology Workshop, published as Human Language Technology, pp. 108-113. San Francisco, CA: Morgan Kaufmann.
36. Pallett, D. S., Fiscus, J. G., Fisher, W. M., Garofolo, J. S., Lund, B. & Pryzbocki, M. (1993). Benchmark tests for the ARPA spoken language program. Proceedings of the ARPA Workshop on Human Language Technology, pp. 51-73. March, San Francisco, CA: Morgan Kaufmann.
37. Paul, D. B. & Baker, J. M. (1992). The design for the Wall Street Journal-based CSR corpus. In Proceedings of the DARPA SLS Workshop, February.
38. Price, P. (1990). Evaluation of spoken language systems: the ATIS domain. Proceedings of the Third DARPA Speech and Natural Language Workshop (Stern, R., ed.). San Francisco, CA: Morgan Kaufmann.
39. Ratnaparkhi, A. & Roukos, S. (1994). A maximum entropy model for prepositional phrase attachment. Proceedings of the ARPA Workshop on Human Language Technology, pp. 242-242e, March. San Francisco, CA: Morgan Kaufmann.
40. Rosenfeld, R. (1992). Adaptive statistical language modeling: a maximum entropy approach. Ph.D. Thesis Proposal, Carnegie Mellon University, Pittsburgh, PA.
41. Rosenfeld, R. (1994a). A hybrid approach to adaptive statistical language modeling. Proceedings of the ARPA Workshop on Human Language Technology, pp. 76-81, March. San Francisco, CA: Morgan Kaufmann.
42. Rosenfeld, R. (1994). Adaptive statistical language modeling: a maximum entropy approach. Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA. Also published as Technical Report CMU-CS-94-138, School of Computer Sciences, Carnegie Mellon University, Pittsburgh, PA.
43. Rosenfeld, R. & Huang, X. (1992). Improvements in stochastic language modelling. Proceedings of the DARPA Workshop on Speech and Natural Language, pp. 107-111, February. San Francisco, CA: Morgan Kaufmann.
44. Shannon, C. E. (1948). A mathematical theory of communication. Bell Systems Technical Journal 27, 379-423 (Part I), 623-656 (Part II).
45. Shannon, C. E. (1948). A mathematical theory of communication. Bell Systems Technical Journal 27, 379-423 (Part I), 623-656 (Part II).
46. Shannon, C. E. (1951). Prediction and entropy of printed English. Bell Systems Technical Journal 30, 50-64.
47. Suhm, B. & Waibel, A. (1994). Towards better language models for spontaneous speech. ICSLP 94 2, 831-834.
48. Ward, W. (1990). The CMU air travel information service: understanding spontaneous speech. Proceedings of the DARPA Speech and Natural Language Workshop, pp. 127-129, June.
49. Ward, W. (1991). Evaluation of the CMU ATIS system. Proceedings of the DARPA Speech and Natural Language Workshop, pp. 101-105, February.