A bit of progress in language modeling

Computer Speech and Language

Volumn 15, Issue 4, 2001, Pages 403-434

  Goodman, Joshua T ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; COMPUTATIONAL METHODS; GRAPH THEORY; INTERPOLATION; MATHEMATICAL MODELS; NEURAL NETWORKS; PROBABILITY; SEMANTICS; SPEECH RECOGNITION;

KNESER-NEY SMOOTHING; LANGUAGE MODELING; SENTENCE MIXTURE MODEL; TRIGRAM MODEL;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 0035497388     PISSN: 08852308     EISSN: None     Source Type: Journal    
DOI: 10.1006/csla.2001.0174     Document Type: Article

References (52)

1. Bellegarda, J. R., Butzberger, J. W., Chow, Y.-L., Coccaro, N. B. & Naik, D. (1996). A novel word clustering algorithm based on latent semantic analysis. International Conference on Acoustics, Speech and Signal Processing, volume 1, pp. 172-175.
2. Bellegarda, J. R. (2000). Exploiting latent semantic information in statistical language modeling. Proceedings of the IEEE, 88, 1279-1296.
3. Bengio, Y., Ducharme, R. & Vincent, P. (2000). A neural probabilistic language model. Technical Report 1178, Département d'informatique et recherche opérationnelle, Université de Montréal.
4. Blasig, R. (1999). Combination of words and word categories in varigram histories. In International Conference on Acoustics, Speech and Signal Processing, volume 1, pp. 529-532.
5. Brown, P. F., Cocke, J., DellaPietra, S. A., DellaPietra, V. J., Jelinek, F., Lafferty, J. D., Mercer, R. L. & Roossin, P. S. (1990). A statistical approach to machine translation. Computational Linguistics, 16, 79-85.
6. Brown, P. F., DellaPietra, V. J., deSouza, P. V., Lai, J. C. & Mercer, R. L. (1992). Class-based n-gram models of natural language. Computational Linguistics, 18, 467-479.
7. Cai, C., Rosenfeld, R. & Wasserman, L. (May 2000). Exponential language models, logistic regression, and semantic coherence. Proceedings of NIST/DARPA Speech Transcription Workshop.
8. Charniak, E. (2001). Immediate-head parsing for language models. In ACL-01. pp. 116-123.
9. Chelba, C. & Jelinek, F. (1998). Exploiting syntactic structure for language modeling. Proceedings of the 36th Annual Meeting of the ACL, pp. 225-231.
10. Chen, S. & Rosenfeld, R. (1999a). Efficient sampling and feature selection in whole sentence maximum entropy language models. Proceedings of International Conference on Acoustics, Speech and Signal Processing, 1, pp. 549-552.
11. Chen, S. F. & Goodman, J. (October 1999). An empirical study of smoothing techniques for language modeling. Computer Speech and Language, 13, 359-394.
12. Chen, S. F. & Goodman, J. T. An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard University, 1998. Available from http://www.research.microsoft.com/~joshuago.
13. Chen, S. F. & Rosenfeld, R. (1999b). A gaussian prior for smoothing maximum entropy models. Technical Report CMU-CS-99-108, Computer Science Department, Carnegie Mellon University.
14. Church, K. (1988). A stochastic parts program and noun phrase parser for unrestricted text. Proceedings of the Second Conference on Applied Natural Language Processing, pp. 136-143.
15. Cutting, D. R., Karger, D. R., Pedersen, J. R. & Tukey, J. W. (1992). Scatter/gather: A cluster-based approach to browsing large document collections. SIGIR 92, pp. 318-329.
16. Darroch, J. N. & Ratcliff, D. (1972). Generalized iterative scaling for log-linear models. The Annals of Mathematical Statistics, 43, 1470-1480.
17. Dupont, P. & Rosenfeld, R. (1997). Lattice based language models. Technical Report CMU-CS-97-173, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
18. Good, I. J. (1953). The population frequencies of species and the estimation of population parameters. Biometrika, 40, 237-264.
19. Goodman, J. (2001a). A bit of progress in language modeling, extended version. Technical Report, Microsoft Research. Draft available from http://www.research.microsoft.com/~joshuago/longcombine/ps.
20. Goodman, J. (2001b). Classes for fast maximum entropy training. International Conference on Acoustics, Speech and Signal Processing.
21. Goodman, J. & Gao, J. (2000). Language model size reduction by pruning and clustering. International Conference on Spoken Language Processing, 3, pp. 110-113.
22. Huang, X., Alleva, F., Hon, H.-W., Hwang, M.-Y., Lee, K.-F. & Rosenfeld, R. (1993). The SPHINX-II speech recognition system: An overview. Computer, Speech, and Language, 2, 137-148.
23. Hull, J. (1992). Combining syntactic knowledge and visual text recognition: A hidden Markov model for part of speech tagging in a word recognition algorithm. AAAI Symposium: Probabilistic Approaches to Natural Language, pp. 77-83.
24. Iyer, R. & Ostendorf, M. (1999). Modeling long distance dependence in language: Topic mixtures versus dynamic cache models. IEEE Transactions on Acoustics, Speech and Audio Processing, 7, 30-39.
25. Iyer, R. & Ostendorf, M. & Robin, R. J. (1994). Language modeling with sentence-level mixtures. DARPA-HLT, pp. 82-86.
26. Jelinek, F., Merialdo, B., Roukos, S. & Strauss, M. (1991). A dynamic lm for speech recognition. Proceedings of ARPA Workshop on Speech and Natural Language, pp. 293-295.
27. Jelinek, F. & Mercer, R. L. (1980). Interpolated estimation of Markov source parameters from sparse data. Proceedings of the Workshop on Pattern Recognition in Practice, pp. 381-397. North-Holland, Amsterdam. The Netherlands.
28. Jurafsky, D., Wooters, C., Segal, J., Fosler, E., Tajchman, G. & Morgan, N. (1995). Using a stochastic context-free grammar as a language model for speech recognition. International Conference on Acoustics, Speech and Signal Processing, pp. 189-192.
29. Katz, S. M. (1987). Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35, 400-401.
30. Kernighan, M. D., Church, K. W. & Gale, W. A. (1990). A spelling correction program based on a noisy channel model. Proceedings of the Thirteenth International Conference on Computational Linguistics, pp. 205-210.
31. Kneser, R. & Ney, H. (1993). Improved clustering techniques for class-based statistical language modeling. Eurospeech 93, volume 2, pp. 973-976.
32. Kuhn, R. (1988). Speech recognition and the frequency of recently used words: A modified Markov model for natural language. 12th International Conference on Computational Linguistics, Budapest, Hungary, pp. 348-350.
33. Kuhn, R. & De Mori, R. (1990). A cache-based natural language model for speech reproduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 570-583.
34. Kuhn, R. & De Mori, R. (1992). Correction to a cache-based natural language model for speech reproduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14, 691-692.
35. Kupiec, J. (1989). Probabilistic models of short and long distance word dependencies. Proceedings of ARPA Workshop on Speech and Natural Language, pp. 290-295.
36. Martin, S., Hamacher, C., Liermann, J., Wessel, F. & Ney, H. (1999), Assessment of smoothing methods and complex stochastic language modeling. 6th European Conference on Speech Communication and Technology, Budapest, Hungary, volume 5, pp. 1939-1942.
37. Ney, H., Essen, U. & Kneser, R. (1994). On structuring probabilistic dependences in stochastic language modeling. Computer, Speech, and Language, 8, 1-38.
38. Niesler, T. R., Whittaker, E. W. D. & Woodland, P. C. (1998). Comparison of part-of-speech and automatically derived category-based language models for speech recognition. International Conference on Acoustics, Speech and Signal Processing, volume 1, pp. 177-180.
39. Pereira, F., Tishby, N. & Lee, L. (1993). Distributional clustering of english words. Proceedings of the 31st Annual Meeting of the ACL, pp. 183-190.
40. Press, W. H., Flannery, B. P., Teukolsky, S. A. & Vetterling, W. T. (1988). Numerical Recipes in C. Cambridge University Press, Cambridge.
41. Rosenfeld, R. (1994). Adaptive statistical language modeling: a maximum entropy approach. PhD Thesis. Carnegie Mellon University.
42. Rosenfeld, R. (2000). Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88, 1270-1278.
43. Rosenfeld, R., Chen, S. F. & Zhu, X. (2001). Whole-sentence exponential language models: a vehicle for linguistic-statistical integration. Computer Speech and Language, to appear.
44. Saul, L. & Pereira, F. (1997). Aggregate and mixed-order Markov models for statistical language processing. Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, pp. 81-89.
45. Siu, M. & Ostendorf, M. (2000). Variable n-grams and extensions for conversational speech language modeling. IEEE Transactions on Speech and Audio Processing, 8, 63-75.
46. Srihari, R. & Baltus, C. (1992). Combining statistical and syntactic methods in recognizing handwritten sentences. AAAI Symposium: Probabilistic Approaches to Natural Language. pp. 121-127.
47. Stern, R. M. (1996). Specification of the 1995 ARPA hub 3 evaluation: Unlimited vocabulary NAB news baseline. Proceedings of the DARPA Speech Recognition Workshop, pp. 5-7.
48. Tillmann, C. & Ney, H. (1996). Statistical language modeling and word triggers. Proceedings of the International Workshop "Speech and Computer" (SPECOM 96), pp. 22-27.
49. Ueberla, J. P. (1995). More efficient clustering of n-grams for statistical language modeling. Eurospeech-95, pp. 1257-1260.
50. Weng, F., Stolcke, A. & Sankar, A. (1997). Hub4 language modeling using domain interpolation and data clustering. 1997 DARPA Speech Recognition Workshop, pp. 147-151.
51. Yamamoto, H. & Sagisaka, Y. (1999). Multi-class composite n-gram based on connection direction. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Phoenix, Arizona.
52. Zhang, R., Black, E., Finch, A. & Sagisaka, Y. (2000). Integrating detailed information into a language model. International Conference on Acoustics, Speech and Signal Processing, pp. 1595-1598.