A study of smoothing methods for language models applied to information retrieval

ACM Transactions on Information Systems

Volumn 22, Issue 2, 2004, Pages 179-214

  Zhai, Chengxiang ^a,b   Lafferty, John ^a,c  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b University of Illinois at Urbana Champaign   (United States)

^c Carnegie Mellon University   (United States)

Author keywords

Absolute discounting smoothing; Backoff smoothing; Dirichlet prior smoothing; EM algorithm; Interpolation smoothing; Jelinek Mercer smoothing; Leave one out; Risk minimization; Statistical language models; Term weighting; TF IDF weighting; Two stage smoothing

Indexed keywords

LANGUAGE MODELS; SENSITIVITY PATTERNS; SMOOTHING PARAMETERS; STATISTICAL LANGUAGE MODELING;

ALGORITHMS; COMPUTER SOFTWARE; DATA STORAGE EQUIPMENT; DATABASE SYSTEMS; HEURISTIC METHODS; MATHEMATICAL MODELS; POISSON DISTRIBUTION; ROBUSTNESS (CONTROL SYSTEMS); SENSITIVITY ANALYSIS; SPEECH RECOGNITION; STATISTICAL METHODS;

INFORMATION RETRIEVAL;

EID: 3042824043     PISSN: 10468188     EISSN: None     Source Type: Journal    
DOI: 10.1145/984321.984322     Document Type: Article

References (30)

1. BERGER, A. AND LAFFERTY, J. 1999. Information retrieval as statistical translation. In Proceedings of the 1999 ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 222-229.
2. CHEN, S. F. AND GOODMAN, J. 1998. An empirical study of smoothing techniques for language modeling. Tech. Rep. TR-10-98, Harvard University.
3. FUHR, N. 1992. Probabilistic models in information retrieval. Comput J. 35, 3, 243-255.
4. GOOD, I. J. 1953. The population frequencies of species and the estimation of population parameters. Biometrika 40, parts 3, 4, 237-264.
5. HIEMSTRA, D. AND KRAAIJ, W. 1999. Twenty-one at TREC-7: Ad-hoc and cross-language track. In Proceedings of 7th Text REtrieval Conference (TREC-7). 227-238.
6. JELINEK, F. AND MERCER, R. 1980. Interpolated estimation of markov sourceparameters from sparse data. In Pattern Recognition in Practice, E. S. Gelsema and L. N. Kanal, Eds. 381-402.
7. KATZ, S. M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Trans. Acoustics, Speech and Signal Processing (ASSP) 35 400-401.
8. KNESER, R. AND NEY, H. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE Computer Society Press, Los Alamitos, Calif, 181-184.
9. KRAAIJ, W., WESTERVELD, T., AND HIEMSTRA, D. 2002. The importance of prior probabilities for entry-page search. In Proceedings of SIGIR'02. ACM, New York, 27-34.
10. KWOK, K. AND CHAN, M. 1998. Improving two-stage ad-hoc retrieval for short queries. In Proceedings of SIGIR'98. ACM, New York, 250-256.
11. LAFFBRTY, J. AND ZHAI, C. 2001. Document language models, query models, and risk minimization for information retrieval. In Proceedings of SIGIR'01, ACM, New York, 111-119.
12. LAVBENKO, V. AND CROFT, B. 2001. Relevance-based language models. In Proceedings of SIGIR'01. ACM, New York, 120-127.
13. MACKAY, D. AND PETO, L. 1995. A hierarchical Dirichlet language model. Nat. Lang. Eng. 1, 3, 289-307.
14. MILLER, D. H., LEEK, T., AND SCHWARTZ, R. 1999. A hidden Markov model information retrieval system. In Proceedings of the 1999 ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 214-221.
15. NEY, H., ESSEN, U., AND KNESER, R. 1994. On structuring probabilistic dependencies in stochastic language modeling. Comput. Speech Lang. 8, 1-38.
16. NEY, H., ESSEN, U., AND KNESER, R. 1995. On the estimation of 'small' probabilities by leaving-one-out. IEEE Trans. Pattern Anal. Machine Intel. 17, 12, 1202-1212.
17. PONTE, J. 1998. A language modeling approach to information retrieval. Ph.D. dissertation. Univ. of Massachusetts at Amherst.
18. PONTE, J. AND CROFT, W. B. 1998. A language modeling approach to information retrieval. In Proceedings of the ACM SIGIR'98. ACM, New York, 275-281.
19. ROBERTSON, S. E., VAN RUSBERGEN, C. J., AND PORTER, M. F. 1981. Probabilistic models of indexing and searching. In Information Retrieval Research, R. N. Oddy et al., Eds. Butterworths, 35-56.
20. ROBERTSON, S. E., WALKER, S., JONES, S., M. HANCOCK-BEAULIEU, M., AND GATFORD, M. 1995. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC-3), D. K. Harman, Ed. 109-126.
21. SALTON, G. AND BUCKLEY, C. 1988. Term-weighting approaches in automatic text retrieval. Inf. Proc. Manage. 24, 513-523.
22. SALTON, G. AND BUCKLEY, C. 1990. Improving retrieval performance by relevance feedback. J. Amer. Soc. Inf. Sci. 44, 4, 288-297.
23. SALTON, G., WONG, A., AND YANG, C. S. 1975. A vector space model for automatic indexing. Commun. ACM 18, 11, 613-620.
24. SINGHAL, A., BUCKLEY, C., AND MITRA, M. 1996. Pivoted document length normalization. In Proceedings of the 1996 ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 21-29.
25. SONG, F. AND CROFT, B. 1999. A general language model for information retrieval. In Proceedings of the 1999 ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 279-280.
26. SPARCK JONES, K. AND WILLETT, P., Eds. 1997. Readings in Information Retrieval. Morgan-Kaufmann, San Mateo, Calif.
27. VAN RIJSBERGEN, C. J. 1986. A non-classical logic for information retrieval. Comput. J. 29, 6, 481-485.
28. VOORHEES, E. AND HARMAN, D., Eds. 2001. Proceedings of Text REtrieval Conference (TREC1-9). NIST Special Publications, http://trec.nist.gov/pubs.html.
29. WONG, S. K. M. AND YAO, Y. Y. 1995. On modeling information retrieval with probabilistic inference. ACM Trans. Inf. Syst. 13, 1, 69-99.
30. ZHAI, C. AND LAFFERTY, J. 2001. Model-based feedback in the KL-divergence retrieval model. In Proceedings of the 10th International Conference on Information and Knowledge Management (CIKM 2001). 403-410.