IDF revisited: A simple new derivation within the Robertson-Spärck Jones probabilistic model

Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Volumn , Issue , 2007, Pages 751-752

  Lee, Lillian ^a  

^a Department of Computer Science and School of Operations Research and Information Engineering   (United States)

Author keywords

IDF; Inverse document frequency; Probabilistic model; Term weighting

Indexed keywords

COMPUTATIONAL COMPLEXITY; MATHEMATICAL MODELS; PROBABILISTIC LOGICS; PROBLEM SOLVING;

INVERSE DOCUMENT FREQUENCY (IDF); PROBABILISTIC MODEL; TERM WEIGHTING;

INFORMATION RETRIEVAL;

EID: 36448987481     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1277741.1277891     Document Type: Conference Paper

References (15)

1. K. W. Church and W. A. Gale. Inverse document frequency (IDF): A measure of deviations from Poisson. In Proceedings of the Third Workshop on Very Large Corpora (WVLC), pages 121-130, 1995.
2. W. B. Croft and D. J. Harper. Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35(4):285-295, 1979.
3. Reprinted in Karen Spärck Jones and Peter Willett, eds., Readings in Information Retrieval, Morgan Kaufmann, pp. 339-344, 1997.
4. A. P. de Vries and T. Roelleke. Relevance information: A loss of entropy but a gain for idf? In Proceedings of SIGIR, pages 282-289, 2005.
5. H. Fang, T. Tao, and C. Zhai. A formal study of information retrieval heuristics. In Proceedings of SIGIR, pages 49-56, 2004.
6. W. R. Greiff. A theory of term weighting based on exploratory data analysis. In Proceedings of SIGIR, pages 11-19, New York, NY, USA, 1998.
7. D. Harman. The history of IDF and its influences on IR and other fields. In Charting a New Course: Natural Language Processing and Information Retrieval: Essays in Honour of Karen Spärck Jones, pages 69-79. Springer, 2005.
8. C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval, chapter 11 (Probabilistic information retrieval). Cambridge University Press, 2007. Draft of April 28.
9. K. Papineni. Why inverse document frequency? In Proceedings of the NAACL, pages 1-8, 1995.
10. S. E. Robertson. Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5):503-520, 2004.
11. S. E. Robertson and K. Spärck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3):129-146, 1976.
12. S. E. Robertson and S. Walker. On relevance weights with little relevance information. In Proceedings of SIGIR, pages 16-24, 1997.
13. K. Spärck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11-21, 1972.
14. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, second edition, 1999.
15. S. K. M. Wong and Y. Y. Yao. A note on inverse document frequency weighting scheme [sic]. Technical Report TR-89-990, Cornell University, Ithaca, NY, USA, 1989.