An information-theoretic perspective of tf-idf measures

Information Processing and Management

Volumn 39, Issue 1, 2003, Pages 45-65

  Aizawa, Akiko ^a  

^a NATIONAL INSTITUTE OF INFORMATICS   (Japan)

Author keywords

Information theory; Term weighting theories; Text categorization; tf idf

Indexed keywords

COMPUTATIONAL METHODS; INFORMATION THEORY; PROBABILITY; TEXT PROCESSING;

TERM WEIGHTING THEORIES;

INFORMATION RETRIEVAL;

EID: 0037213089     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0306-4573(02)00021-3     Document Type: Article

References (45)

1. Aizawa A. The feature quantity: an information-theoretic perspective of tfidf-like measures. Proceedings of the 23rd ACM SIGIR conference on research and development in information retrieval. 2000;104-111.
2. Aizawa A. Linguistic techniques to improve the performance of automatic text categorization. Proceedings of the sixth natural language processing Pacific rim symposium (NLPRS 2001). 2001;307-314.
3. Amati G., Van Rijsbergen K. Semantic information retrieval. Crestani F., Lalmas M., Van Rijsbergen C.J. Information retrieval: uncertainty and logics. 1998;189-219 Kluwer Academic Press, Boston.
4. Baayen R.H. Word frequency distribution. 2001;Kluwer Academic Publishers, Boston.
5. Baeza-Yates R., Ribeiro-Neto B. Modern information retrieval. 1988;Addison-Wesley, New York.
6. Brookes B.C. The Shannon model of IR systems. Journal of Documentation. 28:1972;160-162.
7. Church K.W., Gale W. Inverse document frequency (IDF): a measure of deviations from Poisson. Natural language processing using very large corpora. 1999;283-295 Kluwer Academic Press, Boston.
8. Church K.W., Hanks P. Word association norms, mutual information and lexicography. Computational Linguistics. 6(1):1990;22-29.
9. Cover T.M., Thomas J.A. Elements of information theory. 1991;John Wiley and Sons Inc, New York.
10. Crestani F. Exploiting the similarity of non-matching terms at retrieval time. Journal of Information Retrieval. 2(1):2000;23-43.
11. Croft W.B., Harper D.J. Using probabilistic models of document retrieval without relevance information. Journal of Documentation. 35(4):1979;285-295.
12. Dennis S.F. The construction of a thesaurus automatically from a sample of text. Stevens M.E., Giuliano V.E., Heilprin L.B. Statistical association methods for mechanized documentation, symposium proceedings (Miscellaneous publication 269). 1964;National Bureau of Standards, Washington, DC.
13. Dunning T. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics. 19(1):1993;61-74.
14. Fuhr N. Models for retrieval with probabilistic indexing. Information Processing and Management. 25(1):1989;55-72.
15. Fung P., McKeown K. A technical word and term translation aid using noisy parallel corpora across language groups. The Machine Translation Journal. 12(1-2):1996;53-87.
16. Grefenstette G. Explorations in automatic thesaurus discovery. 1994;Kluwer Academic Publishers, Boston.
17. Greiff W.R. A theory of term weighting based on exploratory data analysis. Proceedings of the 21st international conference on research and development in information retrieval (SIGIR'98). 1998;11-18.
18. Hiemstra D. A probabilistic justification for using. tf×idf term weighting in information retrieval International Journal on Digital Libraries. 3(2):2000;131-139.
19. Joachims T. A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. Proceedings of the 14th international conference on machine learning (ICML'97). 1997;143-151.
20. Kageura K., Umino B. Methods of automatic term recognition: a review. Terminology. 3(2):1996;259-289.
21. Kageura K., Yoshioka M., Tsujii K., Yoshikane F., Takeuchi K., Koyama T. Evaluation of the term recognition task. Proceedings of the 1st NTCIR workshop on research in Japanese text retrieval and term recognition. 1999;42-49.
22. Kita K. Probabilistic language models. 1999;University of Tokyo Press, Tokyo.
23. Koller D., Sahami M. Toward optimal feature selection. Proceedings of the international conference on machine learning (ICML'96). 1996;284-292.
24. Koller D., Sahami M. Hierarchically classifying documents using very few words. Proceedings of the international conference on machine learning (ICML'97). 1997;170-178.
25. Lewis D.D., Ringuette M. Comparison of two learning algorithms for text categorization. Proceedings of the 3rd annual symposium on document analysis and information retrieval (SDAIR'94). 1994;81-93.
26. Luhn H.P. A statistical approach to mechanized encoding and searching of literary information. IBM Journal of Research and Development. 1(4):1957;309-317.
27. Manning C.D., Schtüze H. Foundations of statistical natural language processing. 1999;MIT Press, Cambridge.
28. Matsumoto Y., Kitauchi A., Yamashita T., Hirano Y., Matsuda K., Asahara M. Morphological analysis system ChaSen 2.0.2 users manual. NAIST Technical Report, NAIST-IS-TR99012. 1999;Nara Institute of Science and Technology.
29. McCallum A., Nigam K. A comparison of event models for naive Bayes text classification. Learning for text categorization, Technical Report WS-98-05. (pp. 41-48):1998;The AAAI Press, Menlo Park.
30. Mladenić D. Feature subset selection in text-learning. Proceedings of the 10th European conference on machine learning (ECML'98). 1998;95-100.
31. Nagao M., Mizutani M., Ikeda H. An automated method for the extraction of important words from Japanese scientific documents. Transactions of Information Processing Society of Japan. 17(2):1976;110-117.
32. National Center for Science Information Systems (1999). NTCIR workshop 1 - Proceedings of the first NTCIR workshop on research in Japanese text retrieval and term recognition. http://research.nii.ac.jp/̃ntcadm/workshop/OnlineProceedings/.
33. Robertson S.E. On term selection for query expansion. Journal of Documentation. 46(4):1990;359-364.
34. Robertson S.E. Query-document symmetry and dual models. Journal of Documentation. 50(3):1994;233-238.
35. Robertson S.E., Sparck-Jones K. Relevance weighting of search terms. Journal of the American Society of Information Science. 27:1976;129-146.
36. Salton G., Buckley C. Weighting approaches in automatic text retrieval. Information Processing and Management. 24(5):1988;513-523.
37. Salton G., McGill M.J. Introduction to modern information retrieval. 1983;McGraw-Hill, New York.
38. Slonim N., Tishby N. Document clustering using word clusters via the information bottleneck method. Proceedings of the 23rd international conference on research and development in information retrieval (SIGIR 2000). 2000;208-215.
39. Smadja F. Retrieving collocations from text: Xtract. Computational Linguistics. 19(1):1993;143-178.
40. Sparck-Jones K. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation. 28(1):1972;11-21.
41. Van Rijsbergen C.J., Happer D.J., Porter M.F. The selection of good search terms. Information Processing and Management. 17:1981;77-91.
42. Wiener E., Pedersen J.O., Weighend A.S. A neural network approach to topic spotting. Proceedings of the 4th annual symposium on document analysis and information retrieval (SDAIR'95). 1995;317-332.
43. Wong S.K.M., Yao Y.Y. An information-theoretic measure of term specificity. Journal of the American Society for Information Science. 43(1):1992;54-61.
44. Yang Y., Liu X. A re-examination of text categorization methods. Proceedings of the 22nd international conference on research and development in information retrieval (SIGIR'99). 1999;42-49.
45. Yang Y., Pedersen O. A comparative study on feature selection in text categorization. Proceedings of the 14th international conference on machine learning (ICML'97). 1997;412-420.