N-gram IDF: A global term weighting scheme based on information distance

WWW 2015 - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 960-970

  Shirakawa, Masumi ^a   Hara, Takahiro ^a   Nishio, Shojiro ^a  

^a OSAKA UNIVERSITY   (Japan)

Author keywords

IDF; Infor mation Distance; Kolmogorov Complexity; MED; Multiword Expression; Term Weighting

Indexed keywords

INVERSE PROBLEMS;

INFOR-MATION DISTANCE; INVERSE DOCUMENT FREQUENCY; KOLMOGOROV COMPLEXITY; MULTI-WORD EXPRESSIONS; SHANNON-FANO CODING; STATE-OF-THE-ART METHODS; TERM WEIGHTING; TERM WEIGHTING SCHEME;

WORLD WIDE WEB;

EID: 84968732930     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2736277.2741628     Document Type: Conference Paper

References (48)

1. M. I. Abouelhoda, S. Kurtz, and E. Ohlebusch. Replacing Suffix Trees with Enhanced Suffix Arrays. Journal of Discrete Algorithms, 2(1):53-86, Mar. 2004.
2. A. Aizawa. An Information-Theoretic Perspective of TF-IDF Measures. Information Processing and Management, 39(1):45-65, Jan. 2003.
3. C. H. Bennett, P. Gs, M. Li, P. M. Vityi, and W. H. Zurek. Information Distance. IEEE Transactions on Information Theory, 44(4):1407-1423, July 1998.
4. A. Blumer, J. Blumer, D. Haussler, R. M. McConnell, and A. Ehrenfeucht. Complete Inverted Files for Efficient Text Retrieval and Analysis. Journal of the ACM, 34(3):578-595, July 1987.
5. A. Bookstein and D. R. Swanson. Probabilistic Models for Automatic Indexing. Journal of the American Society for Information Science, 25(5):312-318, Sept./Oct. 1974.
6. G. Bouma. Normalized (Pointwise) Mutual Information in Collocation Extraction. In Proceedings of International Conference of the German Society for Computational Linguistics and Language Technology (GSCL), pages 31-40, Sep./Oct. 2009.
7. F. Bu, X. Zhu, and M. Li. Measuring the Non-compositionality of Multiword Expressions. In Proceedings of International Conference on Computational Linguistics (COLING), pages 116-124, Aug. 2010.
8. K. W. Church and W. A. Gale. Poisson Mixtures. Natural Language Engineering, 1(2):163-190, 1995.
9. K. W. Church and P. Hanks. Word Association Norms, Mutual Information, and Lexicography. Computational Linguistics, 16(1):22-29, Mar. 1990.
10. R. L. Cilibrasi and P. M. Vityi. Clustering by Compression. IEEE Transactions on Information Theory, 51(4):1523-1545, Apr. 2005.
11. R. L. Cilibrasi and P. M. Vityi. The Google Similarity Distance. IEEE Transactions on Knowledge and Data Engineering, 19(3):370-383, Mar. 2007.
12. J. F. da Silva and J. G. P. Lopes. A Local Maxima Method and a Fair Dispersion Normalization for Extracting Multi-word Units from Corpora. In Proceedings of Meeting on Mathematics of Language (MOL), pages 369-381, July 1999.
13. G. Dias. Mining Textual Associations in Text Corpora. In Proceedings of ACM SIGKDD Workshop on Text Mining, pages 20-23, Aug. 2000.
14. B. C. Fung, K. Wang, and M. Ester. Hierarchical Document Clustering Using Frequent Itemsets. In Proceedings of SIAM International Conference on Data Mining (SDM), pages 59-70, May 2003.
15. T. Gagie, G. Navarro, and S. J. Puglisi. New Algorithms on Wavelet Trees and Applications to Information Retrieval. Theoretical Computer Science, 426-427:25-41, Apr. 2012.
16. W. R. Greiff. A Theory of Term Weighting Based on Exploratory Data Analysis. In Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 11-19, Aug. 1998.
17. R. Grossi, A. Gupta, and J. S. Vitter. High-Order Entropy-Compressed Text Indexes. In Proceedings of ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 841-850, 2003.
18. S. P. Harter. A Probabilistic Approach to Automatic Keyword Indexing. Part I. On the Distribution of Specialty Words in a Technical Literature. Journal of the American Society for Information Science, 26(4):197-206, July/Aug. 1975.
19. K. S. Hasan and V. Ng. Conundrums in Unsupervised Keyphrase Extraction: Making Sense of the State-of-The-Art. In Proceedings of International Conference on Computational Linguistics (COLING), pages 365-373, Aug. 2010.
20. D. Hiemstra. A Probabilistic Justification for Using TF DF Term Weighting in Information Retrieval. International Journal on Digital Libraries, 3(2):131-139, Aug. 2000.
21. K. S. Jones. A Statistical Interpretation of Term Specificity and its Application in Retrieval. Journal of Documentation, 28:11-21, 1972.
22. A. Kolmogorov. On Tables of Random Numbers. Sankhya Ser. A, 25:369-376, 1963.
23. M. Lan, C. L. Tan, J. Su, and Y. Lu. Supervised and Traditional Term Weighting Methods for Automatic Text Categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4):721-735, Apr. 2009.
24. R. Landauer. Irreversibility and Heat Generation in the Computing Process. IBM Journal of Research and Development, 5(3):183-191, July 1961.
25. M. Li and P. M. Vityi. An Introduction to Kolmogorov Complexity and Its Applications. Springer, Berlin, 3rd edition, 2009.
26. C. D. Manning, P. Raghavan, and H. Schze. Introduction to Information Retrieval. Cambridge University Press, Cambridge, 2008.
27. D. Metzler. Generalized Inverse Document Frequency. In Proceedings of ACM Conference on Information and Knowledge Management (CIKM), pages 399-408, Oct. 2008.
28. N. Mishra, R. S. Roy, N. Ganguly, S. Laxman, and M. Choudhury. Unsupervised Query Segmentation Using only Query Logs. In Proceedings of International World Wide Web Conference (WWW), pages 91-92, Mar./Apr. 2011.
29. K. Narisawa, S. Inenaga, H. Bannai, and M. Takeda. Efficient Computation of Substring Equivalence Classes with Suffix Arrays. In Proceedings of Symposium on Combinatorial Pattern Matching (CPM), pages 340-351, July 2007.
30. D. Okanohara and J. Tsujii. Text Categorization with All Substring Features. In Proceedings of SIAM International Conference on Data Mining (SDM), pages 838-846, Apr./May 2009.
31. C. OrǍsan, V. Pekar, and L. Hasler. A Comparison of Summarisation Methods Based on Term Specificity Estimation. In Proceedings of International Conference on Language Resources and Evaluation (LREC), pages 1037-1041, May 2004.
32. K. Papineni. Why Inverse Document Frequency? In Proceedings of Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1-8, June 2001.
33. P. Pecina. An Extensive Empirical Study of Collocation Extraction Methods. In Proceedings of ACL Student Research Workshop, pages 13-18, June 2005.
34. J. W. Reed, Y. Jiao, T. E. Potok, B. A. Klump, M. T. Elmore, and A. R. Hurson. TF-ICF: A New Term Weighting Scheme for Clustering Dynamic Data Streams. In Proceedings of International Conference on Machine Learning and Applications (ICMLA), pages 258-263, Dec. 2006.
35. J. D. M. Rennie and T. Jaakkola. Using Term Informativeness for Named Entity Detection. In Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 353-360, Aug. 2005.
36. S. Robertson. Understanding Inverse Document Frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5):503-520, 2004.
37. S. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In Proceedings of Text Retrieval Conference (TREC), pages 109-126, 1994.
38. T. Roelleke and J. Wang. TF-IDF Uncovered: A Study of Theories and Probabilities. In Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 435-442, July 2008.
39. F. Rousseau and M. Vazirgiannis. Graph-of-word and TW-IDF: New Approach to Ad Hoc IR. In Proceedings of ACM Conference on Information and Knowledge Management (CIKM), pages 59-68, Oct./Nov. 2013.
40. R. S. Roy, N. Ganguly, M. Choudhury, and S. Laxman. An IR-based Evaluation Framework for Web Search Query Segmentation. In Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 881-890, Aug. 2012.
41. G. Salton and M. J. McGill. Introduction to Modern Information Retrieval. McGraw-Hill, New York, 1983.
42. G. Salton, A. Wong, and C.-S. Yang. A Vector Space Model for Automatic Indexing. Communications of the ACM, 18(11):613-620, 1975.
43. P. Schone and D. Jurafsky. Is Knowledge-Free Induction of Multiword Unit Dictionary Headwords a Solved Problem? In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 100-108, June 2001.
44. J. Sivic and A. Zisserman. Video Google: A Text Retrieval Approach to Object Matching in Videos. In Proceedings of IEEE International Conference on Computer Vision (ICCV), pages 1470-1477, Oct. 2003.
45. M. Timonen. Term Weighting in Short Documents for Document Categorization, Keyword Extraction and Query Expansion. PhD thesis, University of Helsinki, 2013.
46. H. C. Wu, R. W. P. Luk, K. F. Wong, and K. L. Kwok. Interpreting TF-IDF Term Weights as Making Relevance Decisions. ACM Transactions on Information Systems, 26(3):13:1-13:37, June 2008.
47. M. Yamamoto and K. W. Church. Using Suffix Arrays to Compute Term Frequency and Document Frequency for All Substrings in a Corpus. Computational Linguistics, 27(1):1-30, Mar. 2001.
48. W. Zhang, T. Yoshida, X. Tang, and T.-B. Ho. Improving Effectiveness of Mutual Information for Substantival Multiword Expression Extraction. Expert Systems with Applications, 36(8):10919-10930, Oct. 2009.