A comparison of techniques for estimating IDF values to generate lexical signatures for the web

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2008, Pages 39-46

  Klein, Martin ^a   Nelson, Michael L ^a  

^a ECS Building   (United States)

Author keywords

Inverse document frequency; Lexical signature

Indexed keywords

COMPARISON STUDY; DATA SETS; DOCUMENT FREQUENCY; ESTIMATION METHODS; ESTIMATION TECHNIQUES; FUTURE APPLICATIONS; INTERNET ARCHIVE; INVERSE DOCUMENT FREQUENCY; N-GRAMS; RANK CORRELATION; REAL TIME; TERM FREQUENCY; TEXTUAL CONTENT; WEB INTERFACE; WEB PAGE;

ESTIMATION; KNOWLEDGE MANAGEMENT; TECHNICAL PRESENTATIONS; TEXT PROCESSING; WORLD WIDE WEB;

FREQUENCY ESTIMATION;

EID: 77951106747     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1458502.1458510     Document Type: Conference Paper

References (24)

1. L. A. Adamic and B. A. Huberman. Zipf's Law and the Internet. Glottometrics, 3:143-150, 2002.
2. J. Bar-Ilan, M. Mat-Hassan, and M. Levene. Methods for Comparing Rankings of Search Engine Results. Computer Networks, 50(10):1448-1463, 2006.
3. W.-T. M. Chiang, M. Hagenbuchner, and A. C. Tsoi. The WT10G Dataset and the Evolution of the Web. In Proceedings of WWW '05, pages 938-939, 2005.
4. R. Fagin, R. Kumar, and D. Sivakumar. Comparing Top k Lists. In Proceedings of SODA '03, pages 28-36.
5. A. Franz and T. Brants. All Our N-Gram are Belong to You. http://googleresearch.blogspot.com/ 2006/08/all-our-n-gram-are-belong-to-you. html.
6. T. L. Harrison and M. L. Nelson. Just-in-Time Recovery of Missing Web Pages. In Proceedings of HYPERTEXT '06, pages 145-156, 2006.
7. D. Hawking. Overview of the TREC-9 Web Track. In NIST Special Publication 500-249: TREC-9, pages 87-102, 2001.
8. P. G. Ipeirotis and L. Gravano. Distributed search over the hidden web: Hierarchical database sampling and selection. In Proceedings of VLDB '02, pages 394-405. VLDB Endowment, 2002.
9. F. Keller and M. Lapata. Using the Web to Obtain Frequencies for Unseen Bigrams. Computational Linguistics, 29(3):459-484, 2003.
10. M. Klein and M. L. Nelson. Approximating Document Frequency with Term Count Values. Technical Report arXiv:0807.3755, Old Dominion University, 2008.
11. M. Klein and M. L. Nelson. Revisiting Lexical Signatures to (Re-)Discover Web Pages. In Proceedings of ECDL '08, 2008.
12. A. Kolcz, A. Chowdhury, and J. Alspector. Improved Robustness of Signature-Based Near-Replica Detection via Lexicon Randomization. In Proceedings of KDD '04, pages 605-610, 2004.
13. G. Leech, L. P. Grayson, and A. Wilson. Word Frequencies in Written and Spoken English: based on the British National Corpus. Longman, London, 2001.
14. Y. Ling, X. Meng, and W. Meng. Automated extraction of hit numbers from search result pages. In Proceedings of WAIM '06, pages 73-84, 2006.
15. F. McCown and M. L. Nelson. Agreeing to Disagree: Search Engines and their Public Interfaces. In Proceedings of JCDL '07, pages 309-318, 2007.
16. F. McCown, J. A. Smith, and M. L. Nelson. Lazy Preservation: Reconstructing Websites by Crawling the Crawlers. In Proceedings of WIDM '06, pages 67-74, 2006.
17. P. Nakov and M. Hearst. A Study of Using Search Engine Page Hits as a Proxy for n-gram Frequencies. In Proceedings of RANLP '05, 2005.
18. S.-T. Park, D. M. Pennock, C. L. Giles, and R. Krovetz. Analysis of Lexical Signatures for Improving Information Persistence on the World Wide Web. ACM Transactions on Information Systems, 22(4):540-572, 2004.
19. T. A. Phelps and R. Wilensky. Robust Hyperlinks Cost Just Five Words Each. Technical report, University of California at Berkeley, 2000.
20. J. Staddon, P. Golle, and B. Zimny. Web based inference detection. In USENIX Security Symposium, 2007.
21. K. Sugiyama, K. Hatano, M. Yoshikawa, and S. Uemura. Re nement of TF-IDF Schemes for Web Pages using their Hyperlinked Neighboring Pages. In Proceedings of HYPERTEXT '03, pages 198-207.
22. M. Theall. Methodologies for Crawler Based Web Surveys. Internet Research: Electronic Networking and Applications, 12:124-138, 2002.
23. X. Wan and J. Yang. Wordrank-based Lexical Signatures for Finding Lost or Related Web Pages. In APWeb, pages 843-849, 2006.
24. X. Zhu and R. Rosenfeld. Improving Trigram Language Modeling with the World Wide Web. In Proceedings of ICASSP '01, pages 533-536, 2001.