Do not crawl in the dust: Different urls with similar text

16th International World Wide Web Conference, WWW2007

Volumn , Issue , 2007, Pages 111-120

  Bar Yossef, Ziv ^a,b   Keidar, Idit ^a   Schonfeld, Uri ^c  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^b GOOGLE INC   (United States)

^c Engineering IV   (United States)

Author keywords

Anti aliasing; Crawling; Duplicate detection; Search engines; URL normalization

Indexed keywords

INFORMATION ANALYSIS; PROBLEM SOLVING; SEARCH ENGINES; SERVERS; STATISTICAL METHODS; WEBSITES;

DUPLICATE DETECTION; URL NORMALIZATION; WEB SERVER LOGS;

TEXT PROCESSING;

EID: 35348921241     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1242572.1242588     Document Type: Conference Paper

References (22)

1. R. Agrawal and R. Srikant. Fast algorithms for mining association rules. In Proc. 20th VLDB, pages 487-499, 1994.
2. Z. Bar-Yossef, I. Keidar, and U. Schonfeld. Do not crawl in the DUST: different URLs with similar text. Technical Report CCIT Report #601, Dept. Electrical Engineering, Technion, 2006.
3. K. Bharat and A. Z. Broder. Mirror, Mirror on the Web: A Study of Host Pairs with Replicated Content. Computer Networks, 31(11-16):1579-1590, 1999.
4. K. Bharat, A. Z. Broder, J. Dean, and M. R. Henzinger. A comparison of techniques to find mirrored hosts on the WWW. IEEE Data Engin. Bull., 23(4):21-26, 2000.
5. M. Bognar. A survey on abstract rewriting. Available online at: www.di.ubi.pt/~desousa/1998-1999/logica/mb.ps, 1995.
6. S. Brin, J. Davis, and H. Garcia-Molina. Copy Detection Mechanisms for Digital Documents. In Proc. 14th SIGMOD, pages 398-409, 1995.
7. A. Z. Broder, S. C. Classman, and M. S. Manasse. Syntactic clustering of the web. In Proc. 6th WWW, pages 1157-1166, 1997.
8. J. Cho, N. Shivakumar, and H. Garcia-Molina. Finding replicated web collections. In Proc. 19th SIGMOD, pages 355-366, 2000.
9. E. Di Iorio, M. Diligenti, M. Gori, M. Maggini, and A. Pucci. Detecting Near-replicas on the Web by Content and Hyperlink Analysis. In Proc. 11th WWW, 2003.
10. F. Douglis, A. Feldman, B. Krishnamurthy, and J. Mogul. Rate of change and other metrics: a live study of the world wide web. In Proc. 1st USITS, 1997.
11. H. Garcia-Molina, L. Gravano, and N. Shivakumar. dscam: Finding document copies across multiple databases. In Proc. 4th PDIS, pages 68-79, 1996.
12. M. R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, 1979.
13. Google Inc. Google sitemaps. http://sitemaps.google.com.
14. D. Gusfleld. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambridge University Press, 1997.
15. T. C. Hoad and J. Zobel. Methods for identifying versioned and plagiarized documents. J. Amer. Soc. Infor. Sci. Tech., 54(3):203-215, 2003.
16. N. Jain, M. Dahlin, and R. Tewari. Using bloom filters to refine web search results. In Proc. 7th WebDB, pages 25-30, 2005.
17. T. Kelly and J. C. Mogul. Aliasing on the world wide web: prevalence and performance implications. In Proc. 11th WWW, pages 281-292, 2002.
18. S. J. Kim, H. S. Jeong, and S. H. Lee. Reliable evaluations of URL normalization. In Proc. 4th ICCSA, pages 609-617, 2006.
19. H. Liang. A URL-String-Based Algorithm for Finding WWW Mirror Host. Master's thesis, Auburn University, 2001.
20. F. McCown and M. L. Nelson. Evaluation of crawling policies for a web-repository crawler. In Proc. 17th HYPERTEXT, pages 157-168, 2006.
21. U. Schonfeld, Z. Bar-Yossef, and I. Keidar. Do not crawl in the DUST: different URLs with similar text. In Proc. 15th WWW, pages 1015-1016, 2006.
22. N. Shivakumar and H. Garcia-Molina. Finding Near-Replicas of Documents and Servers on the Web. In Proc. 1st WebDB, pages 204-212, 1998.