De-duping URLs via rewrite rules

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn , Issue , 2008, Pages 186-195

  Dasgupta, Anirban ^a   Kumar, Ravi ^a   Sasturkar, Amit ^a  

^a YAHOO INC   (United States)

Author keywords

De duping; Rewrite rules; URL normalization

Indexed keywords

CANONICAL FORMS; DE-DUPING; EFFICIENT ALGORITHMS; LARGE-SCALE EXPERIMENTS; PRINCIPAL FUNCTIONS; REWRITE RULES; TRANSFORMATION RULES; URL NORMALIZATION;

EDUCATION; INFORMATION MANAGEMENT; INFORMATION RETRIEVAL; INTERNET; LEARNING ALGORITHMS; MINING; SEARCH ENGINES; SET THEORY; WORLD WIDE WEB;

EQUIVALENCE CLASSES;

EID: 65449167674     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1401890.1401917     Document Type: Conference Paper

References (17)

1. R. Ananthakrisha, S. Chaudhuri, and V. Ganti. Eliminating fuzzy duplicates in data warehouse. In Proc. 38th VLDB, pages 586-597, 2002.
2. D. Angluin. Finding patterns common to a set of strings. In Proc. 11th STOC, pages 130-141, 1979.
3. D. Angluin. Inference of reversible languages. J. ACM, 29(3):741-765, 1982.
4. D. Angluin and C. H. Smith. Inductive inference: Theory and methods. ACM Comput. Surv., 15(3):237-269, 1983.
5. Z. Bar-Yossef, I. Keidar, and U. Schonfeld. Do not crawl in the DUST: Different URLs with similar text. In Proc. 16th WWW, pages 111-120, 2007.
6. M. Bognar. A survey of abstract rewriting, 1995. www.di.ubi.pt/~desousa/ 1998-1999/logica/mb.ps.
7. A. Broder. On the resemblance and containment of documents. In SEQS: Sequences '91, 1998.
8. A. Broder, S. C. Glassman, M. Manasse, and G. Zweig. Syntactic clustering of the web. Computer Networks, 29(8-13):1157-1166, 1997.
9. M. Charikar. Similarity estimation techniques from rounding algorithms. In Proc. 34th STOC, pages 380-388, 2002.
10. S. Chaudhuri, V. Ganti, and R. Motwani. Robust idenfication of fuzzy duplicates. In Proc. 21st ICDE, pages 865-876, 2005.
11. Z. Chen, D. V. Kalashnikov, and S. Mehrotra. Adaptive graphical approach to entity resolution. In Proc. ACM/IEEE JCDL, pages 204-213, 2007.
12. D. Fetterly, M. Manasse, and M. Najork. On the evolution of clusters of near-duplicate web pages. In Proc. 1st Conf. on Latin American Web Congress, page 37, 2003.
13. H. Garcia-Molina. Pair-wise entity resolution: Overview and challenges. In Proc. CIKM, 2006.
14. M. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In Proc. 29th SIGIR, pages 284-291, 2006.
15. G. S. Manku, A. Jain, and A. D. Sarma. Detecting near-duplicates for web crawling. In Proc. 16th WWW, pages 141-150, 2007.
16. M. Najork. Systems and methods for inferring uniform resource locator (URL) normalization rules, 2006. US Patent Application Publication, 2006/0218143.
17. A. Pereira, R. Baeza-Yates, and N. Ziviani. Where and how duplicates occur in the web. In Proc. 4th Latin American Web Congress, pages 127-134, 2006.