A pattern tree-based approach to learning URL normalization rules

Proceedings of the 19th International Conference on World Wide Web, WWW '10

Volumn , Issue , 2010, Pages 611-620

  Lei, Tao ^a   Cai, Rui ^b   Yang, Jiang Ming ^b   Ke, Yan ^c   Fan, Xiaodong ^c   Zhang, Lei ^b  

^a PEKING UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

^c MICROSOFT   (United States)

Author keywords

url deduplication; url normalization; url pattern

Indexed keywords

CANONICAL FORM; DEDUPLICATION; EFFICIENT ALGORITHM; ENGINEERING PERSPECTIVE; GLOBAL PERSPECTIVE; INDEX COMPRESSION; LEARNING PROCESS; OFFLINE; PATTERN TREES; REWRITE RULES; STATISTICAL INFORMATION; TRAINING DATA; TRAINING SAMPLE; TRAINING SETS; URL NORMALIZATION;

ALGORITHMS; COMPUTATIONAL EFFICIENCY; INFORMATION RETRIEVAL; SEARCH ENGINES; TREES (MATHEMATICS); WAVELET TRANSFORMS;

WEBSITES;

EID: 77954589296     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1772690.1772753     Document Type: Conference Paper

References (18)

1. Google webmaster central blog: specify your canonical. http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html.
2. Uniform Resource Identifier (URI): Generic Syntax. RFC 3986. http://tools.ietf.org/html/rfc3986.
3. URL Normalization. http://en.wikipedia.org/wiki/URL normalization.
4. A. Agarwal, H. S. Koppula, K. P. Leela, K. P. Chitrapura, S. Garg, and P. K. GM. URL normalization for de-duplication of web pages. In Proc. CIKM, pages 1987-1990, 2009.
5. A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Commun. ACM, 51(1):117-122, 2008.
6. D. Angluin. Finding patterns common to a set of strings. In SOTC, pages 130-141, 1979.
7. Z. Bar-Yossef, I. Keidar, and U. Schonfeld. Do not crawl in the dust: different URLs with similar text. In WWW, pages 111-120, 2007.
8. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. Computer Networks, 30(1-7):107-117, 1998.
9. A. Broder, S. C. Glassman, M. Manasse, and G. Zweig. Syntactic clustering of the Web. Computer Networks, 29(8-13):1157-1166, 1997.
10. A. C. Carvalho, E. S. Moura, A. S. Silva, K. Berlt, and A. Bezerra. A cost-effective method for detecting web site replicas on search engine databases. Data Knowl. Eng., 62(3):421-437, 2007.
11. M. Charikar. Similarity estimation techniques from rounding algorithms. In Proc. SOTC, pages 380-388, 2002.
12. A. Chowdhury, O. Frieder, D. A. Grossman, and M. C. McCabe. Collection statistics for fast duplicate document detection. TOIS, 20(2):171-191, 2002.
13. A. Dasgupta, R. Kumar, and A. Sasturkar. De-duping URLs via rewrite rules. In KDD, pages 186-194, 2008.
14. M. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In SIGIR, pages 284-291, 2006.
15. G. S. Manku, A. Jain, and A. D. Sarma. Detecting near-duplicates for web crawling. In Proc. WWW, pages 141-150, 2007.
16. M. Najork. Systems and methods for inferring uniform resource locator (URL) normalization rules. US Patent Application Publication, 2006/0218143, Microsoft Corporation, 2006.
17. M. O. Rabin. Fingerprinting by random polynomials. Technical Report TR-15-81, Center for Research in Computing Technology, Harvard University, 1981.
18. C. Xiao, W. Wang, X. Lin, and J. X. Yu. Efficient similarity joins for near duplicate detection. In Proc. WWW, pages 131-140, 2008.