URL normalization for de-duplication of web pages

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2009, Pages 1987-1990

  Agarwal, Amit ^a,b   Koppula, Hema Swetha ^a   Leela, Krishna P ^a   Chitrapura, Krishna Prasad ^a   Garg, Sachin ^a   Pavan Kumar, G M ^a  

^a YAHOO RESEARCH   (United States)

^b Infibeam   (India)

Author keywords

Decision tree; Page importance; Search engines; URL de duplication

Indexed keywords

BUILDING BLOCKES; EXPERIMENTAL EVALUATION; MACHINE LEARNING TECHNIQUES; MINE RULES; PAGE IMPORTANCE; RULE EXTRACTION; SET OF RULES; URL NORMALIZATION; WEB PAGE; WEB SEARCHES;

DECISION TREES; INFORMATION RETRIEVAL; KNOWLEDGE MANAGEMENT; LEARNING ALGORITHMS; MINING; SEARCH ENGINES; WEBSITES;

INTERNET;

EID: 74549172900     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1645953.1646283     Document Type: Conference Paper

References (12)

1. S. Abiteboul, M. Preda, and G. Cobena. Adaptive on-line page importance computation. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 280-290, May 2003.
2. Z. Bar-Yossef, I. Keidar, and U. Schonfeld. Do not crawl in the dust: different urls with similar text. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 111-120, May 2007.
3. T. Berners-Lee, L. Masinter, and M. McCahill. Uniform resource locators (url), 1994.
4. A. Broder. On the resemblance and containment of documents. In SEQUENCES '97: Proceedings of the Compression and Complexity of Sequences 1997, page 21, June 1997.
5. A. Dasgupta, R. Kumar, and A. Sasturkar. De-duping urls via rewrite rules. In KDD '08: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 186-194, August 2008.
6. D. Fetterly, M. Manasse, and M. Najork. On the evolution of clusters of near-duplicate web pages. In LA-WEB '03: Proceedings of the First Conference on Latin American Web Congress, page 37, November 2003.
7. D. Gusfield. Algorithms on strings, trees, and sequences: computer science and computational biology. Cambridge University Press, New York, 1997.
8. M. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 284{291, August 2006.
9. S. Kamvar, T. Haveliwala, C. Manning, and G. Golub. Exploiting the block structure of the web for computing pagerank. Technical report, Stanford University, 2003.
10. G. S. Manku, A. Jain, and A. D. Sarma. Detecting near-duplicates for web crawling. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 141{150, May 2007.
11. L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, November 1999.
12. J. R. Quinlan. Induction of decision trees. Mach. Learn., 1(1):81{106, March 1986.