Learning URL patterns for webpage de-duplication

WSDM 2010 - Proceedings of the 3rd ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2010, Pages 381-390

  Koppula, Hema Swetha ^a   Leela, Krishna P ^a   Agarwal, Amit ^b   Chitrapura, Krishna Prasad ^a   Garg, Sachin ^a   Sasturkar, Amit ^c  

^a YAHOO RESEARCH   (United States)

^b Picsquarecom   (India)

^c YAHOO INC   (United States)

Author keywords

Decision trees; Generalization; MapReduce; Page importance; Search engines; Site specific delimiters; Webpage de duplication

Indexed keywords

BUILDING BLOCKES; DELIMITERS; MACHINE LEARNING TECHNIQUES; MINE RULES; RULE EXTRACTION; SET OF RULES; SITE-SPECIFIC; TRANSFORMATION RULES; WEB SEARCHES; WEB-PAGE;

DECISION TREES; INFORMATION RETRIEVAL; INTERNET; LEARNING ALGORITHMS; MINING; SCALABILITY; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 77950949494     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1718487.1718535     Document Type: Conference Paper

References (19)

1. Hadoop: Open source implementation of mapreduce. http://lucene.apache. org/hadoop/.
2. S. Abiteboul, M. Preda, and G. Cobena. Adaptive on-line page importance computation. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 280-290, May 2003.
3. A. Agarwal, H. S. Koppula, K. P. Leela, K. P. Chitrapura, S. Garg, P. K. GM, C. Haty, A. Roy, and A. Sasturkar. Url normalization for de-duplication of web pages. In CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management, pages 1987-1990, November 2009.
4. Z. Bar-Yossef, I. Keidar, and U. Schonfeld. Do not crawl in the dust: different urls with similar text. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 111-120, May 2007.
5. E. Baykan, M. Henzinger, L. Marian, and I. Weber. Purely url-based topic classification. In WWW '09: Proceedings of the 18th international conference on World wide web, pages 1109-1110, April 2009.
6. T. Berners-Lee, L. Masinter, and M. McCahill. Uniform resource locators (url), 1994.
7. A. Broder. On the resemblance and containment of documents. In SEQUENCES '97: Proceedings of the Compression and Complexity of Sequences 1997, page 21, June 1997.
8. M. S. Charikar. Similarity estimation techniques from rounding algorithms. In STOC '02: Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 380-388, May 2002.
9. A. Dasgupta, R. Kumar, and A. Sasturkar. De-duping urls via rewrite rules. In KDD '08: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 186-194, August 2008.
10. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. In OSDI'04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, pages 10-10, December 2004.
11. D. Fetterly, M. Manasse, and M. Najork. On the evolution of clusters of near-duplicate web pages. In LA-WEB '03: Proc. of the First Conference on Latin American Web Congress, page 37, November 2003.
12. D. Gusfield. Algorithms on strings, trees, and sequences: computer science and computational biology. Cambridge University Press, New York, 1997.
13. M. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 284-291, August 2006.
14. M. Henzinger and S. Lawrence. Extracting knowledge from the world wide web. Proceedings of the National Academy of Sciences, 101:5186-5191, April 2004.
15. S. Kamvar, T. Haveliwala, C. Manning, and G. Golub. Exploiting the block structure of the web for computing pagerank. Technical report, Stanford University, 2003.
16. M.-Y. Kan and H. O. N. Thi. Fast webpage classification using url features. In CIKM '05: Proceedings of the 14th ACM international conference on Information and knowledge management, pages 325-326, November 2005.
17. G. S. Manku, A. Jain, and A. D. Sarma. Detecting near-duplicates for web crawling. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 141-150, May 2007.
18. L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, November 1999.
19. J. R. Quinlan. Induction of decision trees. Mach. Learn., 1(1):81-106, March 1986.