Identifying equivalent URLs using URL signatures

SITIS 2008 - Proceedings of the 4th International Conference on Signal Image Technology and Internet Based Systems

Volumn , Issue , 2008, Pages 203-210

  Soon, Lay Ki ^a   Lee, Sang Ho ^a  

^a Soongsil University   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION SERVICES; MARKUP LANGUAGES; METADATA; STANDARDS; WEBSITES; WORLD WIDE WEB;

URL NORMALIZATIONS; WEB INFORMATION RETRIEVALS; WEB PAGES;

INTERNET;

EID: 60349095201     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SITIS.2008.21     Document Type: Conference Paper

References (11)

1. Bar-Yossef, Z., Keidar, I., Schonfeld, U., "Do Not Crawl in the DUST: Different URLs with Similar Text", in the Proceedings of the International World Wide Web Conference (WWW 2007), May 2007, pp. 111 - 120.
2. Berners-Lee, T., Fielding, R., Masinter, L., "Uniform Resource Identifier (URI): General Syntax", available at http://gbiv.com/protocols/ uri/rfc/rfc3986.html.
3. Chakrabarti, S.: Mining the Web, Discovering Knowledge from Hypertext Data, Morgan Kaufmann Publishers, Elservier, San Francisco, CA, 2003.
4. Han, J., Kamber, M.: Data Mining Concepts and Techniques, Morgan Kaufmann Publishers, Elsevier, San Francisco, CA, 2006.
5. Kim, S. J., Jeong, H. S., and Lee, S. H., "Reliable Evaluations of URL Normalization", in Proceedings of the 2006 International Conference on Computational Science and its Applications (ICCSA), Glasgow, May 2006, pp. 609-617.
6. Kim, S. J. and Lee, S. H., "Implementation of a Web Root and Statistics on the Korean Web", in Proceedings of the Second International Conference on Human.Society@Internet (HIS), Seoul, June 2003, pp. 341 - 350.
7. Lee, S. H., Kim, S. J., Hong, S. H., "On URL Normalization", in Proceedings of the 2005 International Conference on Computational Science and its Applications (ICCSA), Singapore, May 2005, pp. 1076-1085.
8. Netcraft June 2008 Web Server Survey, available at: http://news.netcraft. com/archives/web-server-survey.html
9. Pant, G., Srinivasan, P., Menczer, F., "Crawling the Web", Web Dynamics 2004, pp. 153 - 178.
10. The MD5 Message-Digest Algorithm, available at: http://tools.ietf.org/ html/rfc1321
11. Web Data Extractor, available at: http://www.webextractor.com/