A dynamic URL assignment method for parallel web crawler

CIMSA 2010 - IEEE International Conference on Computational Intelligence for Measurement Systems and Applications, Proceedings

Volumn , Issue , 2010, Pages 119-123

  Guerriero, A ^a   Ragni, F ^b   Martines, C ^b  

^a POLITECNICO DI BARI   (Italy)

^b Joinet Srl   (Italy)

Author keywords

Cache memories; Component; Distributed computing; Fuzzy clustering

Indexed keywords

COMPONENT; DISTRIBUTED COMPUTING; DISTRIBUTED ENVIRONMENTS; DYNAMIC CLUSTERING; GRID COMPUTING TECHNOLOGY; INTERNET SURFERS; LOAD-BALANCING; LOW COSTS; PARALLEL WEB CRAWLERS; SEARCH TERMS; URL ASSIGNMENT; WEB CRAWLERS; WEB ROBOTS; WEB SPIDERS;

ARTIFICIAL INTELLIGENCE; CLUSTER ANALYSIS; FUZZY CLUSTERING; FUZZY SYSTEMS; GRID COMPUTING; INTERNET; MEASUREMENTS;

CACHE MEMORY;

EID: 78649592869     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CIMSA.2010.5611764     Document Type: Conference Paper

References (14)

1. A. Guerriero, C. Pasquale, F. Ragni - "Java based architecture for Grid Application", Proceeding of VECIMS 2009 - IEEE International Conference on Virtual Environments, Human-Computer Interfaces, and Measurement Systems May 11-13, 2009 Hong Kong, China.
2. C. Guaragnella, A. Guerriero, C. Pasquale, F. Ragni - Jobs run-time scheduling in a Java Based Grid Architecture, in proceeding of ICIC2009 - IEEE International Conference on Intelligent Computing September 16-19, 2009 in Ulsan, Korea LNCS 5754, pp. 453-463, 2009. © Springer-Verlag Berlin Heidelberg 2009
3. A. Guerriero, C. Pasquale, F. Ragni - "Resources and Jobs Fuzzy Classification in a Java Based Grid Architecture", Proceeding of CIMSA 2009 - IEEE International Conference on Computational Intelligence for Measurement System and Applications May 11-13, 2009 Hong Kong, China.
4. Y. Wan, H. Tong. - "URL Assignment Algorithm of Crawler in Distributed System Based on Hash". - IEEE International Conference on Networking, Sensing and Control, ICNSC 2008, Hainan, China, 6-8 April 2008. pages 1632-1635, IEEE, 2008.
5. M. Marin, C. Bonacic - "Bulk-Synchronous On-Line Crawling on Clusters of Computers" - Proceedings of the 16th Euromicro Conference on Parallel, Distributed and Network-Based Processing (PDP 2008) -ISBN - ISSN:1066-6192, 978-0-7695-3089-5
6. W. Hui-chang, R. Shu-hua, T. Qi-jie.- "The Implementation of a Web Crawler URL Filter Algorithm Based on Caching" - 2009 Second International Workshop on Computer Science and Engineering, Qingdao, ChinaOctober 28-October 30- ISBN: 978-0-7695-3881-5.
7. http://tools.ietf.org/html/rfc3986 - RFC 3986/STD 66 (2005) - the current generic URI syntax specification.
8. WU Lihui, WANG Bin, and YU Zhihua, "Design and Realization of a General Web Crawler," Computer Engineering, February 2005, pp. 123-124.
9. http://www.iana.org/assignments/uri-schemes.html - Official IANA-registered schemes
10. Hu, L., Guo, D., Che, X.: A Fast Resource Selection Approach for Grid Applications Based on Fuzzy Clustering Technology. In: The 10th IEEE International Conference on High Performance Computing and Communications.
11. Applying Preference-based Fuzzy Clustering Technology to Improve Grid Resources Selection Performance - Dong Guo, Liang Hu, Shilan Jin and Bingxin Guo - Fourth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD 2007).
12. XL. Gui, QJ. Wang, WQ. Gong, DP Qiang, "Study of a Machine Selection Algorithm for Grid Computing", Journal of Computer Research and Development, Peking, 2004-12, pp. 2189-2194.
13. Hsuan-Shih Lee - "An optimal algorithm for computing the max-min transitive closure of a fuzzy similaritymatrix" - Fuzzy Sets and Systems 123 (2001) 129-136.
14. B. S. Everitt, S. Landau, M. Leese, Clustering Analysis, Oxford University Press, Oxford.