A parallel crawling schema using dynamic partition

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 3036, Issue , 2004, Pages 287-294

  Dong, Shoubin ^a   Lu, Xiaofeng ^a   Zhang, Ling ^a  

^a SOUTH CHINA UNIVERSITY OF TECHNOLOGY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

MIDDLEWARE;

DYNAMIC PARTITION; GRID MIDDLEWARE; ITS EFFICIENCIES; KEY ISSUES; LOAD BALANCE; ON DYNAMICS; PROTOTYPE SYSTEM;

SEARCH ENGINES;

EID: 35048871532     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-540-24685-5_36     Document Type: Article

References (9)

1. S. Brin and L. Page: The anatomy of a large-scale hypertextual web search engine. Computer Networks (1998) 107-117
2. P. Boldi, B. Codenotti, M. Santini, and S. Vigna: Ubicrawler: A scalable fully distributed web crawler. In: Proc. AusWeb02. The Eighth Australian World Wide Web Conference (2002)
3. D. Zeinalipour-Yazti, M. Dikaiakos: Design and Implementation of a Distributed Crawler and Filtering Processor. In: A. Halevy, A. Gal (Eds.): Proceedings of the Fifth International Workshop on Next Generation Information Technologies and Systems (NGITS'2002). Lecture Notes in Computer Science, vol. 2382. Springer (2002) 58-74
4. V. Shkapenyuk and T. Suel: Design and implementation of a high-performance distributed Web crawler. In: Proceedings of the 18th International Conference on Data Engineering (ICDE'02). San Jose, CA (2002) 357-368
5. R. L. Walker: Dynamic load balancing model: Preliminary results for parallel pseudosearch engine indexers/crawler mechanisms using MPI and genetic programming. VECPAR 2000. Porto, Portugal (2000) 61-74
6. P. Boldi, B. Codenotti, M. Santini, and S. Vigna: Trovatore: Towards a highly scalable distributed web crawler. In: Proc. of 10th International World Wide Web Conference. Hong Kong, China (2001)
7. J. Cho and H. Garcia-Molina: Parallel crawlers. In: Proc. of the 11th International World-Wide Web Conference (2002)
8. P. Andrews, T. Sherwin and B. Banister: A centralized data access model for grid computing. In: Proceeding of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSS'03). San Diego, California (2003)
9. Marc Najork and Janet L. Wiener: Breadth-first search crawling yields high quality pages. In: Proc. of 10th International World Wide Web Conference. Hong Kong, China (2001)