On the Feasibility of Geographically Distributed Web Crawling

ACM International Conference Proceeding Series

Volumn 2008-June, Issue , 2008, Pages

  Cambazoglu, B Barla ^a   Junqueira, Flavio ^a   Plachouras, Vassilis ^a   Telloli, Luca ^a  

^a YAHOO RESEARCH   (Spain)

Author keywords

Distributed Web crawling; spatial locality; throughput

Indexed keywords

BENEFIT AND CHALLENGES; DISTRIBUTED CRAWLER; DISTRIBUTED WEB CRAWLING; GEOGRAPHICAL LOCALITY; NETWORK ACCESS; NETWORK COSTS; PERFORMANCE; PORTUGAL; SPATIAL LOCALITY; WEB-SITES;

WEB CRAWLER;

EID: 84990920578     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.4108/ICST.INFOSCALE2008.3550     Document Type: Conference Paper

References (33)

1. S. Abiteboul, M. Preda, and G. Cobena. Adaptive on-line page importance computation. In Proceedings of the 12th International World Wide Web Conference, pages 280–290, 2003.
2. R. Baeza-Yates, C. Castillo, F. Junqueira, V. Plachouras, and F. Silvestri. Challenges in distributed information retrieval. In Proceedings of the 23rd International Conference on Data Engineering, pages 6–20, 2007.
3. R. Baeza-Yates, C. Castillo, M. Marin, and A. Rodriguez. Crawling a country: better strategies than breadth-first for web page ordering. In Special Interest Tracks and Posters of the 14th International World Wide Web Conference, 2005.
4. P. Boldi, B. Codenotti, M. Santini, and S. Vigna. Ubicrawler: a scalable fully distributed web crawler. Software Practice & Experience, 34(8):711–726, 2004.
5. B. E. Brewington and G. Cybenko. How dynamic is the web? Computer Networks: The International Journal of Computer and Telecommunications Networking, 33(1-6):257–276, 2000.
6. B. B. Cambazoglu, A. Turk, and C. Aykanat. Data-parallel web crawling models. Lecture Notes in Computer Science, 3280:801–809, 2004.
7. B. B. Cambazoglu, A. Turk, E. Karaca, C. Aykanat, and T. Kucukyilmaz. Architecture of a grid-enabled search engine. Information Processing & Management, 43(3):609–623, 2007.
8. J. Cho and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. In Proceedings of the 26th International Conference on Very Large Data Bases, pages 200–209, 2000.
9. J. Cho and H. Garcia-Molina. Synchronizing a database to improve freshness. In Proceedings of ACM International Conference on Management of Data, pages 117–128, 2000.
10. J. Cho and H. Garcia-Molina. Parallel crawlers. In Proceedings of the 11th International World Wide Web Conference, pages 124–135, 2002.
11. J. Cho and H. Garcia-Molina. Estimating frequency of change. ACM Transactions on Internet Technology, 3(3):256–290, 2003.
12. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through url ordering. Computer Networks and ISDN Systems, 30(1-7):161–172, 1998.
13. C. Chung and C. Clarke. Topic-oriented collaborative crawling. In Proceedings of the 11th International Conference on Information and Knowledge Management, pages 34–42, 2002.
14. D. Eichmann. Ethical web agents. In Second International Conference on World Wide Web, pages 3–13, 1994.
15. D. Eichmann. The rbse spider – balancing effective search against web load. In First International Conference on World Wide Web, pages 113–120, 1994.
16. J. Exposto, J. Macedo, A. Pina, A. Alves, and J. Rufino. Geographical partition for distributed web crawling. In Proceedings of the 2005 Workshop on Geographic Information Retrieval, 2005.
17. J. Exposto, J. Macedo, A. Pina, A. Alves, and J. Rufino. Efficient partitioning strategies for distributed web crawling. In Proceedings of the 21st International Conference on Information Networking, 2007.
18. Z. Gyongyi and H. Garcia-Molina. Link spam alliances. In Proceedings of the 31st international Conference on Very Large Data Bases, pages 517–528, 2005.
19. Z. Gyongyi and H. Garcia-Molina. Web spam taxonomy. In First International Workshop on Adversarial Information Retrieval on the Web, 2005.
20. Z. Gyongyi, H. Garcia-Molina, and J. Pedersen. Combating web spam with trustrank. In Proceedings of the 30th International Conference on Very Large Databases, pages 576–587, 2004.
21. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219–229, 1999.
22. B. Huffaker, M. Fomenkov, D. J. Plummer, D. Moore, and K. Claffy. Distance metrics in the internet. In Proceedings of the International Telecommunications Symposium, 2002.
23. M. Koster. A standard for robot exclusion. http://www.robotstxt.org/wc/norobots.html, 1994.
24. S. Lawrence and C. L. Giles. Accessibility of information on the web. Intelligence, 11(1):32–39, 2000.
25. H.-T. Lee, D. Leonard, X. Wang, and D. Loguinov. Irlbot: Scaling to 6 billion pages and beyond. In Proceedings of the 17th International World Wide Web Conference, pages 427–436, 2008.
26. M. Najork and J. L. Wiener. Breadth-first crawling yields high-quality pages. In Proceedings of the 10th International World Wide Web Conference, pages 114–118, 2001.
27. C. Olston and S. Pandey. Recrawl scheduling based on information longevity. In Proceedings of the 17th International World Wide Web Conference, pages 437–446, 2008.
28. G. Pant, S. Bradshaw, and F. Menczer. Search engine-crawler symbiosis: Adapting to community interests. In 7th European Conference on Research and Advanced Technology for Digital Libraries, pages 221–232, 2003.
29. G. Pant, P. Srinivasan, and F. Menczer. Crawling the web. In Web Dynamics: Adapting to Change in Content, Size, Topology and Use, edited by M. Levene and A. Poulovassilis, pages 153–178, 2004.
30. V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In Proceedings of the 18th International Conference on Data Engineering, pages 357–368, 2002.
31. S. Teng, Q. Lu, M. Eichstaedt, D. Ford, and T. Lehman. Collaborative web crawling: Information gathering/processing over internet. In 32nd Hawaii International Conference on System Sciences, 1999.
32. P. Yalagandula, S. Nath, H. Yu, P. B. Gibbons, and S. Seshan. Beyond availability: Towards a deeper understanding of machine failure characteristics in large distributed systems. In Proceedings of the 1st Workshop on Real Large Distributed Systems, 2004.
33. D. Zeinalipour-Yazti and M. D. Dikaiakos. Design and implementation of a distributed crawler and filtering processor. In Proceedings of the Next Generation Information Technologies and Systems, pages 58–74, 2002.