Design and implementation of a high-performance distributed web crawler

Proceedings - International Conference on Data Engineering

Volumn , Issue , 2002, Pages 357-368

  Shkapenyuk, Vladislav ^a   Suel, Torsten ^a  

^a POLYTECHNIC UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER OPERATING SYSTEMS; COMPUTER SIMULATION; SCHEDULING; WORLD WIDE WEB;

DISTRIBUTED WEB CRAWLERS;

SEARCH ENGINES;

EID: 0036204395     PISSN: 10844627     EISSN: None     Source Type: Journal    
DOI: 10.1109/ICDE.2002.994750     Document Type: Article

References (26)

1. Searching the web
2. Approximating aggregate queries about web pages via random walks
3. The connectivity server: Fast access to linkage information on the web
4. The anatomy of a large-scale hypertextual web search engine
5. Crawling towards eternity: Building an archive of the world wide web
6. Mining the web’s link structure
7. Distributed hypertext resource discovery through examples
8. Focused crawling: A new approach to topic-specific web resource discovery
9. The evolution of the web and implications for an incremental crawler
10. Synchronizing a database to improve freshness
11. Efficient crawling through url ordering
12. Focused crawling using context graphs
13. Measuring index quality using random walks on the web
14. On near-uniform URL sampling
15. Mercator: A scalable, extensible web crawler
16. WebBase: A repository of web pages
17. Archiving the Internet
18. Derandomizing algorithms for routing and sorting on meshes
19. Breadth-first search crawling yields high-quality pages
20. Crawling the hidden web
21. Using reinforcement learning to spider the web efficiently
22. Compressing the graph structure of the web
23. Controlling robots of web search engines