Parallel crawlers

Proceedings of the 11th International Conference on World Wide Web, WWW '02

Volumn , Issue , 2002, Pages 124-135

  Cho, Junghoo ^a   Garcia Molina, Hector ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

Parallelization; Web crawler; Web spider

Indexed keywords

CAN DESIGN; PARALLELIZATIONS; PROPOSED ARCHITECTURES; WEB CRAWLERS; WEB SPIDERS;

WORLD WIDE WEB;

EID: 67649866504     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/511446.511464     Document Type: Conference Paper

References (26)

1. T. E. Anderson, M. D. Dahlin, J. M. Neefe, D. A. Patterson, D. S. Roselli, and R. Y. Wang. Serverless network file systems. In Proc. of SOSP, 1995.
2. A. Barabasi and R. Albert. Emergence of scaling in random networks. Science, 286(509), 1999.
3. A. Z. Broder, S. R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, and J. L. Wiener. Graph structure in the web. In Proc. of WWW Conf., 2000.
4. M. Burner. Crawling towards eterneity: Building an archive of the world wide web. Web Techniques Magazine, 2(5), May 1998.
5. S. Chakrabarti, M. van den Berg, and B. Dom. Focused crawling: A new approach to topic-specific web resource discovery. In Proc. of WWW Conf., 1999.
6. J. Cho and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. In Proc. of VLDB Conf., 2000.
7. J. Cho and H. Garcia-Molina. Synchronizing a database to improve freshness. In Proc. of SIGMOD Conf., 2000.
8. J. Cho and H. Garcia-Molina. Parallel crawlers. Technical report, UCLA Computer Science, 2002.
9. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through URL ordering. In Proc. of WWW Conf., 1998.
10. E. Coffman, Jr., Z. Liu, and R. R. Weber. Optimal robot scheduling for web search engines. Technical report, INRIA, 1997.
11. M. Diligenti, F. M. Coetzee, S. Lawrence, C. L. Giles, and M. Gori. Focused crawling using context graphs. In Proc. of VLDB Conf., 2000.
12. D. Eichmann. The RBSE spider: Balancing effective search against web load. In Proc. of WWW Conf., 1994.
13. Google Inc. http://www.google.com.
14. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. Word Wide Web, 2(4):219-229, December 1999.
15. A. Heydon and M. Najork. High-performance web crawling. Technical report, SRC Research Report, 173, Compaq Systems Research Center, September 2001.
16. D. Hirschberg. Parallel algorithms for the transitive closure and the connected component problem. In Proc. of STOC Conf., 1976.
17. M. Koster. Robots in the web: threat or treat? ConneXions, 4(4), April 1995.
18. O. A. McBryan. GENVL and WWWW: Tools for taming the web. In Proc. of WWW Conf., 1994.
19. R. C. Miller and K. Bharat. SPHINX: a framework for creating personal, site-specific web crawlers. In Proc. of WWW Conf., 1998.
20. D. Nassimi and S. Sahni. Parallel permutation and sorting algorithms and a new generalized connection network. Journal of ACM, 29:642-667, July 1982.
21. M. T. Ozsu and P. Valduriez. Principles of Distributed Database Systems. Prentice Hall, 1999.
22. L. Page and S. Brin. The anatomy of a large-scale hypertextual web search engine. In Proc. of WWW Conf., 1998.
23. B. Pinkerton. Finding what people want: Experiences with the web crawler. In Proc. of WWW Conf., 1994.
24. Robots exclusion protocol. http://info.webcrawler. com/mak/projects/ robots/exclusion.html.
25. A. S. Tanenbaum and R. V. Renesse. Distributed operating systems. ACM Computing Surveys, 17(4), December 1985.
26. G. K. Zipf. Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology. Addison-Wesley, 1949.