Breadth-first search crawling yields high-quality pages

Proceedings of the 10th International Conference on World Wide Web, WWW 2001

Volumn , Issue , 2001, Pages 114-118

  Najork, Marc ^a   Wiener, Janet L ^a  

^a HEWLETT PACKARD COMPANY   (United States)

Author keywords

Breadth first search; Crawl order; Crawling; Page quality metric; Pagerank

Indexed keywords

WORLD WIDE WEB;

BREADTH-FIRST SEARCH; CRAWL ORDER; CRAWLING; PAGE QUALITY; PAGERANK;

WEB CRAWLER;

EID: 85006710010     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/371920.371965     Document Type: Conference Paper

References (11)

1. K. Bharat, A. Broder, M. Henzinger, P. Kumar, S. Venkatasubramanian. The connectivity server: Fast access to linkage information on the web. In Proceedings of the 7th International World Wide Web Conference, pages 469-477, Brisbane, Australia, April 1998. Elsevier Science.
2. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. In Proceedings of the 7th International World Wide Web Conference, pages 107-117, Brisbane, Australia, April 1998. Elsevier Science.
3. M. Burner. Crawling towards eternity: Building an archive of the world wide web. Web Techniques Magazine, 2(5):37-40, May 1997.
4. J. Cho, H. Garcia-Molina, L. Page. Efficient crawling through URL ordering. In Proceedings of the 7th International World Wide Web Conference, pages 161-172, Brisbane, Australia, April 1998. Elsevier Science.
5. Google Inc. Press release: "Google launches world's largest search engine." June 26, 2000. Available at http://www.google.com/press/pressrel/pressrelease26.html
6. M. Henzinger, A. Heydon, M. Mitzenmacher, M. Najork. On near-uniform URL sampling. In Proceedings of the 9th International World Wide Web Conference, pages 295-308, Amsterdam, Netherlands, May 2000. Elsevier Science.
7. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219-229, Dec. 1999.
8. J. Kleinberg. Authoritative sources in a hyperlinked environment. In Proceedings of the 9th ACM-SIAM Symposium on Discrete Algorithms, pages 668-677, San Francisco, CA, Jan. 1998.
9. P. Lyman, H. Varian, J. Dunn, A. Strygin, K. Swearingen. How much information School of Information Management and Systems, Univ. of California at Berkeley, 2000. Available at http://www.sims.berkeley.edu/how-much-info
10. Mercator Home Page. http://www.research.digital.com/SRC/mercator
11. J. L. Wiener, R. Wickremesinghe, M. Burrows, K. Randall, R. Stata. Better link compression. Manuscript in progress. Compaq Systems Research Center, 2001.