Efficient URL caching for world wide web crawling

Proceedings of the 12th International Conference on World Wide Web, WWW 2003

Volumn , Issue , 2003, Pages 679-689

  Broder, Andrei Z ^a   Najork, Marc ^b   Wiener, Janet L ^c  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b MICROSOFT RESEARCH   (United States)

^c HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

caching; crawling; distributed crawlers; URL caching; web crawlers; web graph models

Indexed keywords

CACHING; CRAWLING; DISTRIBUTED CRAWLER; URL CACHING; WEB CRAWLERS; WEB GRAPHS;

ALGORITHMS; HTTP; SYSTEMS ANALYSIS; WORLD WIDE WEB;

CACHE MEMORY;

EID: 84880464612     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/775152.775247     Document Type: Conference Paper

References (38)

1. N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. Journal of Computer and System Sciences, 58(1):137-147, 1999.
2. L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems Journal, 5(2):78-101, 1966.
3. K. Bharat and A. Z. Broder. A technique for measuring the relative size and overlap of public web search engines. In Proceedings of the 7th World Wide Web Conference, pages 379-388, 1998. http://www7.scu.edu.au/programme/ fullpapers/1937/com1937.htm.
4. P. Boldi, B. Codenotti, M. Santini, and S. Vigna. Trovatore: Towards a highly scalable distributed web crawler. In Poster Proceedings of the 10th International World Wide Web Conference, 2001. http://www10.org/cdrom/posters/ 1033.pdf.
5. P. Boldi, B. Codenotti, M. Santini, and S. Vigna. Ubicrawler: A scalable fully distributed web crawler. In Proceedings of the 8th Australian World Wide Web Conference, July 2002. http://ausweb.scu.edu.au/aw02/papers/refereed/vigna/ paper.html.
6. A. Borodin and R. El-Yaniv. Online computation and competitive analysis. Cambridge University Press, 1998.
7. S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. In Proceedings of the 7th World Wide Web Conference, pages 107-117, 1998. http://www7.scu.edu.au/programme/fullpapers/1921/com1921.htm.
8. A. Z. Broder. Some applications of Rabin's fingerprinting method. In R. Capocelli, A. De Santis, and U. Vaccaro, editors, Sequences II: Methods in Communications, Security, and Computer Science, pages 143-152. Springer-Verlag, 1993.
9. A. Z. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, and J. L. Wiener. Graph structure in the Web. In Proceedings of the 9th World Wide Web Conference, pages 309-320, 2000. http://www9.org/ w9cdrom/160/160.html.
10. M. Burner. Crawling towards eternity: Building an archive of the world wide web. Web Techniques Magazine, 2(5), May 1997.
11. M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. Lecture Notes in Computer Science, 2380:693-703, 2002.
12. J. Cho and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. In Proceedings of the 26th International Conference on Very Large Databases, 2000. http://rose.cs.ucla.edu/∼cho/papers/cho-evol. pdf.
13. J. Cho and H. Garcia-Molina. Parallel crawlers. In Proceedings of the 11th International World Wide Web Conference, 2002.
14. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through URL ordering. In Proceedings of the 7th International World Wide Web Conference, pages 161-172, Brisbane, 1998. http://www7.scu.edu.au/programme/fullpapers/1919/ com1919.htm.
15. F. J. Corbato. A Paging Experiment with the Multics System. Project MAC Memo MAC-M-384, Massachusetts Institute of Technology, 1968.
16. J. Edwards, K. S. McCurley, and J. A. Tomlin. An adaptive model for optimizing performance of an incremental web crawler. In Proceedings of the 10th International World Wide Web Conference, pages 106-113, May 2001.
17. D. Fetterly, M. Manasse, M. Najork, and J. L. Wiener. A large-scale study of the evolution of web pages. In Proceedings of the 12th International World Wide Web Conference, 2003.
18. P. B. Gibbons and Y. Matias. Synopsis data structures for massive data sets. In Proceedings of the 10th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 909-910. ACM Press, 1999.
19. G. H. Gonnet and R. Baeza-Yates. Handbook of algorithms and data structures: in Pascal and C. Addison-Wesley, second edition, 1991.
20. Google. http://www.google.com.
21. J. L. Hennessy and D. A. Patterson. Computer Architecture a Quantitive Approach. Morgan Kaufmann, San Francisco, third edition, 2002.
22. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219-229, 1999.
23. P. R. Jelenković. Asymptotic approximation of the move-to-front search cost distribution and least-recently used caching fault probabilities. Ann. Appl. Prob., 9(2):430-464, 1999. Available from http://comet.ctr.columbia. edu/∼predrag/mypub/mtfRevised.ps.
24. D. E. Knuth. An analysis of optimum caching. Journal of Algorithms, 6(2):181-199, 1985.
25. Reprinted as Chapter 17 of Selected Papers on Analysis of Algorithms by Donald E. Knuth, Stanford, California, Center for the Study of Language and Information, 2000.
26. D. E. Knuth. The Art of Computer Programming: Sorting and Searching. Addison-Wesley, 1973.
27. B. Krishnamurthy and J. Rexford. Web Protocols and Practice. HTTP/1.1: Networking Protocols, Caching, and Traffic Measurement. Addison-Wesley, May 2001.
28. S. Lawrence and C. L. Giles. Searching the World Wide Web. Science, 280(5360):98-100, 1998. (Pubitemid 28169089)
29. S. Lawrence and C. L. Giles. Accessibility of information on the web. Nature, 400:107-109, 1999. (Pubitemid 29327520)
30. M. Najork and A. Heydon. High-performance web crawling. SRC Research Report 173, Compaq Systems Research Center, Palo Alto, CA, Sept. 2001.
31. M. Najork and J. L. Wiener. Breadth-First Crawling yields high-quality pages. In Proceedings of the 10th International World Wide Web Conference, pages 114-118, May 2001.
32. Pew Internet and American Life Project Survey. Search engines: A Pew Internet project data memo, July 2002. http://www.pewinternet.org/reports/toc. asp?Report=64.
33. M. O. Rabin. Fingerprinting by random polynomials. Technical Report TR-15-81, Center for Research in Computing Technology, Harvard University, 1981.
34. K. Randall, R. Stata, R. Wickremesinghe, and J. L. Wiener. The Link Database: Fast Access to Graphs of the Web. In Proceedings of the Data Compression Conference, pages 122-131, April, 2002.
35. R. Sedgewick. Algorithms in C, Part 5: Graph Algorithms. Addison-Wesley, third edition, 2001.
36. V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In IEEE International Conference on Data Engineering (ICDE), Feb. 2002.
37. T. Suel. Personal communication. Jan. 2003.
38. A. S. Tanenbaum and A. S. Woodhull. Operating Systems - Design and Implementation. Prentice-Hall, second edition, 1997.