Scheduling algorithms for Web crawling

Proceedings - WebMedia and LA-Web 2004

Volumn , Issue , 2004, Pages 10-17

  Castillo, Carlos ^a   Marin, Mauricio ^b   Baeza Yates, Ricardo ^a   Rodriguez, Andrea ^c  

^a UNIVERSITY OF CHILE   (Chile)

^b UNIVERSIDAD DE MAGALLANES   (Chile)

^c UNIVERSITY OF CONCEPCIÓN   (Chile)

Author keywords

[No Author keywords available]

Indexed keywords

DOWNLOADING; FRAMEWORKS; SCHEDULING ALGORITHMS; WEB CRAWLERS;

ALGORITHMS; BANDWIDTH; COMPUTER SIMULATION; LARGE SCALE SYSTEMS; PROBLEM SOLVING; SCHEDULING; SEARCH ENGINES; WEBSITES;

WORLD WIDE WEB;

EID: 15844394068     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WEBMED.2004.1348139     Document Type: Conference Paper

References (44)

1. Robotcop. www.robotcop.org, 2002.
2. HT://Dig. http://www.htdig.org/, 2004. GPL software.
3. S. Ailleret. Larbin, http://larbin.sourceforge.net/index-eng.html, 2004. GPL software.
4. R. Baeza-Yates and C. Castillo. Relating web characteristics with link based web page ranking. In Proceedings of String Processing and Information Retrieval, pages 21-32, Laguna San Rafael, Chile, November 2001. IEEE Cs. Press.
5. R. Baeza-Yates and C. Castillo. Balancing volume, quality and freshness in web crawling. In Soft Computing Systems - Design, Management and Applications, pages 565-572. IOS Press, 2002.
6. O. Brandman, J. Cho, H. Garcia-Molina, and N. Shivakumar. Crawler-friendly web servers. In Proceedings of the Workshop on Performance and Architecture of Web Servers (PAWS), Santa Clara, California, USA, June 2000.
7. B. Brewington, G. Cybenko, R. Stata, K. Bharat, and F. Maghoul. How dynamic is the web? In Proceedings of the Ninth Conference on World Wide Web, pages 257-276, Amsterdam, Netherlands, May 2000.
8. S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1-7): 107-117, April 1998.
9. A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, and J. Wiener. Graph structure in the web: Experiments and models. In Proceedings of the Ninth Conference on World Wide Web, pages 309-320, Amsterdam, Netherlands, May 2000.
10. R. D. Burke. Salticus: guided crawling for personal digital libraries. In Proceedings of the first ACM/IEEE-CS joint conference on Digital Libraries, pages 88-89, Roanoke, Virginia, June 2001.
11. M. Burner. Crawling towards eternity - building an archive of the world wide web. Web Techniques, 2(5), May 1997.
12. S. Chakrabarti. Mining the Web. Morgan Kaufmann, 2003.
13. J. Cho. The evolution of the web and implications for an incremental crawler. In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt, September 2000. Morgan Kaufmann.
14. J. Cho and R. Adams. Page quality: In search of an unbiased Web ranking. Technical report, UCLA Computer Science, 2004.
15. J. Cho and H. Garcia-Molina. Synchronizing a database to improve freshness. In Proceedings of ACM International Conference on Management of Data (SIGMOD), pages 117-128, Dallas, Texas, USA, May 2000.
16. J. Cho and H. Garcia-Molina. Parallel crawlers. In Proceedings of the eleventh international conference on World Wide Web, pages 124-135, Honolulu, Hawaii, USA, May 2002. ACM Press.
17. J. Cho, H. García-Molina, and L. Page. Efficient crawling through URL ordering. In Proceedings of the seventh conference on World Wide Web, Brisbane, Australia, April 1998.
18. J. Cho, N. Shivakumar, and H. Garcia-Molina. Finding replicated web collections. In ACM SIGMOD, pages 355-366, 1999.
19. A. Czumaj, I. Finch, L. Gasieniec, A. Gibbons, P. Leng, W. Rytter, and M. Zito. Efficient Web searching using temporal factors. Theoretical Computer Science, 262(1-2):569-582, 2001.
20. A. S. da Silva, E. A. Veloso, P. B. Golgher, B. A. Ribeiro-Neto, A. H. F. Laender, and N. Ziviani. Cobweb - a crawler for the brazilian web. In Proceedings of String Processing and Information Retrieval (SPIRE), pages 184-191, Cancun, Mexico, September 1999. IEEE Cs. Press.
21. L. Dacharay. WebBase. http://freesoftware.fsf.org/webbase/, 2002. GPL Software.
22. M. Diligenti, F. Coetzee, S. Lawrence, C. L. Giles, and M. Gori. Focused crawling using context graphs. In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt, September 2000.
23. F. Douglis, A. Feldmann, B. Krishnamurthy, and J. C. Mogul. Rate of change and other metrics: a live study of the world wide web. In USENIX Symposium on Internet Technologies and Systems, pages 147-158, Monterey, California, USA, December 1997.
24. R. W. Edward G. Coffman, Z. Liu. Optimal robot scheduling for web search engines. Journal of Scheduling, 1(1): 15-29, 1998.
25. J. Edwards, K. S. McCurley, and J. A. Tomlin. An adaptive model for optimizing performance of an incremental web crawler. In Proceedings of the Tenth Conference on World Wide Web, pages 106-113, Hong Kong, May 2001. Elsevier.
26. D. Eichmann. The RBSE spider: balancing effective search against web load. In Proceedings of the first World Wide Web Conference, Geneva, Switzerland, May 1994.
27. R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. RFC 2616 - HTTP/1.1, the hypertext transfer protocol. http://w3.org/Protocols/rfc2616/rfc2616.html, 1999.
28. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web Conference, 2(4):219-229, April 1999.
29. M. Koster. Robots in the web: threat or treat? Connexions, 9(4), April 1995.
30. R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Trawling the Web for emerging cyber-communities. Computer Networks, 31(11-16):1481-1493, 1999.
31. S. Lawrence and C. L. Giles. Searching the World Wide Web. Science, 280(5360):98-100, 1998.
32. L. Lim, M. Wang, S. Padmanabhan, J. S. Vitter, and R. Agarwal. Characterizing Web document change. In Proceedings of the Second International Conference on Advances in Web-Age Information Management, volume 2118 of Lecture Notes in Computer Science, pages 133-144, London, UK, July 2001. Springer.
33. B. Liu. Characterizing web response time. Master's thesis, Virginia State University, Blacksburg, Virginia, USA, April 1998.
34. B. Liu and E. A. Fox. Web traffic latency: Characteristics and implications. J.UCS: Journal of Universal Computer Science, 4(9):763-778, 1998.
35. O. A. McBryan. GENVL and WWWW: Tools for taming the web. In Proceedings of the first World Wide Web Conference, Geneva, Switzerland, May 1994.
36. R. Miller and K. Bharat. Sphinx: A framework for creating personal, site-specific web crawlers. In Proceedings of the seventh conference on World Wide Web, Brisbane, Australia, April 1998.
37. M. Najork and J. L. Wiener. Breadth-first crawling yields high-quality pages. In Proceedings of the Tenth Conference on World Wide Web, pages 114-118, Hong Kong, May 2001. Elsevier.
38. A. Ntoulas, J. Cho, and C. Olston. What's new on the web?: the evolution of the web from a search engine perspective. In Proceedings of the 13th conference on World Wide Web, pages 1-12, New York, NY, USA, May 2004. ACM Press.
39. B. Pinkerton. Finding what people want: Experiences with the WebCrawler. In Proceedings of the first World Wide Web Conference, Geneva, Switzerland, May 1994.
40. S. Raghavan and H. Garcia-Molina. Crawling the hidden web. In Proceedings of the Twenty-seventh International Conference on Very Large Databases (VLDB), pages 129-138, Rome, Italy, 2001. Morgan Kaufmann.
41. V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357 - 368, San Jose, California, February 2002. IEEE Cs. Press.
42. StatMarket. Search engine referrals nearly double worldwide. http://websidestory.com/pressroom/-pressreleases.html?id=181, 2003.
43. J. Talim, Z. Liu, P. Nain, and E. G. C. Jr. Controlling the robots of web search engines. In Proceedings of ACM Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS/Performance), pages 236-244, Cambridge, Massachusetts, USA, June 2001.
44. P.-N. Tan and V. Kumar. Discovery of web robots session based on their navigational patterns. Data Mining and Knowledge discovery, 6(1):9-35, 2002.