Crawling a country: Better strategies than breadth-first for web page ordering

14th International World Wide Web Conference, WWW2005

Volumn , Issue , 2005, Pages 864-872

  Baeza Yates, Ricardo ^a   Castillo, Carlos ^a   Marin, Mauricio ^b   Rodriguez, Andrea ^c  

^a UNIVERSITY OF CHILE   (Chile)

^b UNIVERSIDAD DE MAGALLANES   (Chile)

^c UNIVERSITY OF CONCEPCIÓN   (Chile)

Author keywords

Scheduling policy; Web crawler; Web page importance

Indexed keywords

BREADTH-FIRST; BREADTH-FIRST SEARCH; ORDERING STRATEGY; PAGERANK; SCHEDULING POLICIES; WEB CRAWLERS; WEB CRAWLING; WEB GRAPHS; WEB PAGE;

SCHEDULING; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 77953053635     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1062745.1062768     Document Type: Conference Paper

References (59)

1. Robotcop. www.robotcop.org, 2002.
2. HT://Dig. http://www.htdig.org/, 2004. GPL software.
3. S. Abiteboul, M. Preda, and G. Cobena. Adaptive on-line page importance computation. In Proceedings of the twelfth international conference on World Wide Web, pages 280-290. ACM Press, 2003.
4. S. Ailleret. Larbin. http://larbin.sourceforge.net/index-eng.html, 2004. GPL software.
5. A. Arasu, J. Cho, H. Garcia-Molina, A. Paepcke, and S. Raghavan. Searching the Web. ACM Transactions on Internet Technology (TOIT), 1(1):2-43, August 2001.
6. R. Baeza-Yates and C. Castillo. Balancing volume, quality and freshness in web crawling. In Soft Computing Systems - Design, Management and Applications, pages 565-572, Santiago, Chile, 2002. IOS Press Amsterdam.
7. R. Baeza-Yates and C. Castillo. Crawling the infinite Web: five levels are enough. In Proceedings of the third Workshop on Web Graphs (WAW), volume 3243 of Lecture Notes in Computer Science, pages 156-167, Rome, Italy, October 2004. Springer.
8. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. ACM Press / Addison-Wesley, 1999.
9. R. Baeza-Yates, F. Saint-Jean, and C. Castillo. Web structure, dynamics and page quality. In Proceedings of String Processing and Information Retrieval (SPIRE), volume 2476 of Lecture Notes in Computer Science, pages 117 - 132, Lisbon, Portugal, 2002. Springer.
10. P. Boldi, B. Codenotti, M. Santini, and S. Vigna. UbiCrawler: a scalable fully distributed Web crawler. Software, Practice and Experience, 34(8):711-726, 2004.
11. P. Boldi, M. Santini, and S. Vigna. Do your worst to make the best: Paradoxical effects in pagerank incremental computations. In Proceedings of the third Workshop on Web Graphs (WAW), volume 3243 of Lecture Notes in Computer Science, pages 168-180, Rome, Italy, October 2004. Springer.
12. O. Brandman, J. Cho, H. Garcia-Molina, and N. Shivakumar. Crawler-friendly web servers. In Proceedings of the Workshop on Performance and Architecture of Web Servers (PAWS), Santa Clara, California, USA, June 2000.
13. S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1-7):107-117, April 1998.
14. R. D. Burke. Salticus: guided crawling for personal digital libraries. In Proceedings of the first ACM/IEEE-CS joint conference on Digital Libraries, pages 88-89, Roanoke, Virginia, June 2001.
15. M. Burner. Crawling towards eternity - building an archive of the world wide web. Web Techniques, 2(5), May 1997.
16. C. Castillo, M. Marin, A. Rodra-Molina, and L. Page. E cient crawling through URL ordering. In Proceedings of the seventh conference on World Wide Web. Elsevier Science, April 1998.
17. S. Chakrabarti. Mining the Web. Morgan Kaufmann Publishers, 2003.
18. S. Chakrabarti, M. van den Berg, and B. Dom. Focused crawling: a new approach to topic-specific web resource discovery. Computer Networks, 31(11-16):1623-1640, 1999.
19. J. Cho and R. Adams. Page quality: In search of an unbiased Web ranking. Technical report, UCLA Computer Science, 2004.
20. J. Cho and H. Garcia-Molina. Synchronizing a database to improve freshness. In Proceedings of ACM International Conference on Management of Data (SIGMOD), pages 117-128, Dallas, Texas, USA, May 2000.
21. J. Cho and H. Garcia-Molina. Parallel crawlers. In Proceedings of the eleventh international conference on World Wide Web, pages 124-135, Honolulu, Hawaii, USA, May 2002. ACM Press.
22. J. Cho, H. García-Molina, and L. Page. Effcient crawling through URL ordering. In Proceedings of the seventh conference on World Wide Web. Elsevier Science, April 1998.
23. J. Cho, N. Shivakumar, and H. Garcia-Molina. Finding replicated web collections. In ACM SIGMOD, pages 355-366, 1999.
24. N. Craswell, F. Crimmins, D. Hawking, and A. Moffat. Performance and cost tradeoffs in web search. In Proceedings of the 15th Australasian Database Conference, pages 161-169, Dunedin, New Zealand, January 2004.
25. A. Czumaj, I. Finch, L. Gasieniec, A. Gibbons, P. Leng, W. Rytter, and M. Zito. Efficient Web searching using temporal factors. Theoretical Computer Science, 262(1-2):569-582, 2001.
26. A. S. da Silva, E. A. Veloso, P. B. Golgher, B. A. Ribeiro-Neto, A. H. F. Laender, and N. Ziviani. Cobweb - a crawler for the brazilian web. In Proceedings of String Processing and Information Retrieval (SPIRE), pages 184-191, Cancun, Mexico, September 1999. IEEE CS Press.
27. L. Dacharay. WebBase. http://freesoftware.fsf.org/webbase/, 2002. GPL Software.
28. M. Diligenti, F. Coetzee, S. Lawrence, C. L. Giles, and M. Gori. Focused crawling using context graphs. In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt, September 2000.
29. S. Dill, R. Kumar, K. S. Mccurley, S. Rajagopalan, D. Sivakumar, and A. Tomkins. Self-similarity in the web. ACM Trans. Inter. Tech., 2(3):205-223, 2002.
30. R. W. Edward G. Coffman, Z. Liu. Optimal robot scheduling for web search engines. Journal of Scheduling, 1(1):15-29, 1998.
31. J. Edwards, K. S. McCurley, and J. A. Tomlin. An adaptive model for optimizing performance of an incremental web crawler. In Proceedings of the Tenth Conference on World Wide Web, pages 106-113, Hong Kong, May 2001. Elsevier Science.
32. D. Eichmann. The RBSE spider: balancing effective search against web load. In Proceedings of the first World Wide Web Conference, Geneva, Switzerland, May 1994.
33. N. Eiron, K. S. McCurley, and J. A. Tomlin. Ranking the web frontier. In Proceedings of the 13th international conference on World Wide Web, pages 309-318. ACM Press, 2004.
34. D. Fetterly, M. Manasse, and M. Najork. Spam, damn spam, and statistics: Using statistical analysis to locate spam Web pages. In Proceedings of the seventh workshop on the Web and databases (WebDB), June 2004.
35. Y. Hafri and C. Djeraba. High performance crawling system. In Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, pages 299-306. ACM Press, 2004.
36. M. Henzinger, A. Heydon, M. Mitzenmacher, and M. Najork. On near-uniform url sampling. In Proceedings of the Ninth Conference on World Wide Web, pages 295-308, Amsterdam, Netherlands, May 2000. Elsevier Science.
37. M. Hersovici, M. Jacovi, Y. S. Maarek, D. Pelleg, M. Shtalhaim, and S. Ur. The shark-search algorithm. An application: tailored Web site mapping. In Proceedings of the seventh conference on World Wide Web, pages 317-326. Elsevier Science, April 1998.
38. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web Conference, 2(4):219-229, April 1999.
39. M. G. Kendall. Rank Correlation Methods. Gri n, London, England, 1970.
40. M. Koster. Robots in the web: threat or treat ? ConneXions, 9(4), April 1995.
41. O. A. McBryan. GENVL and WWWW: Tools for taming the web. In Proceedings of the first World Wide Web Conference, Geneva, Switzerland, May 1994.
42. A. K. McCallum, K. Nigam, J. Rennie, and K. Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3(2):127-163, 2000.
43. F. Menczer, G. Pant, P. Srinivasan, and M. E. Ruiz. Evaluating topic-driven web crawlers. In Proceedings of the 24th conference on research and development in information retrieval (SIGIR), pages 241-249. ACM Press, 2001.
44. R. Miller and K. Bharat. Sphinx: A framework for creating personal, site-specific web crawlers. In Proceedings of the seventh conference on World Wide Web, Brisbane, Australia, April 1998. Elsevier Science.
45. M. Najork and J. L. Wiener. Breadth-first crawling yields high-quality pages. In Proceedings of the Tenth Conference on World Wide Web, pages 114-118, Hong Kong, May 2001. Elsevier Science.
46. A. Ntoulas, J. Cho, and C. Olston. What's new on the web?: the evolution of the web from a search engine perspective. In Proceedings of the 13th conference on World Wide Web, pages 1-12, New York, NY, USA, May 2004. ACM Press.
47. L. Page, S. Brin, R. Motwani, and T. Winograd. The Pagerank citation algorithm: bringing order to the web. Technical report, Stanford Digital Library Technologies Project, 1998.
48. G. Pant, S. Bradshaw, and F. Menczer. Search engine-crawler symbiosis. In Proceedings of the European Conference on Digital Libraries (ECDL), volume 2769 of Lecture Notes in Computer Science, pages 221-232. Springer, August 2003.
49. B. Pinkerton. Finding what people want: Experiences with the WebCrawler. In Proceedings of the first World Wide Web Conference, Geneva, Switzerland, May 1994.
50. S. Raghavan and H. Garcia-Molina. Crawling the hidden web. In Proceedings of the Twenty-seventh International Conference on Very Large Databases (VLDB), pages 129-138, Rome, Italy, 2001. Morgan Kaufmann.
51. K. M. Risvik and R. Michelsen. Search engines and web dynamics. Computer Networks, 39(3), June 2002.
52. V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California, February 2002. IEEE CS Press.
53. J. Talim, Z. Liu, P. Nain, and E. G. C. Jr. Controlling the robots of web search engines. In Proceedings of ACM Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS/Performance), pages 236-244, Cambridge, Massachusetts, USA, June 2001.
54. P.-N. Tan and V. Kumar. Discovery of web robots session based on their navigational patterns. Data Mining and Knowledge discovery, 6(1):9-35, 2002. (Pubitemid 37113874)
55. The Economist. Country Profiles, 2002.
56. United Nations. Population Division, 2002.
57. United Nations. Human Development Reports, 2003.
58. D. Zeinalipour-Yazti and M. D. Dikaiakos. Design and implementation of a distributed crawler and filtering processor. In Proceedings of the fifth Next Generation Information Technologies and Systems (NGITS), volume 2382 of Lecture Notes in Computer Science, pages 58-74, Caesarea, Israel, June 2002. Springer.
59. H. Zhang, A. Goel, R. Govindan, K. Mason, and B. V. Roy. Making eigenvector-based reputation systems robust to collusion. In Proceedings of the third Workshop on Web Graphs (WAW), volume 3243 of Lecture Notes in Computer Science, pages 92-104, Rome, Italy, October 2004. Springer.