An adaptive model for optimizing performance of an incremental web crawler

Proceedings of the 10th International Conference on World Wide Web, WWW 2001

Volumn , Issue , 2001, Pages 106-113

  Edwards, Jenny ^a   McCurley, Kevin ^b   Tomlin, John ^b  

^a UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^b IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

Crawler; Incremental crawler; Optimization; Scalability

Indexed keywords

OPTIMIZATION; SCALABILITY; WEBSITES;

ADAPTIVE APPROACH; ADAPTIVE MODELING; COMPUTATIONAL RESULTS; CRAWLER; INCREMENTAL CRAWLER; MAGIC BULLETS; OPTIMIZATION MODELING; OPTIMIZING PERFORMANCE;

WEB CRAWLER;

EID: 84874252492     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/371920.371960     Document Type: Conference Paper

References (13)

1. B. Brewington and G. Cybenko. How dynamic is the web In Proceedings of the 9th World Wide Web Conference (WWW9), 2000.
2. B. Brewington and G. Cybenko. Keeping up with the changing web. Computer, pages 52-58, May 2000.
3. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. In Proceedings of the 7th World Wide Web Conference (WWW7), 1998.
4. A. Broder, S. Glassman, M. Manasse, G. Zweig. Syntactic clustering of the web. In Proceedings of 6th International World Wide Web Conference (WWW6), 1997.
5. J. Cho and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. In Proceedings of 26th International Conference on Very Large Databases (VLDB), 2000.
6. J. Cho and H. Garcia-Molina. Synchronizing a database to improve freshness. In Proceedings of 2000 ACM International Conference on Management of Data (SIGMOD), 2000.
7. J. Cho, H. Garcia-Molina, L. Page. EÆcient crawling through URL ordering. In Proceedings of the 7th World Wide Web Conference (WWW7), 1998.
8. E. Cofiman, Z. Liu, R. Weber. Optimal robot scheduling for web search engines, Rapport de recherche no 3317. Technical report, INRIA Sophia Antipolis, 1997.
9. F. Douglis, A. Feldmann, B. Krishnamurthy. Rate of change and other metrics: A live study of the world wide web. In Proceedings of USENIX Symposium on Internetworking Technologies and Systems, 1997.
10. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219-229, 1999.
11. MINOS. (http://www.sbsi-sol-optimize.com/minos.htm).
12. NEOS Server for Optimization. (http://www-neos.mcs.anl.gov).
13. C. Wills and M. Mikhailov. Towards a better understanding of web resources and server responses for improved caching. In Proceedings of the 8th World Wide Web Conference (WWW8), 1999.