Observed web robot behavior on decaying web subsites

D-Lib Magazine

Volumn 12, Issue 2, 2006, Pages 59-76

  Smith, Joan A ^a   McCown, Frank ^a   Nelson, Michael L ^a  

^a ECS Building   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33644529758     PISSN: 10829873     EISSN: 10829873     Source Type: Journal    
DOI: 10.1016/j.serrev.2006.03.001     Document Type: Article

References (19)

1. Allan Heydon and Marc Najork. Mercator: A scalable, extensible web crawler. In WWW '99: Proceedings of the 8th international conference on World Wide Web, pages 219-229, 1999. 〈doi:10.1023/A:1019213109274〉.
2. Frank McCown, Joan A. Smith, Michael L. Nelson, and Johan Bollen. Reconstructing websites for the lazy webmaster. Technical report, Old Dominion University, 2005. 〈http://arxiv.org/abs/cs.IR/0512069〉.
3. Onn Brandman, Junghoo Cho, Hector Garcia-Molina, and Narayanan Shivakumar. Crawler-friendly web servers. SIGMETRICS Perform. Eval. Rev., 28(2):9-14, 2000. 〈doi:10.1145/362883.362894〉.
4. Sriram Raghavan and Hector Garcia-Molina. Crawling the hidden web. In Proceedings of the 27th International Conference on Very Large Data Bases, 2001. Pages 129-138. 〈http://citeseer.ist.psu.edu/raghavan01crawling.html〉.
5. Zoltan Gyongyi, Hector Garcia-Molina, and Jan Pedersen. Combating web spam with TrustRank. In Proceedings of the 30th International Conference on Very Large Data Bases, pages 576-587, 2004. 〈http://www.vldb.org/conf/2004/ RS15P3.PDF〉.
6. Michelangelo Diligenti, Marco Gori, and Marco Maggini. Web page scoring systems for horizontal and vertical search. In WWW 2002: Proceedings of the 11th international conference on World Wide Web, May 2002. 〈doi:10.1145/511446. 511512〉.
7. Taher H. Haveliwala. Topic-sensitive page rank. In WWW 2002: Proceedings of the 11th international conference on World Wide Web, May 2002. 〈doi10.1145/511446.511513〉.
8. Junghoo Cho and Hector Garcia-Molina. Effective page refresh policies for web crawlers. ACM Trans. Database Syst., 28(4):390-426, 2003. 〈doi:10.1145/958942.958945〉.
9. Junghoo Cho, Narayanan Shivakumar, and Hector Garcia-Molina. Finding replicated web collections. In Proceedings from SIGMOD '00, pages 355-366, 2000. 〈doi: 10.1145/342009.335429〉.
10. Virgilio Almeida, Daniel Menasce, Rudolf Riedi, Flavia Peligrinelli, Rodrigo Fonseca, and Wagner Meira Jr. Analyzing web robots and their impact on caching. In Proceedings of the 6th Web Caching Workshop, 2001. 〈http://citeseer.ist.psu.edu/almeida01analyzing.html〉.
11. Kevin Hemenway and Tara Calishain. Spidering Hacks. O'Reilly Media, Inc., first edition, November 2003.
12. Dirk Lewandowski, Henry Wahlig, and Gunnar Meyer-Beautor. The freshness of Web search engines' databases. Journal of Information Science 31, 2005. 〈http://eprints.rclis.org/archive/00004619/〉.
13. Dirk Lewandowski. Web searching, search engines and Information Retrieval. Journal of Information Services & Use 18(3) 2005. 〈http://eprints.rclis.org/archive/00004620/〉.
14. Adeniyi Oke and Richard B. Bunt. Hierarchical workload characterization for a busy web server. In Computer Performance Evaluation/TOOLS, pages 309-328, 2002. 〈http://citeseer.ist.psu.edu/oke02hierarchical.html〉.
15. Edward T. O'Neill, Brian F. Lavoie, and Rick Bennett. Trends in the evolution of the public web: 1998-2002. D-Lib Magazine, 9(4), April 2003. 〈doi:10.1045/april2003-lavoie〉.
16. Web Characterization Terminology and Definition Sheet, W3C Working Draft 24-May-1999, May 1999. Brian Lavoie and Henrik Frystyk Nielsen, Editors. W3C Working Draft. 24-May-1999. 〈http://www.w3.org/1999/05/WCA-terms/〉.
17. Frank McCown, Xiaoming Liu, Michael L. Nelson, and Mohammed Zubair. Search engine coverage of the OAI-PMH corpus. Old Dominion University, 2005. To appear in IEEE Internet Computing March/April 2006.
18. The Wayback Machine Frequently Asked Questions, 〈http://www.archive. org/about/faqs.php〉. Quoted from website on 18 January 2006.
19. Tony Hammond, Timo Hannay, Ben Lund, and Joanna Scott. Social bookmarking tools (I): A general overview. D-Lib Magazine, 11 (4), April 2005. 〈doi: 10.1045/april2005-hammond〉.