Factors affecting Website reconstruction from the Web infrastructure

Proceedings of the ACM International Conference on Digital Libraries

Volumn , Issue , 2007, Pages 39-48

  McCown, Frank ^a   Diawara, Norou ^b   Nelson, Michael L ^a  

^a Old Dominion University   (United States)

^b OLD DOMINION UNIVERSITY   (United States)

Author keywords

Digital preservation; Search engine caches; Web archiving

Indexed keywords

MATHEMATICAL MODELS; SEARCH ENGINES; SECURITY OF DATA; STATISTICAL METHODS;

DIGITAL PRESERVATION; WEB ARCHIVING; WEB INFRASTRUCTURE (WI); WEBSITE RECONSTRUCTION;

WEBSITES;

EID: 36348992623     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1255175.1255182     Document Type: Conference Paper

References (38)

1. L. A. Adamic and B. A. Huberman. Zipf's law and the Internet. Glottometrics, 3:143-150, 2002.
2. Alexa toolbar, http://download.alexa.com/.
3. Z. Bar-Yossef, A. Z. Broder, R. Kumar, and A. Tomkins. Sic transit gloria telae: towards an understanding of the web's decay. In WWW '04: Proceedings of the 13th international conference on World Wide Web, pages 328-337, 2004.
4. Z. Bar-Yossef and M. Gurevich. Random sampling from a search engine's index. In WWW '06: Proceedings of the 15th international conference on World Wide Web, pages 367-376, 2006.
5. B. E. Brewington and G. Cybenko. How dynamic is the Web? Computer Networks, 33(1-6):257-276, 2000.
6. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the Web. Computer Networks & ISDN Systems, 29(8-13):1157-1166, 1997.
7. J. Cho and S. Roy. Impact of search engines on page popularity. In WWW '04: Proceedings of the 13th international conference on World Wide Web, pages 20-29, 2004.
8. D. Clinton. Beyond the SOAP search API, Dec. 2006. http://google-code- updates.blogspot.com/2006/ 12/beyond-soap-search-api.html. [9] M. Cutts. GoogleGuy's posts, June 2005. http: //www.webmasterworld.com/forum30/29720.htm.
9. Z. Dalai, S. Dash, P. Dave, L. Francisco-Revilla, R. Furuta, U. Karadkar, and F. Shipman. Managing distributed collections: Evaluating web page changes, movement, and replacement. In JCDL '04: Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, pages 160-168, 2004.
10. F. Douglis, A. Feldmann, and B. Krishnamurthy. Rate of change and other metrics: a live study of the World Wide Web. In Proceedings of the USENIX Symposium on Internet Technologies and Systems, 1997.
11. D. Fetterly, M. Manasse, M. Najork, and J. Wiener. A large-scale study of the evolution of web pages. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 669-678, 2003.
12. J. Gait. Google says: Toolbar PageRank is for entertainment purposes only, 2004. http://forums, searchenginewatch.com/showthread.php?t=3054.
13. Google Sitemap Protocol, https : //www. google. com/ webmasters/tools/ docs/en/protocol.html.
14. Google webmaster help center: Webmaster guidelines, 2007. http ://www.google.com/support/webmasters/ bin/answer.py?answer=35769.
15. A. Gulli and A. Signorini. The indexable web is more than 11.5 billion pages. In WWW '05: Special interest tracks and posters of the 14th international conference on World Wide Web, pages 902-903, May 2005.
16. T. L. Harrison and M. L. Nelson. Just-in-time recovery of missing web pages. In HYPERTEXT '06: Proceedings of the 17th ACM conference on Hypertext and Hypermedia, pages 145-156, Aug. 2006.
17. Internet Archive FAQ: How can I get my site included in the Archive? http ://www.archive.org/about/faqs.php.
18. Jon. How the Google cache can save your a$$, Dec. 2005. http ://www.smartmoneydaily.com/Business/ How-the-Google-Cache-can-Save-You. aspx.
19. W. Koehler. An analysis of web page and web site constancy and permanence. Journal of the American Society for Information Science, 50(2):162-180, 1999.
20. S. Lawrence, D. M. Pennock, G. W. Flake, R. Krovetz, F. M. Coetzee, E. Glover, F. A. Nielsen, A. Kruger, and C. L. Giles. Persistence of web references in scientific research. Computer, 34(2):26-31, 2001.
21. C. Marhsall, F. McCown, and M. L. Nelson. Evaluating personal archiving strategies for Internet-based information. In Proceedings of IS&T Archiving 2007, May 2007.
22. F. McCown. Mark Foley websites - reconstructed, 2006. http://www.cs.odu.edu/~fmccown/foley/.
23. F. McCown, X. Liu, M. L. Nelson, and M. Zubair. Search engine coverage of the OAI-PMH corpus. IEEE Internet Computing, 10(2):66-73, Mar/Apr 2006.
24. F. McCown and M. L. Nelson. Evaluation of crawling policies for a web-repository crawler. In HYPERTEXT '06: Proceedings of the 17th ACM conference on Hypertext and Hypermedia, pages 145-156, 2006.
25. F. McCown and M. L. Nelson. Agreeing to disagree: Search engines and their public interfaces. In JCDL '07: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital Libraries, 2007.
26. F. McCown and M. L. Nelson. Characterization of search engine caches. In Proceedings of IS&T Archiving 2007, 2007.
27. F. McCown, J. A. Smith, M. L. Nelson, and J. Bollen. Lazy preservation: Reconstructing websites by crawling the crawlers. In Proceedings from the 8th A CM International Workshop on Web Information and Data Management (WIDM '06), pages 67-74, 2006.
28. G. Mohr, M. Kimpton, M. Stack, and I. Ranitovic. An introduction to Heritrix, an archival quality web crawler. In Proceedings of the 4th International Web Archiving Workshop (IWAW '04), Sept. 2004.
29. M. L. Nelson and B. D. Allen. Object persistence and availability in digital libraries. D-Lib Magazine, 8(1), 2002.
30. M. L. Nelson, J. A. Smith, I. Garcia del Campo, H. Van de Sompel, and X. Liu. Efficient, automatic web resource harvesting. In Proceedings from the 8th ACM International Workshop on Web Information and Data Management (WIDM '06), pages 43-50, 2006.
31. A. Ntoulas, J. Cho, and C. Olston. What's new on the Web? The evolution of the Web from a search engine perspective. In WWW '04: Proceedings of the 13th international conference on World Wide Web, pages 1-12, 2004.
32. S. Olsen. Court backs thumbnail image linking. CNET News.com, July 2003. http://news.com.com/2100-1025.3-1023629.html.
33. S. Olsen. Google cache raises copyright concerns. CNET News.com, July 2003. http ://news.com.com/2100-1038.3-1024234.html.
34. M. Thelwall. Methodologies for crawler based web surveys. Internet Research, 12(2):124-138, 2002.
35. M. Thelwall and D. Stuart. Web crawling ethics revisited: Cost, privacy, and denial of service. Journal of the American Society for Information Science and Technology, 57(13):1771-1779, 2006.
36. M. Thelwall and L. Vaughan. A fair history of the Web? Examining country balance in the Internet Archive. Library & Information Science Research, 26(2):162-176, 2004.
37. L. Vaughan and M. Thelwall. Search engine coverage bias: Evidence and possible causes. Information Processing & Management, 40(4):693-707, 2004.
38. Yahoo Site Explorer. http://siteexplorer.search.yahoo.com/.