Managing duplicates in a web archive

Proceedings of the ACM Symposium on Applied Computing

Volumn 1, Issue , 2006, Pages 818-825

  Gomes, Daniel ^a   Santos, André L ^a   Silva, Mário J ^a  

^a UNIVERSITY OF LISBON   (Portugal)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER ARCHITECTURE; DATA STORAGE EQUIPMENT; ITERATIVE METHODS; REQUIREMENTS ENGINEERING; WORLD WIDE WEB;

STORAGE SERVERS; WEB STORAGE SYSTEMS;

INFORMATION MANAGEMENT;

EID: 33751049379     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1141277.1141465     Document Type: Conference Paper

References (40)

1. B. Berliner. CVS II: Parallelizing software development. In Proceedings of the USENIX Winter 1990 Technical Conference, pages 341-352, Berkeley, CA, 1990. USENIX Association.
2. K. Bharat and A. Broder. Mirror, mirror on the web: a study of host pairs with replicated content. In Proceedings of the Eighth International. Conference on World Wide Web, pages 1579-1590. Elsevier North-Holland, Inc., 1999.
3. S. Brin, J. Davis, and H. García-Molina. Copy detection mechanisms for digital documents, pages 398-409, 1995.
4. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems, 30(1-7):107-117, 1998.
5. T. Burkard. Herodotus: A peer-to-peer web archival system, 2002.
6. M. Burner and B. Kahle. WWW Archive File Format Specification, September 1996.
7. B. Callaghan, B. Pawlowski, and P. Staubach. RFC 1813: NFS Version 3 Protocol Specification. Sun Microsystems, Inc., June 1995.
8. J. Campos. Versus: a web repository. Master thesis, 2003.
9. N. Cardoso, M. J. Silva, and M. Costa. The XLDB Group at CLEF'2004.
10. W. Cathro and T. Boston. Development of a digital services architecture at the national library of australia. EduCause, Australasia S003, page 24, 2003.
11. J. Cho and H. Garcia-Molina. Estimating frequency of change. ACM Trans. Inter. Tech., 3(3):256-290, 2003.
12. J. Cho, H. Garcia-Molina, T. Haveliwala, W. Lam, A. Paepcke, S. Raghavan, and G. Wesley. Stanford webbase components and applications. Technical report, Stanford Database Group, July 2004.
13. B. F. Cooper, A. Crespo, and H. Garcia-Molina. The Stanford archival repository project: Preserving our digital past. Technical report 2002-47, Department of Computer Science, Stanford University, October 2002.
14. A. Crespo and H. Garcia-Molina. Archival storage for digital libraries. In Proceedings of the Third ACM International Conference on Digital Libraries, 1998.
15. L. Daigle, D. van Gulik, R. Iannella, and P. Faltstrom. Uniform Resource Names (URN) Namespace Definition Mechanisms, October 2002.
16. T. Denehy and W. Hsu. Duplicate management for reference data. Technical report RJ 10305, IBM Research, October 2003.
17. D. Fetterly, M. Manasse, M. Najork, and J. Wiener. A large-scale study of the evolution of web pages. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 669-678, New York, NY, USA, 2003. ACM Press.
18. R. A. Finkel, A. Zaslavsky, K. Monostori, and H. Schmidt. Signature extraction for overlap detection in documents. In M. J. Oudshoorn, editor, Twenty-Fifth Australasian Computer Science Conference (ACSC2002), Melbourne, Australia, 2002. ACS.
19. D. Gomes, A. L. Santos, and M. J. Silva. Webstore: A manager for incremental storage of contents. DI/FCUL TR 04-15, Department of Informatics, University of Lisbon, November 2004.
20. D. Gomes and M. J. Silva. Characterizing a national community web. ACM Trans. Inter. Tech., 5(3):508-531, 2005.
21. Y. Hafri and C. Djeraba. Dominos: A new web crawler's design. In 4th International Web Archiving Workshop (IWAW04), Bath, UK, September 2004.
22. J. Hakala. Collecting and preserving the web: Developing and testing the nedlib harvester. RLG Diginews, 5(2), April 2001.
23. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219-229, 1999.
24. R. J. Honicky and E. L. Miller. A fast algorithm for online placement and reorganization of replicated data. Nice, France, Apr. 2003.
25. T. Kelly and J. Mogul. Aliasing on the world wide web: Prevalence and performance implications. In Proceedings of the 11th International World Wide. Web Conference, Honolulu, Hawaii, May 2002.
26. C. Lampos, M. Eirinaki, D. Jevtuchova, and M. Vazirgiannis. Archiving the greek web. In 4th International Web Archiving Workshop (IWAW04), Bath, UK, September 2004.
27. J. MacDonald. Versioned file archiving, compression, and distribution.
28. J. Mogul. A trace-based analysis of duplicate suppression in HTTP. Technical Report 99/2, Compaq Computer Corporation Western Research Laboratory, November 1999.
29. N. Noronha, J. P. Campos, D. Gomes, M. J. Silva, and J. Borbinha. A deposit for digital collections. In Proc. 5td European Conf. Research and Advanced Technology for Digital Libraries, ECDL, pages 200-212. Springer-Verlag, 2001.
30. A. Patterson. Why writing your own search engine is hard. Queue, 2(2):48-53, 2004.
31. K. Persson. Kulturarw description. http://www.kb.se/kw3/ENG/Description. htm, March 2005.
32. M. O. Rabin. Probabilistic algorithm in finite fields. Technical Report MIT/LCS/TR-213, 1979.
33. S. Rhea, P. Eaton, D. Geels, H. Weatherspoon, B. Zhao, and J. Kubiatowicz. Pond: the Oceanstore Prototype. In Proceedings of the 2nd Usenix Conference on File and Storage Technologies (FAST'03), 2003.
34. O. Rodeh and A. Teperman. zfs: A scalable distributed file system using object disks. In Proceedings of the 20 th IEEE/11 th NASA Goddard Conference on Mass Storage Systems and Technologies (MSS'03), page 207. IEEE Computer Society, 2003. zFS extends the research done in the DSF project.
35. N. Shivakumar and H. García-Molina. SCAM: A copy detection mechanism for digital documents. In Proceedings of the Second Annual Conference on the Theory and Practice of Digital Libraries, 1995.
36. N. Shivakumar and H. Garcia-Molina. Finding near-replicas of documents and servers on the web. In Selected papers from the International Workshop on The World Wide Web and Databases, pages 204-212. Springer-Verlag, 1999.
37. V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In Proceedings of the 18th International Conference on Data Engineering (ICDE'02), page 357. IEEE Computer Society, 2002.
38. M. J. Silva. Searching and archiving the web with tumba! In CAPSI 2003 - 4a. Conferência da Associação Portuguesa de Sistemas de Informação, Porto, Portugal, November 2003.
39. D. Spinellis. The decay and failures of web references. Communications of the ACM, 46(1):71-77, 2003.
40. L. L. You and C. Karamanolis. Evaluation of efficient archival storage techniques. In 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, College Park, MD, April 2004.