Towards continuous web archiving: First results and an agenda for the future

D-Lib Magazine

Volumn 8, Issue 12, 2002, Pages

  Masanès, Julien ^a  

^a Dept des Arts du Spectacle   (France)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 1642345851     PISSN: 10829873     EISSN: None     Source Type: Journal    
DOI: 10.1045/december2002-masanes     Document Type: Article

References (16)

1. Cathro, W., Webb, C. & Whiting, J. (2001). "Archiving the Web: the PANDORA archive at the National Library of Australia." Preserving the Present for the Future Web Archiving Conference, Copenhagen, 18-19 June 2001. Available at: .
2. Electronic Collections Coordinating Group, National Library Of Canada, (1998). "Networked Electronic Publications Policy and Guidelines". Available at: .
3. Arvidson, A., Persson, K. & Mannerheim, J. (2000). "The Kulturarw3 Project - the Royal Swedish Web Archiw3e: an example of 'complete' collection of web pages." 66th IFLA Council and General Conference, Jerusalem, Israel, 13-18 August 2000. Available at: .
4. Kahle, B. (2002). "Editors' Interview: The Internet Archive." RLG DigiNews, 6 (3), 15 June 2002. Available at: .
5. See for example political sites archiving in The Netherlands, Available at: . Or the Digital Archive for Chinese Studies (DACHS), Available at: . We are trying to make an inventory of on-going web archiving project, so you are welcome to send information about the ones of which you are aware.
6. Hakala, J. (2001). "Collecting and Preserving the Web: Developing and Testing the NEDLIB Harvester." RLG DigiNews, 5 (2), 15 April 2001. Available at: .
7. The main discussion list on this topic is web-archive@cru.fr. Information available at: .
8. cf. 2nd ECDL Workshop on Web Archiving, Rome, Italy, 19 September 2002. Available at: .
9. Masanès, J. (2002). "Archiving the deep Web" 2nd ECDL Workshop on Web Archiving, Rome, Italy, 19 September 2002. Available at: .
10. Abiteboul, S., Cobéna, G., Masanès, J. & Sedrati, G. (2002). "A first experience in archiving the French Web." In: Research and advanced technology for digital libraries: 6th European conference, ECDL 2002, Agosti, M. & Thanos, C., eds., Rome, Italy, September 16-18, 2002. Lecture Notes in Computer Science, 2458. Berlin: Springer, 1-15. Also available at: .
11. To give an illustration of possible gains, here are figures extracted from our 'Elections 2002' collection. This collection encompasses 2.200 sites or part of sites related to the presidential and parliamentary elections held in France in 2002. On a sample on these sites, 43 of the most captured ones, we have for April 2.103.360 files for 6 captures which represent 108 GB of data. Among these files only 45,7% are unique files, which represent 56.3% of the total amount of data. This means that more than a half of the crawling capacity and 43.7 of the storage capacity is 'wasted' in this case. It is really beneficial to have crawler able to manage sites changes in this kind of 'continuous' crawl. The small crawler we have used, HTTRACK (see ), is able to do incremental crawl and with a few scripts and a database, it can be used to handle automatic crawl of hundreds of sites.
12. Masanès, J. (2001). "The BnF's project for Web archiving." What's next for digital deposit libraries? ECDL Workshop, Darmstadt, Germany, 8 September 2001. Available at: .
13. Brin, S. & Page, L. (1998). "The Anatomy of a Large-scale Hypertextual Web Search Engine." Computer Networks and ISDN Systems, 30 (1-7), 107-117. Full version published in the proceedings of the 7th International World Wide Web Conference, Brisbane, Australia, 14-18 April 1998. Available at: .
14. Véronique Berton, Virginie Breton, Dominique Chrishmann, Christine Genin, Loïc Le Bail, Soraya Salah, Jean-Yves Sarazin and Julian Masanès.
15. Abiteboul, S., Preda, M. & Cobéna, G. (2002). "Computing Web page importance without storing the graph of the Web (extended abstract)." Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 25 (1), 27-33. Available at: .
16. Thanks to Gregory Cobéna from INRIA for his help on this part.