Probabilistic deduplication for cluster-based storage systems

Proceedings of the 3rd ACM Symposium on Cloud Computing, SoCC 2012

Volumn , Issue , 2012, Pages

  Frey, Davide ^a   Kermarrec, Anne Marie ^a   Kloudas, Konstantinos ^a  

^a INRIA   (France)

Author keywords

Cloud computing; Deduplication; Distributed storage systems; Set intersection

Indexed keywords

BACK-UP SYSTEMS; CLUSTER-BASED; CLUSTER-BASED STORAGE; COMPUTATIONAL COSTS; DATA BLOCKS; DATA SETS; DEDUPLICATION; DISTRIBUTED STORAGE SYSTEM; EXPERIMENTAL PLATFORM; INDEXING MECHANISMS; LOAD-BALANCING; MEMORY OVERHEADS; MEMORY RESOURCES; SET INTERSECTION; STORAGE SYSTEMS; WIKIPEDIA;

CLOUD COMPUTING; DIGITAL STORAGE; MULTIPROCESSING SYSTEMS;

COST REDUCTION;

EID: 84870480815     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2391229.2391246     Document Type: Conference Paper

References (23)

1. http://dumps.wikimedia.org/enwiki/.
2. https://www.grid5000.fr/.
3. Atul Adya, William J. Bolosky, Miguel Castro, Gerald Cermak, Ronnie Chaiken, John R. Douceur, Jon Howell, Jacob R. Lorch, Marvin Theimer, and Roger P. Wattenhofer. FARSITE: Federated, Available, and Reliable Storage for an Incompletely Trusted Environment. In OSDI, 2002.
4. Deepavali Bhagwat, Kave Eshghi, Darrell D. E. Long, and Mark Lillibridge. Extreme Binning: Scalable, Parallel Deduplication for Chunk-based File Backup. In MASCOTS, 2009.
5. S. Brin, J. Davis, and H. Garcia-Molina. Copy Detection Mechanisms for Digital Documents. In SIGMOD, 1995.
6. Wei Dong, Fred Douglis, Kai Li, Hugo Patterson, Sazzala Reddy, and Philip Shilane. Tradeoffs in Scalable Data Routing for Deduplication Clusters. In FAST, 2011.
7. Cezary Dubnicki, Leszek Gryz, Lukasz Heldt, Michal Kaczmarczyk, Wojciech Kilian, Przemyslaw Strzelczak, Jerzy Szczepkowski, Cristian Ungureanu, and Michal Welnicki. HYDRAstor: a Scalable Secondary Storage. In FAST, 2009.
8. M. Durand and P. Flajolet. Loglog counting of large cardinalities. In ESA, 2003.
9. P. Flajolet and G. N. Martin. Probabilistic counting algorithms for data base applications. Journal of Computer and System Sciences, 31, 1985.
10. J. F. Gantz, C. Chute, A. Manfrediz, S. Minton, D. Reinsel, W. Schlichting, and A. Toncheva. The Diverse and Exploding Digital Universe: An Updated Forecast of Worldwide Information Growth Through 2011. Technical report, An IDC White Paper - sponsored by EMC, March 2008.
11. Fanglu Guo and Petros Efstathopoulos. Building a High-performance Deduplication Systems. In USENIX ATC, 2011.
12. Erik Kruus, Cristian Ungureanu, and Cezary Dubnicki. Bimodal Content Defined Chunking for Backup Streams. In FAST, 2010.
13. Purushottam Kulkarni, Fred Douglis, Jason LaVoie, and John M. Tracey. Redundancy elimination within large collections of files. In USENIX ATC, 2004.
14. Mark Lillibridge, Kave Eshghi, Deepavali Bhagwat, Vinay Deolalikar, Greg Trezise, and Peter Camble. Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality. In FAST, 2009.
15. Dutch T. Meyer and William J. Bolosky. A Study of Practical Deduplication. In FAST, 2011.
16. Sebastian Michel, Matthias Bender, Nikos Ntarmos, Peter Triantafillou, Gerhard Weikum, and Christian Zimmer. Discovering and Exploiting Keyword and Attribute-Value Co-occurrences to Improve P2P Routing Indices. In CIKM, 2006.
17. Athicha Muthitacharoen, Benjie Chen, and David Mazières. A low-bandwidth network file system. In SOSP, 2001.
18. C. Policroniades and I. Pratt. Alternatives for Detecting Redundancy in Storage Systems Data. In USENIX ATC, 2004.
19. Himabindu Pucha, David G. Andersen, and Michael Kaminsky. Exploiting Similarity for Multi-Source Downloads Using File Handprints. In NSDI, 2007.
20. Sean Rhea, Russ Cox, and Alex Pesterev. Fast, inexpensive content-addressed storage in foundation. In USENIX ATC, 2008.
21. Kanat Tangwongsan, Himabindu Pucha, David G. Andersen, and Michael Kaminsky. Efficient Similarity Estimation for Systems Exploiting Data Redundancy. In INFOCOM, 2010.
22. Cristian Ungureanu, Benjamin Atkin, Akshat Aranya, Salil Gokhale, Stephen Rago, Grzegorz Calkowski, Cezary Dubnicki, and Aniruddha Bohra. HydraFS: a High-Throughput File System for the HYDRAstor Content-Addressable Storage System. In FAST, 2010.
23. Grant Wallace, Fred Douglis, Hangwei Qian, Philip Shilane, Stephen Smaldone, Mark Chamness, and Windsor Hsu. Characteristics of Backup Workloads in Production Systems. In FAST, 2012.