Hash challenges: Stretching the limits of compare-by-hash in distributed data deduplication

Information Processing Letters

Volumn 112, Issue 10, 2012, Pages 380-385

  Barreto, João ^a   Veiga, Luís ^a   Ferreira, Paulo ^a  

^a INESC ID   (Portugal)

Author keywords

Compare by hash; Data deduplication; Distributed file systems; Distributed systems

Indexed keywords

BROADBAND CONNECTION; COMMUNICATION OVERHEADS; COMPARE-BY-HASH; DEDUPLICATION; DISTRIBUTED DATA; DISTRIBUTED FILE SYSTEMS; DISTRIBUTED SYSTEMS; PERFORMANCE GAIN; REDUNDANT DATA;

COMPUTER APPLICATIONS; DATA PROCESSING;

ELECTRONIC DATA INTERCHANGE;

EID: 84857486926     PISSN: 00200190     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipl.2012.01.012     Document Type: Article

References (18)

1. W. Hu, T. Yang, and J.N. Matthews The good, the bad and the ugly of consumer cloud storage SIGOPS Oper. Syst. Rev. 44 2010 110 115
2. D. Belson, Akamai state of the Internet report, Q1 2011, Technical report, Akamai, August 2011.
3. S. Ihm, K. Park, V.S. Pai, Towards understanding developing world traffic, in: Proceedings of the 4th ACM Workshop on Networked Systems for Developing Regions.
4. A. Muthitacharoen, B. Chen, D. Mazieres, A low-bandwidth network file system, in: 8th ACM SOSP, 2001, pp. 174-187. (Pubitemid 33612613)
5. N. Mandagere, P. Zhou, M.A. Smith, S. Uttamchandani, Demystifying data deduplication, in: Proceedings of the ACM/IFIP/USENIX Middleware Conference Companion, 2008, pp. 12-17.
6. A. Trigdell, P. Mackerras, The rsync algorithm, Tech. rep., Australian National University, 1998.
7. National Institute of Standards and Technology, FIPS PUB 180-1: Secure Hash Standard, NIST.
8. N. Jain, M. Dahlin, R. Tewari, Taper: Tiered approach for eliminating redundancy in replica synchronization, in: USENIX FAST, 2005, p. 21.
9. U. Irmak, T. Suel, Hierarchical substring caching for efficient content distribution to low-bandwidth clients, in: WWW, 2005, pp. 43-53.
10. D. Bobbarjung, S. Jagannathan, and C. Dubnicki Improving duplicate elimination in storage systems ACM Trans. Storage 2 4 2006 424 448 (Pubitemid 46192923)
11. K. Eshghi, M. Lillibridge, L. Wilcock, G. Belrose, R. Hawkes, Jumbo store: providing efficient incremental upload and versioning for a utility rendering service, in: USENIX FAST, 2007, p. 22.
12. M.C. Chan, T.Y.C. Woo, Cache-based compaction: A new technique for optimizing web transfer, in: INFOCOM, 1999, pp. 117-125.
13. L. Cox, B. Noble, Pastiche: Making backup cheap and easy, in: 5th OSDI, ACM, 2002, pp. 285-298.
14. N. Tolia, M. Kozuch, M. Satyanarayanan, B. Karp, A. Perrig, T. Bressoud, Opportunistic use of content addressable storage for distributed file systems, in: USENIX Annual Technical Conference, 2003, pp. 127-140.
15. S. Annapureddy, M.J. Freedman, D. Mazières, Shark: scaling file servers via cooperative caching, in: USENIX NSDI, 2005, pp. 129-142.
16. S. Ihm, K. Park, V.S. Pai, Wide-area network acceleration for the developing world, in: USENIX Annual Technical Conference.
17. T.G. Kurtz, and U. Manber A probabilistic distributed algorithm for set intersection and its analysis Theor. Comput. Sci. 49 1987 267 282
18. B. Kalyanasundaram, and G. Schnitger The probabilistic communication complexity of set intersection SIAM J. Discr. Math. 5 4 1992 545 557