Similarity and locality based indexing for high performance data deduplication

IEEE Transactions on Computers

Volumn 64, Issue 4, 2015, Pages 1162-1176

  Xia, Wen ^a   Jiang, Hong ^b   Feng, Dan ^a   Hua, Yu ^a  

^a HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

^b UNIVERSITY OF NEBRASKA LINCOLN   (United States)

Author keywords

Data deduplication; index structure; performance evaluation; storage system

Indexed keywords

DATA COMMUNICATION SYSTEMS; RANDOM ACCESS STORAGE;

DATA DE DUPLICATIONS; DUPLICATE ELIMINATION; INDEX STRUCTURE; PERFORMANCE DATA; PERFORMANCE EVALUATION; SIMILARITY DETECTION; STATELESS ROUTING; STORAGE SYSTEMS;

DIGITAL STORAGE;

EID: 84925068051     PISSN: 00189340     EISSN: None     Source Type: Journal    
DOI: 10.1109/TC.2014.2308181     Document Type: Article

References (31)

1. S. Quinlan and S. Dorward, "Venti: A New Approach to Archival Storage, " Proc. First USENIX Conf. File and Storage Technologies, 2002.
2. P. Kulkarni, F. Douglis, J. LaVoie, and J. Tracey, "Redundancy Elimination within Large Collections of Files, " Proc. USENIX Ann. Technical Conf., pp. 59-72, 2004.
3. C. Policroniades and I. Pratt, "Alternatives for Detecting Redundancy in Storage Systems Data, " Proc. Ann. Conf. USENIX Ann. Technical Conf., 2004.
4. B. Zhu, K. Li, and H. Patterson, "Avoiding the Disk Bottleneck in the Data Domain Deduplication File System, " Proc. Sixth USENIX Conf. File and Storage Technologies, 2008.
5. K. Jin and E. Miller, "The Effectiveness of Deduplication on Virtual Machine Disk Images, " Proc. SYSTOR 2009: The Israeli Experimental Systems Conf., pp. 1-12, 2009.
6. J. Wei, H. Jiang, K. Zhou, and D. Feng, "MAD2: A Scalable High-Throughput Exact Deduplication Approach for Network Backup Services, " Proc. IEEE 26th Symp. Mass Storage Systems and Technologies (MSST), pp. 1-14, 2010.
7. D. Meyer and W. Bolosky, "A Study of Practical Deduplication, " Proc. Ninth USENIX Conf. File and Storage Technologies, 2011.
8. A. Muthitacharoen, B. Chen, and D. Mazieres, "A Low-Bandwidth Network File System, " Proc. 18th ACM Symp. Operating Systems Principles, pp. 174-187, 2001.
9. M. Lillibridge, K. Eshghi, D. Bhagwat, V. Deolalikar, G. Trezise, and P. Camble, "Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality, " Proc. Seventh Conf. File and Storage Technologies, pp. 111-123, 2009.
10. D. Bhagwat, K. Eshghi, D. Long, and M. Lillibridge, "Extreme Binning: Scalable, Parallel Deduplication for Chunk-Based File Backup, " Proc. IEEE Int'l Symp. Modeling, Analysis & Simulation of Computer and Telecomm. Systems, pp. 1-9, 2009.
11. B. Debnath, S. Sengupta, and J. Li, "Chunkstash: Speeding Up Inline Storage Deduplication Using Flash Memory, " Proc. 2010 USENIX Conf. USENIX Ann. Technical Conf., 2010.
12. W. Dong, F. Douglis, K. Li, H. Patterson, S. Reddy, and P. Shilane, "Tradeoffs in Scalable Data Routing for Deduplication Clusters, " Proc. Ninth USENIX Conf. File and Storage Technologies, 2011.
13. E. Kruus, C. Ungureanu, and C. Dubnicki, "Bimodal Content Defined Chunking for Backup Streams, " Proc. Eighth USENIX Conf. File and Storage Technologies, 2010.
14. N. Agrawal, W. Bolosky, J. Douceur, and J. Lorch, "A Five-Year Study of File-System Metadata, " ACM Trans. Storage, vol. 3, no. 3, article 9, 2007.
15. G. Wallace, F. Douglis, H. Qian, P. Shilane, S. Smaldone, M. Chamness, and W. Hsu, "Characteristics of Backup Workloads in Production Systems, " Proc. 10th USENIX Conf. File and Storage Technologies, 2012.
16. A. Broder, "On the Resemblance and Containment of Documents, " Proc. Compression and Complexity of Sequences, 1997.
17. D. Bhagwat, K. Eshghi, and P. Mehra, "Content-Based Document Routing and Index Partitioning for Scalable Similarity-Based Searches in a Large Corpus, " Proc. 13th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 105-112, 2007.
18. Y. Tan, H. Jiang, D. Feng, L. Tian, Z. Yan, and G. Zhou, "SAM: A Semantic-Aware Multi-Tiered Source De-Duplication Framework for Cloud Backup, " Proc. IEEE 39th Int'l Conf. Parallel Processing, pp. 614-623, 2010.
19. "Linux download, ftp://ftp.kernel.org/, " 2014.
20. Y. Xing, Z. Li, and Y. Dai, "PeerDedupe: Insights into the Peer-Assisted Sampling Deduplication, " Proc. IEEE 10th Int'l Conf. Peer-to-Peer Computing (P2P), 2010.
21. F. Guo and P. Efstathopoulos, "Building a High-Performance Deduplication System, " Proc. 2011 Conf. USENIX Ann. Technical Conf., 2011.
22. R. Pagh and F.F. Rodler, "Cuckoo Hashing, " J. Algorithms, vol. 51, no. 2, pp. 122-144, 2004.
23. A. Broder, "Some Applications of Rabinas Fingerprinting Method, " Sequences II: Methods in Comm., Security, and Computer Science. Springer, 1993.
24. K. Eshghi and H.K. Tang, "A Framework for Analyzing and Improving Content-Based Chunking Algorithms, " Technical Report HPL-2005-30R1 HP Labs, 2005.
25. C. Dubnicki, L. Gryz, L. Heldt, M. Kaczmarczyk, W. Kilian, P. Strzelczak, J. Szczepkowski, C. Ungureanu, and M. Welnicki, "HYDRAstor: A Scalable Secondary Storage, " Proc. Seventh Conf. File and Storage Technologies, pp. 197-210, 2009.
26. U. Manber et al., "Finding Similar Files in a Large File System, " Proc. USENIX Winter Technical Conf., pp. 1-10, 1994.
27. L. Aronovich, R. Asher, E. Bachmat, H. Bitner, M. Hirsch, and S. Klein, "The Design of a Similarity Based Deduplication System, " Proc. SYSTOR 2009: The Israeli Experimental Systems Conf., 2009.
28. V. Michael, S. Stefan, and M. Geoffrey, "Cumulus: Filesystem Backup to the Cloud, " Proc. Seventh USENIX Conf. File and Storage Technologies, 2009.
29. P. Bhatotia, R. Rodrigues, and A. Verma, "Shredder: GPUAccelerated Incremental Storage and Computation, " Proc. 10th USENIX Conf. File and Storage Technologies, 2012.
30. W. Xia, H. Jiang, D. Feng, L. Tian, M. Fu, and Z. Wang, "PDedupe: Exploiting Parallelism in Data Deduplication System, " Proc. IEEE Seventh Int'l Conf. Networking, Architecture and Storage (NAS), pp. 338-347, 2012.
31. K. Srinivasan, T. Bisson, G. Goodson, and K. Voruganti, "iDedup: Latency-Aware, Inline Data Deduplication for Primary Storage, " Proc. 10th USENIX Conf. File and Storage Technologies, 2012.