DS-Dedupe: A scalable, low network overhead data routing algorithm for inline cluster deduplication system

2014 International Conference on Computing, Networking and Communications, ICNC 2014

Volumn , Issue , 2014, Pages 895-899

  Sun, Zhen ^a   Xiao, Nong ^a   Liu, Fang ^a   Fu, Yinjin ^a  

^a NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

Author keywords

data deduplication; data routing algorithm; network overhead; scalability

Indexed keywords

SCALABILITY;

COMMUNICATION OVERHEADS; DATA DE DUPLICATIONS; DE DUPLICATIONS; METADATA SERVERS; NETWORK OVERHEAD; SIMILARITY COEFFICIENTS; SIMILARITY INDICES; STORAGE EFFICIENCY;

ROUTING ALGORITHMS;

EID: 84899537680     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICCNC.2014.6785456     Document Type: Conference Paper

References (15)

1. Villars, R.L., Olofson, C.W., Eastwood, M.: Big Data: What It Is and Why You Should Care. White Paper, IDC (2011).
2. John Gantz, David Reinsel. The Digital Universe Decade - Are You Ready? [R]. IDC White Paper, 2010: 1~16.
3. Deepavali Bhagwat, Kave Eshghi, Darrell D.E. Long and Mark Lillibridge. Extreme Binning: Scalable, Parallel Deduplication for Chunk based File Backup [C]. In Proceedings of the 17th IEEE/ACM International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS'2009), Piscataway, NJ, USA: IEEE, 2009:1~9.
4. Mark Lillibridge, Kave Eshghi, Deepavali Bhagwat, Vinay Deolalikar, Greg Trezise, Peter Camble. Sparse indexing: large scale, inline deduplication using sampling and locality [C]. In Proceedings of the 7th Conference on File and Storage Technologies (FAST'09), Berkeley, CA, USA: USENIX, 2009: 111~123.
5. Wen Xia, Hong Jiang, Dan Feng, Yu Hua. SiLo: a Similarity-locality based Near-exact Deduplication Scheme with Low RAM Overhead and High Throughput [C]. In Proceedings of 2011 USENIX Annual Technical Conference (ATC'11), Berkeley, CA, USA: USENIX, 2011: 285~298.
6. Cezary Dubnicki, Leszek Gryz, Lukasz Heldt, Michal Kaczmarczyk, Wojciech Kilian, Przemyslaw Strzelczak, Jerzy Szczepkowski, Cristian Ungureanu, Michal Welnicki. HYDRAstor: a Scalable Secondary Storage [C]. In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST'09), Berkeley, CA, USA: USENIX, 2009:197~210.
7. Wei Dong, Fred Douglis, Kai Li, Hugo Patterson, Sazzala Reddy, Philip Shilane. Tradeoffs in Scalable Data Routing for Deduplication Clusters [C]. In Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST'11), Berkeley, CA, USA: USENIX, 2011: 15~29.
8. Tianming Yang, Hong Jiang, Dan Feng, Zhongying Niu, Ke Zhou, Y. Wan. DEBAR: a Scalable High-Performance Deduplication Storage System for Backup and Archiving [C]. In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS'10), Piscataway, NJ, USA: IEEE, 2010: 1~12.
9. Davide Frey, Anne-Marie Kermarrec, Konstantinos Kloudas. Probabilistic Deduplication for Cluster-Based Storage Systems [C]. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SOCC'12), San Jose: ACM, 2012: 1~19.
10. Y. Fu, H. Jiang, N. Xiao, L. Tian, F. Liu, "AA-Dedupe: An Application-Aware Source Deduplication Approach for Cloud Backup Services in the Personal Computing Environment," Proc. of the 13th IEEE International Conference on Cluster Computing (Cluster'11), pp. 112-120, 2011.
11. Yinjin Fu, Hong Jiang, Nong Xiao. A Scalable Inline Cluster Deduplication Framework for Big Data Protection. The ACM/IFIP/USENIX 13th International Conference on Middleware (Middleware'12), pages 354-373, 2012.
12. FIU IODedup Traces. http://iotta.snia.org/traces/391
13. Linux Kernel Achives. http://www.kernel.org/
14. Benjamin Zhu, Kai Li, Hugo Patterson. Avoiding the Disk Bottleneck in the Data Domain Deduplication File System [C]. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST'08), Berkeley, CA, USA: USENIX, 2008: 269~282.
15. Fanglu Guo, Petros Efstathopoulos. Buliding a High-performance Deduplication System [C]. In Proceedings of the 2011 USENIX Annual Technical Conference, June 2011.