Building a high-performance deduplication system

Proceedings of the 2011 USENIX Annual Technical Conference, USENIX ATC 2011

Volumn , Issue , 2019, Pages 271-284

  Guo, Fanglu ^a   Efstathopoulos, Petros ^a  

^a SYMANTEC RESEARCH LABS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLIENT-SERVER INTERACTIONS; DUPLICATE DETECTION; EFFECTIVE CAPACITY; EFFICIENCY OPTIMIZATION; ENERGY REQUIREMENTS; ENGINEERING TECHNIQUES; PROTOTYPE IMPLEMENTATIONS; SINGLE-NODE PERFORMANCE;

EID: 85077080685     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. ADYA, A., BOLOSKY, W. J., CASTRO, M., CERMAK, G., CHAIKEN, R., DOUCEUR, J. R., HOWELL, J., LORCH, J. R., THEIMER, M., AND WATTENHOFER, R. P. Farsite: federated, available, and reliable storage for an incompletely trusted environment. SIGOPS Oper. Syst. Rev. 36, SI (2002), 1-14.
2. BRODER, A. Z. Some applications of rabin's fingerprinting method. In Sequences II: Methods in Communications, Security, and Computer Science (1993), Springer-Verlag, pp. 143-152.
3. DATADOMAIN. EMC data domain dd890. http://bit.ly/ eXL6Uc.
4. DUBNICKI, C., GRYZ, L., HELDT, L., KACZMARCZYK, M., KILIAN, W., STRZELCZAK, P., SZCZEPKOWSKI, J., UNGUREANU, C., AND WELNICKI, M. Hydrastor: a scalable secondary storage. In FAST'09: Proccedings of the 7th conference on File and storage technologies (Berkeley, CA, USA, 2009), USENIX Association, pp. 197-210.
5. FORMAN, G., ESHGHI, K., AND CHIOCCHETTI, S. Finding similar files in large document repositories. In KDD'05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining (New York, NY, USA, 2005), ACM, pp. 394-400.
6. GREENBYTES. GB-X Series. http://bit.ly/fTOSXd.
7. HP. StorageWorks D2D Backup Systems. http://bit.ly/ bfhyiU.
8. IBM. ProtecTIER Deduplication Solutions. http://bit.ly/dVg3Z5.
9. JIN, K., AND MILLER, E. L. The effectiveness of deduplication on virtual machine disk images. In SYSTOR'09: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference (New York, NY, USA, 2009), ACM, pp. 1-12.
10. KRUUS, E., UNGUREANU, C., AND DUBNICKI, C. Bimodal content defined chunking for backup streams. In FAST'10: Proceedings of the 8th USENIX conference on File and storage technologies (Berkeley, CA, USA, 2010), USENIX Association, pp. 18-18.
11. KULKARNI, P., DOUGLIS, F., LAVOIE, J., AND TRACEY, J. M. Redundancy elimination within large collections of files. In ATEC'04: Proceedings of the annual conference on USENIX Annual Technical Conference (Berkeley, CA, USA, 2004), USENIX Association, pp. 5-5.
12. LILLIBRIDGE, M., ESHGHI, K., BHAGWAT, D., DEOLALIKAR, V., TREZISE, G., AND CAMBLE, P. Sparse indexing: large scale, inline deduplication using sampling and locality. In FAST'09: Proccedings of the 7th conference on File and storage technologies (Berkeley, CA, USA, 2009), USENIX Association, pp. 111-123.
13. LIU, C., XUE, Y., JU, D., AND WANG, D. A novel optimization method to improve de-duplication storage system performance. In ICPADS'09: Proceedings of the 2009 15th International Conference on Parallel and Distributed Systems (Washington, DC, USA, 2009), IEEE Computer Society, pp. 228-235.
14. NEC. HYDRAstor HS8-3000 Series. http://bit.ly/ hCHxhn.
15. NETAPP. Deduplicating Backup Data Streams with the NetApp VTL, 2009. http://bit.ly/buXcaV.
16. QUINLAN, S., AND DORWARD, S. Venti: A new approach to archival storage. In FAST'02: Proceedings of the Conference on File and Storage Technologies (Berkeley, CA, USA, 2002), USENIX Association, pp. 89-101.
17. UNGUREANU, C., ATKIN, B., ARANYA, A., GOKHALE, S., RAGO, S., CALKOWSKI, G., DUBNICKI, C., AND BOHRA, A. Hydrafs: a high-throughput file system for the hydrastor content-addressable storage system. In FAST'10: Proceedings of the 8th USENIX conference on File and storage technologies (Berkeley, CA, USA, 2010), USENIX Association, pp. 17-17.
18. WEI, J., JIANG, H., ZHOU, K., AND FENG, D. Mad2: A scalable high-throughput exact deduplication approach for network backup services. In MSST'10: Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies (2010), pp. 1-14.
19. ZHU, B., LI, K., AND PATTERSON, R. H. Avoiding the disk bottleneck in the datadomain deduplication file system. In FAST'08: Proceedings of the Conference on File and Storage Technologies (2008), pp. 269-282.