Tradeoffs in scalable data routing for deduplication clusters

Proceedings of FAST 2011: 9th USENIX Conference on File and Storage Technologies

Volumn , Issue , 2011, Pages 15-29

  Dong, Wei ^a   Douglis, Fred ^b   Reddy, Sazzala ^b   Li, Kai ^a,b   Shilane, Philip ^b   Patterson, Hugo ^b  

^a PRINCETON UNIVERSITY   (United States)

^b EMC

Author keywords

[No Author keywords available]

Indexed keywords

COMMERCE; LARGE DATASET;

CLUSTER-BASED; COMMERCIAL SYSTEMS; DATA CENTERS; DE DUPLICATIONS; HIGH THROUGHPUT; LARGE CLUSTERS; STORAGE SYSTEMS; STORAGE UTILIZATION;

DIGITAL STORAGE;

EID: 85077072489     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. A. Adya, W. J. Bolosky, M. Castro, G. Cermak, R. Chaiken, J. R. Douceur, J. Howell, J. R. Lorch, M. Theimer, and R. P. Wattenhofer. Farsite: federated, available, and reliable storage for an incompletely trusted environment. In OSDI'02: Proceedings of the 5th Symposium on Operating Systems Design and Implementation, pages 1-14, 2002.
2. D. Bhagwat, K. Eshghi, D. D. Long, and M. Lillibridge. Extreme binning: scalable, parallel deduplication for chunk-based file backup. In MASCOTS 09: Proceedings of the 17th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, Sept. 2009.
3. B. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422-426, July 1970.
4. S. Brin, J. Davis, and H. García-Molina. Copy detection mechanisms for digital documents. In Proceedings of ACM SIGMOD International Conference on Management of Data, pages 398-409, 1995.
5. P. H. Carns, W. B. Ligon, III, R. B. Ross, and R. Thakur. Pvfs: a parallel file system for linux clusters. In Proceedings of the 4th Annual Linux Showcase and Conference, pages 391-430. MIT Press, 2000.
6. L. P. Cox., C. D. Murray, and B. D. Noble. Pastiche: making backup cheap and easy. In OSDI'02: Proceedings of the 5th Symposium on Operating Systems Design and Implementation, pages 285-298, New York, NY, USA, 2002. ACM.
7. M. Dewaikar. Symantec NetBackup PureDisk: optimizing backups with deduplication for remote offices, data center and virtual machines. http://eval.symantec.com/mktginfo/enterprise/white_papers/b-symantec_ne% tbackup_puredisk_WP-en-us.pdf, September 2009.
8. C. Dubnicki, G. Leszek, H. Lukasz, M. Kaczmarczyk, W. Kilian, P. Strzelczak, J. Szczepkowski, C. Ungureanu, and M. Welnicki. HYDRAstor: a scalable secondary storage. In FAST'09: Proceedings of the 7th conference on File and Storage Technologies, pages 197-210, February 2009.
9. EMC Corporation. Data Domain products. http://www.datadomain.com/products/, 2009.
10. EMC Corporation. DD880: deduplication storage for the core data center. http://www.datadomain.com/pdf/DataDomain-DD880-Datasheet.pdf, 2009.
11. EMC Corporation. Efficient data protection with EMC Avamar global deduplication software. http://www.emc.com/collateral/software/white-papers/ h2681-efdta-prot-av%amar.pdf, July 2009.
12. EMC Corporation. Data Domain Boost Software, 2010. http://www.datadomain.com/products/dd-boost.html.
13. EMC Corporation. Data Domain Global Deduplication Array, 2010. http://www.datadomain.com/products/ global-deduplication-array.html.
14. European Parliament. Directive 2006/24/EC”On the retention of data generated or processed in connection with the provision of publicly available electronic communications services or of public communication networks”, March 2006.
15. J. F. Gantz, C. Chute, A. Manfrediz, S. Minton, D. Reinsel, W. Schlichting, and A. Toncheva. The diverse and exploding digital universe: an updated forecast of worldwide information growth through 2011. An IDC White Paper — sponsored by EMC, March 2008.
16. IBM Corporation. IBM ProtecTIER Deduplication Solutions, 2010. http://www-03.ibm.com/systems/storage/tape/protectier.
17. N. Jain, M. Dahlin, and R. Tewari. Taper: tiered approach for eliminating redundancy in replica synchronization. In FAST'05: Proceedings of the 4th USENIX Conference on File and Storage Technologies, pages 21-21, 2005.
18. E. Kruus, C. Ungureanu, and C. Dubnicki. Bimodal content defined chunking for backup streams. In FAST'10: Proceedings of the 8th Conference on File and Storage Technologies, February 2010.
19. P. Kulkarni, F. Douglis, J. LaVoie, and J. M. Tracey. Redundancy elimination within large collections of files. In Proceedings of the USENIX Annual Technical Conference, pages 59-72, 2004.
20. M. Lillibridge, K. Eshghi, D. Bhagwat, V. Deolalikar, G. Trezise, and P. Camble. Sparse indexing: large scale, inline deduplication using sampling and locality. In FAST'09: Proceedings of the 7th Conference on File and Storage Technologies, pages 111-123, 2009.
21. The Lustre File System, 2010. http://www.lustre.org.
22. U. Manber. Finding similar files in a large file system. In Proceedings of the USENIX Winter Technical Conference, pages 1-10, 1994.
23. D. T. Meyer and W. J. Bolosky. A Study of Practical Deduplication. In FAST'11: Proceedings of the 9th Conference on File and Storage Technologies, February 2011.
24. A. Muthitacharoen, B. Chen, and D. Mazières. A low-bandwidth network file system. In SOSP'01: Proceedings of the 18th ACM Symposium on Operating Systems Principles, pages 174-187, 2001.
25. Network Appliance. NetApp ONTAP. http://www.netapp.com/us/products/platform-os/dedupe.html, 2009.
26. C. Policroniades and I. Pratt. Alternatives for detecting redundancy in storage systems data. In Proceedings of the USENIX Annual Technical Conference, pages 73-86, 2004.
27. S. Quinlan and S. Dorward. Venti: a new approach to archival storage. In FAST'02: Proceedings of the 1st USENIX conference on File and Storage Technologies, 2002.
28. S. C. Rhea, R. Cox, and A. Pesterev. Fast, inexpensive content-addressed storage in foundation. In Proceedings of the USENIX Annual Technical Conference, pages 143-156, 2008.
29. 107th Congress, United States of America. Public Law 107-204:”Sarbanes-Oxley Act of 2002”, July 2002.
30. S. R. Soltis, T. M. Ruwart, and M. T. Okeefe. The global file system. In MSS'96: Proceedings of the 5th NASA Goddard Conference on Mass Storage, pages 319-342, 1996.
31. M. Stonebraker. The case for shared-nothing. IEEE Database Engineering, 9(1), March 1986.
32. N. Tolia, M. Kozuch, M. Satyanarayanan, B. Karp, T. Bressoud, and A. Perrig. Opportunistic Use of Content Addressable Storage for Distributed File Systems. In Proceedings of the USENIX Annual Technical Conference, pages 127-140, 2003.
33. C. Ungureanu, B. Atkin, A. Aranya, S. Gokhale, S. Rago, G. Calkowski, C. Dubnicki, and A. Bohra. HydraFS: a high throughput file system for the HYDRAstor content-addressable storage system. In FAST'10: Proceedings of the 8th Conference on File and Storage Technologies, February 2010.
34. T. Yang, D. Feng, Z. Niu, K. Zhou, and Y. Wan. DEBAR: a scalable high-performance deduplication storage system for backup and archiving. In IEEE International Symposium on Parallel & Distributed Processing, May 2010.
35. L. You and C. Karamanolis. Evaluation of efficient archival storage techniques. In MSS'04: Proceedings of the 21st Symposium on Mass Storage Systems, Apr. 2004.
36. B. Zhu, K. Li, and H. Patterson. Avoiding the disk bottleneck in the Data Domain deduplication file system. In FAST'08: Proceedings of the 6th Conference on File and Storage Technologies, pages 269-282, February 2008.
37. J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337-343, May 1977.