Wan optimized replication of backup datasets using stream-informed delta compression

Proceedings of FAST 2012: 10th USENIX Conference on File and Storage Technologies

Volumn , Issue , 2012, Pages 49-63

  Shilane, Philip ^a   Huang, Mark ^a   Wallace, Grant ^a   Hsu, Windsor ^a  

^a India Center of Excellence

Author keywords

[No Author keywords available]

Indexed keywords

DIGITAL STORAGE;

DATA DOMAINS; DE DUPLICATIONS; DELTA COMPRESSION; DISASTER RECOVERY; ERROR PRONES; LOCAL COMPRESSION; NETWORK CONNECTION; REMOTE LOCATION;

WIDE AREA NETWORKS;

EID: 85077115871     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (41)

1. L. Aronovich, R. Asher, E. Bachmat, H. Bitner, M. Hirsch, and S. T. Klein. The design of a similarity based deduplication system. In Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference, SYSTOR'09, pages 6:1-6:14, New York, NY, USA, 2009.
2. D. Bhagwat, K. Eshghi, D. D. Long, and M. Lillibridge. Extreme binning: scalable, parallel deduplication for chunk-based file backup. In Proceedings of the 17th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, Sept. 2009.
3. D. R. Bobbarjung, S. Jagannathan, and C. Dubnicki. Improving duplicate elimination in storage systems. Trans. Storage, 2:424-448, November 2006.
4. A. Broder. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences, page 21, 1997.
5. A. Broder. Identifying and filtering near-duplicate documents. In Proceedings of the 11th Annual Symposium on Combinatorial Pattern Matching, pages 1-10, 2000.
6. A. Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-wise independent permutations (extended abstract). In Proceedings of the 30th annual ACM symposium on Theory of computing, pages 327-336, New York, NY, USA, 1998.
7. R. C. Burns and D. D. E. Long. Efficient distributed backup with delta compression. In Proceedings of the 5th workshop on I/O in parallel and distributed systems, pages 27-36, New York, NY, USA, 1997.
8. M. C. Chan and T. Y. C. Woo. Cache-based compaction: a new technique for optimizing web transfer. In INFOCOM'99 conference, March 1999.
9. Y. Chen, Z. Qu, Z. Zhang, and B.-L. Yeo. Data redundancy and compression methods for a disk-based network backup system. International Conference on Information Technology: Coding and Computing, 1:778, 2004.
10. B. Debnath, S. Sengupta, and J. Li. Chunkstash: speeding up inline storage deduplication using flash memory. In Proceedings of the USENIX Annual Technical Conference, Berkeley, CA, USA, 2010.
11. W. Dong, F. Douglis, K. Li, H. Patterson, S. Reddy, and P. Shilane. Tradeoffs in scalable data routing for deduplication clusters. In Proceedings of the 9th USENIX Conference on File and Storage Technologies, 2011.
12. F. Douglis and A. Iyengar. Application-specific delta-encoding via resemblance detection. In Proceedings of the USENIX Annual Technical Conference, pages 113-126, 2003.
13. EMC Corporation. Data Domain Boost Software, 2010. http://www.datadomain.com/products/dd-boost.html.
14. K. Eshghi, M. Lillibridge, L. Wilcock, G. Belrose, and R. Hawkes. Jumbo store: Providing efficient incremental upload and versioning for a utility rendering service. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, 2007.
15. J. L. Gailly and M. Adler. The GZIP compressor. http://www.gzip.org.
16. F. Guo and P. Efstathopoulos. Building a high-performance deduplication system. In Proceedings of the USENIX Annual Technical Conference, 2011.
17. J. J. Hunt, K.-P. Vo, and W. F. Tichy. Delta algorithms: an empirical analysis. ACM Trans. Softw. Eng. Methodol., 7:192-214, April 1998.
18. N. Jain, M. Dahlin, and R. Tewari. Taper: tiered approach for eliminating redundancy in replica synchronization. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, 2005.
19. P. Kulkarni, F. Douglis, J. LaVoie, and J. M. Tracey. Redundancy elimination within large collections of files. In Proceedings of the USENIX Annual Technical Conference, pages 59-72, 2004.
20. M. Lillibridge, K. Eshghi, D. Bhagwat, V. Deolalikar, G. Trezise, and P. Camble. Sparse indexing: large scale, inline deduplication using sampling and locality. In Proceedings of the 7th USENIX Conference on File and Storage Technologies, pages 111-123, 2009.
21. J. MacDonald. File system support for delta compression. Master's thesis, Department of Electrical Engineering and Computer Science, University of California at Berkeley, 2000.
22. U. Manber. Finding similar files in a large file system. In Proceedings of the USENIX Winter Technical Conference, pages 1-10, 1994.
23. J. Min, D. Yoon, and Y. Won. Efficient deduplication techniques for modern backup operation. IEEE Transactions on Computers, 99, 2010.
24. J. C. Mogul, F. Douglis, A. Feldmann, and B. Krishnamurthy. Potential benefits of delta encoding and data compression for http. In Proceedings of the ACM SIGCOMM 1997 Conference, pages 181-194, 1997.
25. A. Muthitacharoen, B. Chen, and D. Mazières. A low-bandwidth network file system. In SOSP'01: Proceedings of the 18th ACM Symposium on Operating Systems Principles, pages 174-187, 2001.
26. K. Park, S. Ihm, M. Bowman, and V. S. Pai. Supporting practical content-addressable caching with CZIP compression. In Proceedings of the USENIX Annual Technical Conference, pages 14:1-14:14, Berkeley, CA, USA, 2007.
27. N. Park and D. Lilja. Characterizing datasets for data deduplication in backup applications. In IEEE International Symposium on Workload Characterization, 2010.
28. H. Patterson, S. Manley, M. Federwisch, D. Hitz, S. Kleiman, and S. Owara. Snapmirror: file system based asynchronous mirroring for disaster recovery. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Berkeley, CA, USA, 2002.
29. C. Policroniades and I. Pratt. Alternatives for detecting redundancy in storage systems data. In Proceedings of the USENIX Annual Technical Conference, pages 73-86, 2004.
30. S. Quinlan and S. Dorward. Venti: a new approach to archival storage. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, 2002.
31. M. O. Rabin. Fingerprinting by random polynomials. Technical report, Center for Research in Computing Technology, 1981.
32. Riverbed Technology. Riverbed Steelhead Product Family, 2011. http://www.riverbed.com/us/assets/media/documents/data_sheets/ DataSheet%-Riverbed-FamilyProduct.pdf.
33. N. T. Spring and D. Wetherall. A protocol-independent technique for eliminating redundant network traffic. In Proceedings of the ACM SIGCOMM 2000 Conference, pages 87-95, 2000.
34. T. Suel and N. Memon. Algorithms for delta compression and remote file synchronization. In K. Sayood, editor, Lossless Compression Handbook. 2002.
35. T. Suel, P. Noel, and D. Trendafilov. Improved file synchronization techniques for maintaining large replicated collections over slow networks. In 20th International Conference on Data Engineering, 2004.
36. D. Trendafilov, N. Memon, and T. Suel. zdelta: An efficient delta compression tool. Technical report, Department of Computer and Information Science at Polytechnic University, 2002.
37. A. Tridgell. Efficient algorithms for sorting and synchronization. PhD thesis, Australian National University, April 2000.
38. G. Wallace, F. Douglis, H. Qian, P. Shilane, S. Smaldone, M. Chamness, and W. Hsu. Characteristics of backup workloads in production systems. In Proceedings of the 10th USENIX Conference on File and Storage Technologies, 2012.
39. W. Xia, H. Jiang, D. Feng, and Y. Hua. Silo: A similarity-locality based near-exact deduplication scheme with low ram overhead and high throughput. In Proceedings of the USENIX Annual Technical Conference, 2011.
40. L. You and C. Karamanolis. Evaluation of efficient archival storage techniques. In Proceedings of the 21st Symposium on Mass Storage Systems, Apr. 2004.
41. B. Zhu, K. Li, and H. Patterson. Avoiding the disk bottleneck in the Data Domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies, pages 269-282, February 2008.