WAN-optimized replication of backup datasets using stream-informed delta compression

ACM Transactions on Storage

Volumn 8, Issue 4, 2012, Pages

  Shilane, Phlip ^a   Huang, Mark ^a   Wallace, Grant ^a   Hsu, Windsor ^a  

^a GNSS Technology for Rail Transportation   (United States)

Author keywords

Backup storage; Deduplication; Delta compression; Network replication

Indexed keywords

DATA DOMAINS; DATA SETS; DEDUPLICATION; DELTA COMPRESSION; DISASTER RECOVERY; ERROR PRONES; LOCAL COMPRESSION; NETWORK CONNECTION; REMOTE LOCATION;

DIGITAL STORAGE; DISTRIBUTED COMPUTER SYSTEMS; WIDE AREA NETWORKS;

DATA COMPRESSION;

EID: 84871214454     PISSN: 15533077     EISSN: 15533093     Source Type: Journal    
DOI: 10.1145/2385603.2385606     Document Type: Conference Paper

References (45)

1. ARONOVICH, L., ASHER, R., BACHMAT, E., BITNER, H.,HIRSCH, M., AND KLEIN, S. T. 2009. The design of a similarity based deduplication system. In Proceedings of the Israeli Experimental Systems Conference (SYSTOR '09). ACM, New York, 6:1-6:14.
2. BHAGWAT, D., ESHGHI, K., LONG, D. D., AND LILLIBRIDGE, M. 2009. Extreme binning: scalable, parallel deduplication for chunk-based file backup. In Proceedings of the 17th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.
3. BOBBARJUNG, D. R., JAGANNATHAN, S., AND DUBNICKI, C. 2006. Improving duplicate elimination in storage systems. Trans. Storage 2, 424-448.
4. BRIN, S., DAVIS, J., AND GARĆIA-MOLINA, H. 1995. Copy detection mechanisms for digital documents. In Proceedings of ACM SIGMOD International Conference on Management of Data. 398-409.
5. BRODER, A. 1997. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences. 21.
6. BRODER, A. 2000. Identifying and filtering near-duplicate documents. In Proceedings of the 11th Annual Symposium on Combinatorial Pattern Matching. 1-10.
7. BRODER, A. Z., CHARIKAR, M., FRIEZE, A. M., AND MITZENMACHER, M. 1998. Min-wise independent permutations (extended abstract). In Proceedings of the 30th Annual ACM Symposium on Theory of Computing. ACM, New York, 327-336.
8. BURNS, R. C. AND LONG, D. D. E. 1997. Efficient distributed backup with delta compression. In Proceedings of the 5th Workshop on I/O in Parallel and Distributed Systems. New York, 27-36.
9. CHAN, M. C. ANDWOO, T. Y. C. 1999. Cache-based compaction: a new technique for optimizing web transfer. In Proceedings of the 18th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM '99), 117-125.
10. CHEN, Y., QU, Z., ZHANG, Z., AND YEO, B.-L. 2004. Data redundancy and compression methods for a disk-based network backup system. In Proceedings of the International Conference on Information Technology: Coding and Computing. 778.
11. DEBNATH, B., SENGUPTA, S., AND LI, J. 2010. Chunkstash: speeding up inline storage deduplication using flash memory. In Proceedings of the USENIX Annual Technical Conference.
12. DONG, W., DOUGLIS, F., LI, K., PATTERSON, H., REDDY, S., AND SHILANE, P. 2011. Tradeoffs in scalable data routing for deduplication clusters. In Proceedings of the 9th USENIX Conference on File and Storage Technologies.
13. DOUGLIS, F. AND IYENGAR, A. 2003. Application-specific delta-encoding via resemblance detection. In Proceedings of the USENIX Annual Technical Conference. 113-126.
14. EMC CORPORATION. 2010. Data Domain Boost Software. http://www.datadomain. com/products/dd-boost.html.
15. ESHGHI, K., LILLIBRIDGE, M., WILCOCK, L., BELROSE, G., AND HAWKES, R. 2007. Jumbo store: providing efficient incremental upload and versioning for a utility rendering service. In Proceedings of the 5th USENIX Conference on File and Storage Technologies.
16. GAILLY, J. L. AND ADLER, M. 2003. The GZIP compressor. http://www.gzip.org.
17. GUO, F. AND EFSTATHOPOULOS, P. 2011. Building a high-performance deduplication system. In Proceedings of the USENIX Annual Technical Conference.
18. HUNT, J. J., VO, K.-P., AND TICHY, W. F. 1998. Delta algorithms: an empirical analysis. ACM Trans. Softw. Eng. Methodol. 7, 192-214.
19. JAIN, N., DAHLIN, M., AND TEWARI, R. 2005. Taper: tiered approach for eliminating redundancy in replica synchronization. In Proceedings of the 4th USENIX Conference on File and Storage Technologies.
20. KULKARNI, P.,DOUGLIS, F., LAVOIE, J., AND TRACEY, J. M. 2004. Redundancy elimination within large collections of files. In Proceedings of the USENIX Annual Technical Conference. 59-72.
21. LILLIBRIDGE, M., ESHGHI, K., BHAGWAT, D., DEOLALIKAR, V., TREZISE, G., AND CAMBLE, P. 2009. Sparse indexing: large scale, inline deduplication using sampling and locality. In Proceedings of the 7th USENIX Conference on File and Storage Technologies. 111-123.
22. MACDONALD, J. 2000. File system support for delta compression. M.S. thesis, Department of Electrical Engineering and Computer Science, University of California, Berkeley.
23. MANBER, U. 1994. Finding similar files in a large file system. In Proceedings of the USENIX Winter Technical Conference. 1-10.
24. MIN, J., YOON, D., AND WON, Y. 2010. Efficient deduplication techniques for modern backup operation. IEEE Trans. Comput. 60, 6, 824-840.
25. MOGUL, J. C., DOUGLIS, F., FELDMANN, A., AND KRISHNAMURTHY, B. 1997. Potential benefits of delta encoding and data compression for http. In Proceedings of the ACM SIGCOMM 1997 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication. 181-194.
26. MUTHITACHAROEN, A., CHEN, B., AND MAZÌERES, D. 2001. A low-bandwidth network file system. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP '01). 174-187.
27. PARK, K., IHM, S., BOWMAN, M., AND PAI, V. S. 2007. Supporting practical content-addressable caching with CZIP compression. In Proceedings of the USENIX Annual Technical Conference. 14:1-14:14.
28. PARK, N. AND LILJA, D. 2010. Characterizing datasets for data deduplication in backup applications. In Proceedings of the IEEE International Symposium on Workload Characterization.
29. PATTERSON, H.,MANLEY, S., FEDERWISCH, M.,HITZ, D.,KLEIMAN, S., AND OWARA, S. 2002. Snapmirror: file system based asynchronous mirroring for disaster recovery. In Proceedings of the 1st USENIX Conference on File and Storage Technologies.
30. POLICRONIADES, C. AND PRATT, I. 2004. Alternatives for detecting redundancy in storage systems data. In Proceedings of the USENIX Annual Technical Conference. 73-86.
31. QUINLAN, S. AND DORWARD, S. 2002. Venti: a new approach to archival storage. In Proceedings of the 1st USENIX Conference on File and Storage Technologies.
32. RABIN, M. O. 1981. Fingerprinting by random polynomials. Tech. rep., Center for Research in Computing Technology.
33. RIVERBED TECHNOLOGY. 2011. Riverbed Steelhead Product Family. http://www.riverbed.com/us/assets/media/documents/data sheets/DataSheet- Riverbed-FamilyProduct.pdf.
34. SHILANE, P.,HUANG, M., WALLACE, G., AND HSU.W. 2012a. WAN optimized replication of backup datasets using stream-informed delta compression. In Proceedings of the 10th USENIX Conference on File and Storage Technologies.
35. SHILANE, P., WALLACE, G., HUANG, M., AND HSU, W. 2012b. Delta compressed and deduplicated storage using stream-informed locality. In Proceedings of the 4th USENIX Conference on Hot Topics in Storage and File Systems.
36. SPRING, N. T. AND WETHERALL, D. 2000. A protocol-independent technique for eliminating redundant network traffic. In Proceedings of the ACM SIGCOMM 2000 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication. 87-95.
37. SUEL, T. AND MEMON, N. 2002. Algorithms for delta compression and remote file synchronization. In Lossless Compression Handbook, K. Sayood, Ed. Academic Press, San Diego, CA.
38. SUEL, T., NOEL, P., AND TRENDAFILOV, D. 2004. Improved file synchronization techniques for maintaining large replicated collections over slow networks. In Proceedings of the 20th International Conference on Data Engineering.
39. TRENDAFILOV, D.,MEMON, N., AND SUEL, T. 2002. Zdelta: An efficient delta compression tool. Tech. rep., Department of Computer and Information Science, Polytechnic University.
40. TRIDGELL, A. 2000. Efficient algorithms for sorting and synchronization. Ph.D. thesis, Australian National University.
41. WALLACE, G.,DOUGLIS, F.,QIAN, H., SHILANE, P., SMALDONE, S., CHAMNESS, M., AND HSU, W. 2012. Characteristics of backup workloads in production systems. In Proceedings of the 10th USENIX Conference on File and Storage Technologies.
42. XIA, W., JIANG, H., FENG, D., AND HUA, Y. 2011. Silo: a similarity-locality based near-exact deduplication scheme with low ram overhead and high throughput. In Proceedings of the USENIX Annual Technical Conference.
43. YOU, L. AND KARAMANOLIS, C. 2004. Evaluation of efficient archival storage techniques. In Proceedings of the 21st Symposium on Mass Storage Systems.
44. YOU, L., POLLACK, K., LONG, D. D. E., AND GOPINATH, K. 2011. Presidio: a framework for efficient archival data storage. ACM Trans. Storage 7, 2.
45. ZHU, B., LI, K., AND PATTERSON, H. 2008. Avoiding the disk bottleneck in the Data Domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies. 269-282.