Bimodal content defined chunking for backup streams

Proceedings of FAST 2010: 8th USENIX Conference on File and Storage Technologies

Volumn , Issue , 2010, Pages 239-252

  Kruus, Erik ^a   Ungureanu, Cristian ^a   Dubnicki, Cezary ^b  

^a NEC LABORATORIES AMERICA   (United States)

^b 9LivesData

Author keywords

[No Author keywords available]

Indexed keywords

CODES (SYMBOLS); DIGITAL STORAGE; METADATA; MULTIPROCESSING SYSTEMS;

DATA DE DUPLICATIONS; DISTRIBUTED STORAGE SYSTEM; DUPLICATE ELIMINATION; ERROR CORRECTING CODE; REDUNDANCY TECHNIQUES; STANDARD ALGORITHMS; SYSTEM'S PERFORMANCE; VARIABLE SIZES;

DATA STREAMS;

EID: 85077053929     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. AGARWAL, R. C. Method and computer program product for finding the longest common subsequences between files with applications to differential compression. United States Patent 20060112264, May 2006.
2. ANNAPUREDDY, S., FREEDMAN, M., AND MAZIÈRES, D. Shark: Scaling File Servers via Cooperative Caching. In NSDI '05 Paper [NSDI'05 Technical Program] (2005).
3. BARRETO, J., AND FERREIRA, P. A replicated file system for resource constrained mobile devices. In Proceedings of IADIS International Conference on Applied Computing (2004).
4. BHAGWAT, D., ESHGHI, K., LONG, D. D. E., AND LILLIBRIDGE, M. Extreme binning: Scalable, parallel deduplication for chunk-based file backup. In Proceedings of the 17th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2009) (Sept. 2009).
5. BOBBARJUNG, D. R., JAGANNATHAN, S., AND DUBNICKI, C. Improving duplicate elimination in storage systems. Trans. Storage 2, 4 (2006), 424-448.
6. BRIN, S., DAVIS, J., AND GARCIA-MOLINA, H. Copy detection mechanisms for digital documents. In In Proceedings of the ACM SIGMOD Annual Conference (1995), pp. 398-409.
7. BRODER, A. Some applications of Rabin's fingerprinting method. Sequences II: Methods in Communications, Security, and Computer Science (1993), 143-152.
8. CHOWDHURY, A., FRIEDER, O., GROSSMAN, D., AND MCCABE, M. C. Collection statistics for fast duplicate document detection. ACM Trans. Inf. Syst. 20, 2 (2002), 171-191.
9. DENEHY, T., AND HSU, W. Duplicate management for reference data. Technical report RJ 10305, IBM Research, October 2003.
10. DOUGLIS, F., AND IYENGAR, A. Application-specific Delta-encoding via Resemblance Detection. In Proceedings of the USENIX Annual Technical Conference (2003).
11. DOUGLIS, F., KULKARNI, P., LAVOIE, J. D., AND TRACEY, J. M. Method and apparatus for data redundancy elimination at the block level. United States Patent 20050131939, June 2005.
12. DUBNICKI, C., GRYZ, L., HELDT, L., KACZMARCZYK, M., KILIAN, W., STRZELCZAK, P., SZCZEPKOWSKI, J., UNGUREANU, C., AND WELNICKI, M. HYDRAstor: a Scalable Secondary Storage. In Proccedings of the 7th conference on File and storage technologies (2009), USENIX Association, pp. 197-210.
13. ESHGHI, K., AND TANG, H. K. A framework for analyzing and improving content-based chunking algorithms. Technical report HPL-2005-30R1, HP Laboratories, 10 2005.
14. FORMAN, G., ESHGHI, K., AND CHIOCCHETTI, S. Finding similar files in large document repositories. In KDD'05: Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining (New York, NY, USA, 2005), pp. 394-400.
15. GUREVICH, Y., BJORNER, N. S., AND TEODOSIU, D. Efficient chunking algorithm. United States Patent 20060047855, March 2006.
16. HUA, N., ZHAO, H., LIN, B., AND XU, J. Rank-indexed hashing: A compact construction of bloom filters and variants. In IEEE International Conference on Network Protocols (ICNP 2008) (Oct. 2008), pp. 73-82.
17. JAIN, N., DAHLIN, M., AND TEWARI, R. TAPER: Tiered Approach for Eliminating Redundancy in Replica Sychronization. In USENIX Conference on File and Storage Technologies (FAST05) (Dec 2005).
18. JAIN, N., DAHLIN, M., AND TEWARI, R. TAPER: Tiered Approach for Eliminating Redundancy in Replica Synchronization. Tech. rep., Technical Report TR-05-42, Dept. of Comp. Sc., Univ. of Texas at Austin, 2005.
19. KANIZO, Y., HAY, D., AND KESLASSY, I. Optimal fast hashing. In 28th IEEE International Conference on Computer Communications (INFOCOM) (Apr. 2009), pp. 2500-2508.
20. KULKARNI, P., DOUGLIS, F., LAVOIE, J., AND TRACEY, J. Redundancy Elimination Within Large Collections of Files. In Proceedings of the USENIX Annual Technical Conference (2004).
21. LILLIBRIDGE, M., ESHGHI, K., BHAGWAT, D., DEOLALIKAR, V., TREZISE, G., AND CAMBLE, P. Sparse indexing: large scale, inline deduplication using sampling and locality. In Proceedings of the 7th conference on File and storage technologies (2009), USENIX Association, pp. 111-123.
22. LUFEI, H., SHI, W., AND ZAMORANO, L. On the effects of bandwidth reduction techniques in distributed applications. Proceedings of International Conference on Embedded and Ubiquitous Computing (EUC'04) (2004).
23. LUMETTA, S., AND MITZENMACHER, M. Using the power of two choices to improve bloom filters. Internet Mathematics 4, 1 (2007), 17-34.
24. MOULTON, G. H. System and method for unorchestrated determination of data sequences using sticky byte factoring to determine breakpoints in digital sequences. United States Patent 6810398, October 2004.
25. MUTHITACHAROEN, A., CHEN, B., AND MAZIÈRES, D. A low-bandwidth network file system. In SOSP'01: Proceedings of the eighteenth ACM symposium on Operating systems principles (New York, NY, USA, 2001), pp. 174-187.
26. ´ Proceedings of the USENIX Annual Technical Conference (2004).
27. PORTS, D. R. K., CLEMENTS, A. T., AND DEMAINE, E. D. PersiFS: a versioned file system with an efficient representation. In SOSP'05: Proceedings of the twentieth ACM symposium on Operating systems principles (New York, NY, USA, 2005), pp. 1-2.
28. PUGH, W., AND HENZINGER, M. H. Detecting duplicate and near-duplicate files. United States Patent 6658423, December 2003.
29. RABIN, M. Fingerprinting by random polynomials. Technical report TR-15-81, Harvard University, 2003.
30. SCHLEIMER, S., WILKERSON, D. S., AND AIKEN, A. Winnowing: local algorithms for document fingerprinting. In SIGMOD'03: Proceedings of the 2003 ACM SIGMOD international conference on Management of data (New York, NY, USA, 2003), pp. 76-85.
31. SPIRIDONOV, A., THAKER, S., AND PATWARDHAN, S. Sharing and bandwidth consumption in the low bandwidth file system. Tech. rep., Department of Computer Science, University of Texas at Austin, 2005.
32. SUEL, T., NOEL, P., AND TRENDAFILOV, D. Improved File Synchronization Techniques for Maintaining Large Replicated Collections over Slow Networks. In ICDE'04: Proceedings of the 20th International Conference on Data Engineering (Washington, DC, USA, 2004), p. 153.
33. TRIDGELL, A. Efficient Algorithms for Sorting and Synchronization. PhD thesis, Australian National University, April 2000.
34. TRIDGELL, A., AND MACKERRAS, P. The rsync algorithm. Technical report TR-CS-96-05, Australian National University, Deparment of Computer Science, FEIT, ANU, 1996.
35. YANG, T., JIANG, H., FENG, D., AND NIU, Z. DEBAR: A Scalable High-Performance De-duplication Storage System for Backup and Archiving. CSE Technical reports (2009), 58.
36. YOU, L., AND KARAMANOLIS, C. Evaluation of efficient archival storage techniques. In Proceedings of 21st IEEE/NASA Goddard MSS (2004).
37. YOU, L. L., POLLACK, K. T., AND LONG, D. D. E. Deep Store: An Archival Storage System Architecture. In ICDE'05: Proceedings of the 21st International Conference on Data Engineering (Washington, DC, USA, 2005), pp. 804-8015.
38. ZHU, B., LI, K., AND PATTERSON, H. Avoiding the disk bottleneck in the data domain deduplication file system. In FAST'08: Proceedings of the 6th USENIX Conference on File and Storage Technologies (Berkeley, CA, USA, 2008), USENIX Association, pp. 1-14.
39. ZHU, Y., JIANG, H., AND WANG, J. Hierarchical Bloom filter arrays (HBA): a novel, scalable metadata management system for large cluster-based storage. In Cluster Computing, 2004 IEEE International Conference on (Sept. 2004), pp. 165-174.