The design of a similarity based deduplication system

ACM International Conference Proceeding Series

Volumn , Issue , 2009, Pages 6-

  Aronovich, Lior ^a   Asher, Ron ^a   Bachmat, Eitan ^b   Bitner, Haim ^c   Hirsch, Michael ^a   Klein, Shmuel T ^d  

^a IBM   (United States)

^b BEN GURION UNIVERSITY OF THE NEGEV   (Israel)

^c Marvell Corp   (United States)

^d BAR ILAN UNIVERSITY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

DEDUPLICATION; DESIGN GOAL; IN-LINE; SIMILARITY-MATCHING;

EID: 71049166051     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1534530.1534539     Document Type: Conference Paper

References (30)

1. Amir A., Lewenstein M., Porat E., Faster algorithms for string matching with k mismatches, Journal of Algorithms 50(2) (2004) 257-275.
2. Burton H. Bloom. Space/time Trade-offs in Hash Coding with Allowable Errors. Communications of the ACM, 13 (7). 422-426.
3. D.R. Bobbarjung, D, Jagannathan and C. Dubnicki, Improving duplicate elimination in storage systems, ACM Trans. on Storage, Vol.2(4), 424-448, 2006.
4. Boyer R. S., Moore J. S., A fast string searching algorithm, Communications of the ACM 20 (1977) 762-772.
5. S. Brin, J. Davis, H. Garcia-Molina. Copy Detection Mechanisms for Digital Documents (weblink). 1994, also in Proceedings of ACM SIGMOD, 1995.
6. A.Z. Broder. Identifying and Filtering Near-Duplicate Documents. CPM 2000: 1-10
7. A.Z. Broder. Some applications of Rabin's fingerprint ing method. In R. Capocelli, A. De Santis, and U. Vaccaro, editors, Sequences II: Methods in Communications, Security, and Computer Science, 143-152. Springer-Verlag, 1993.
8. A.Z. Broder. On the resemblance and containment of documents. Proceedings of Compression and Complexity of Sequences 1997, 21-29. IEEE Computer Society, 1997.
9. A.Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-Wise Independent Permutations. Proceedings of the 30th Annual ACM Symposium on Theory of Computing, 327-336, 1998.
10. A.Z. Broder and U. Feige. Min-Wise versus Linear Independence. Proc. of the Eleventh Annual ACMSIAM Symp. on Discrete Algorithms, 147-154, 2000.
11. A.Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the Web. Proceedings of the Sixth International World Wide Web Conference, pages 391-404, 1997.
12. Fischer M. J., Paterson M. S., String matching and other products, in Complexity of Computation, R. M. Karp (editor), SIAM-AMS Proc. 7 (1974) 113-125.
13. B. Garret and C. Bouffard, The Enterprise Strategy Group (ESG) lab validation report for IBM TS7650G ProtecTier, 2008. Available at www.diligent.com.
14. N. Heintze. Scalable Document Fingerprinting. Proceedings of the Second USENIX Workshop on Electronic Commerce, pages 191-200, 1996.
15. N. Jain, M. Dahlin, and R. Tewari. TAPER: Tiered Approach for Eliminating Redundancy in Replica Synchronization. Proceedings of USENIX File And Storage Systems conference (FAST), 2005.
16. M. Lillibridge, K. Eshghi, D. Bhagwat, V. Deolalikar, G. Trezise, and P. Camble, Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality, Proceedings of USENIX File And Storage Systems conference (FAST), 2009.
17. Karp R., Rabin M., Efficien randomized pattern matching algorithms, IBM Journal of Research and Development 31 (1987) 249-260.
18. Knuth D. E., Morris J. H., Pratt V. R., Fast pattern matching in strings, SIAM Journal on Computing 6 (1977) 323-350.
19. P. Kulkarni, F. Douglis, J. D. LaVoie, J. M. Tracey: Redundancy Elimination Within Large Collections of Files. Proceedings of USENIX Annual Technical Conference, pages 59-72, 2004.
20. Udi Manber. Finding Similar Files in A Large File System. Technical Report TR 93-33, Department of Computer Science, University of Arizona, October 1993, also in Proceedings of the USENIX Winter 1994 Technical Conference, pages 17-21. 1994.
21. Landau G. M., Vishkin U., Fast parallel and serial approximate string matching, Journal of Algorithms 10(2) (1989) 157-169.
22. Moulton G. H., Whitehill S. B., Hash fil system and method for use in a commonality factoring system, U.S. Pat. No.6,704,730.
23. Navarro G., A Guided Tour to Approximate String Matching, ACM Computing Surveys, 33(1) (2001) 31-88. (Pubitemid 33768480)
24. S. Quinlan and S. Dorward, Venti: A New Approach to Archival Storage. In Proc. of the USENIX Conf. on File And Storage Technologies (FAST), 2002.
25. N. Shivakumar and H. Garcia-Molina. Building a Scalable and Accurate Copy Detection Mechanism. Proceedings of the 3nd International Conference on Theory and Practice of Digital Libraries, 1996.
26. N. Shivakumar and H. Garcia-Molina. Finding nearreplicas of documents on the web. In Proc. of Workshop on Web Databases (WebDB'98), March 1998.
27. Ukkonen E., On-line construction of suffi trees, Algorithmica 14(3) (1995) 249-260.
28. Weiner P., Linear pattern matching algorithm, Proceedings of the 14th IEEE Symposium on Switching and Automata Theory, (1973) 1-11.
29. B. Zhu, K. Li and H. Patterson, Avoiding the Disk Bottleneck in the Data Domain Deduplication File System, Proc. of FAST 08, the 6th USENIX Conf. on File and Storage Technologies, 279-292.
30. J. Ziv and A. Lempel. A universal algorithm for sequential data compression, IEEE Trans. Inform. Theory, vol.IT-23, pp. 337-343, May 1977. 282