Sparse indexing: Large scale, inline deduplication using sampling and locality

Proceedings of the 7th USENIX Conference on File and Storage Technologies, FAST 2009

Volumn , Issue , 2009, Pages 111-123

  Lillibridge, Mark ^a   Eshghi, Kave ^a   Bhagwat, Deepavali ^b   Deolalikar, Vinay ^a   Trezise, Greg ^c   Camble, Peter ^c  

^a HEWLETT PACKARD LABORATORIES   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c HP Storage Works Division   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INDEXING (OF INFORMATION); MAPS;

BOTTLENECK PROBLEM; DE DUPLICATIONS; DISK-BASED; HEWLETT-PACKARD; INDEX MAPS; LOOKUPS; SAMPLING RATES;

RANDOM ACCESS STORAGE;

EID: 84933071250     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. ADYA, A., BOLOSKY, W. J., CASTRO, M., CERMAK, G., CHAIKEN, R., DOUCEUR, J. R., HOWELL, J., LORCH, J. R., THEIMER, M., AND WATTENHOFER, R. P. FARSITE: Federated, available, and reliable storage for an incompletely trusted environment. In Proc. of the 5th Symposium on Operating Systems Design and Implementation (OSDI) (Boston, MA, December 2002), pp. 1–14.
2. AJTAI, M., BURNS, R., FAGIN, R., LONG, D. D. E., AND STOCKMEYER, L. Compactly encoding unstructured inputs with differential compression. Journal of the Association for Computing Machinery 49, 3 (May 2002), 318–367.
3. ASARO, T., AND BIGGAR, H. Data De-duplication and Disk-to-Disk Backup Systems: Technical and Business Considerations. The Enterprise Strategy Group (July 2007).
4. BIGGAR, H. Experiencing Data De-Duplication: Improving Efficiency and Reducing Capacity Requirements. The Enterprise Strategy Group (Feb. 2007).
5. BINARY TESTING LTD. HP StorageWorks D2D4000 Backup System: a report and full performance test on Hewlett-Packard’s SME data deduplication appliance. Available at http://h18006.www1.hp.com/products/storageworks/d2d 4000/ relatedinfo.html, July 2008.
6. BLOOM, B. H. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM 13, 7 (1970), 422–426.
7. BRIN, S., DAVIS, J., AND GARCÍA-MOLINA, H. Copy detection mechanisms for digital documents. In SIGMOD’95: Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data (San Jose, California, United States, 1995), ACM Press, pp. 398–409.
8. BRODER, A. Z. On the resemblance and containment of documents. In SEQUENCES’97: Proceedings of the Compression and Complexity of Sequences 1997 (Washington, DC, USA, 1997), IEEE Computer Society, pp. 21–29.
9. CHANG, F., DEAN, J., GHEMAWAT, S., HSIEH, W. C., WALLACH, D. A., BURROWS, M., CHANDRA, T., FIKES, A., AND GRUBER, R. E. Bigtable: a distributed storage system for structured data. In OSDI’06: Proc. of the 7th USENIX Symposium on Operating Systems Design and Implementation (Seattle, WA, 2006), pp. 205–218.
10. CHOU, H.-T., AND DEWITT, D. J. An evaluation of buffer management strategies for relational database systems. In Proc. of the 11th Conference on Very Large Databases (VLDB) (Stockholm, Sweden, 1985), VLDB Endowment, pp. 127–141.
11. DOUGLIS, F., AND IYENGAR, A. Application-specific delta-encoding via resemblance detection. In Proceedings of the 2003 USENIX Annual Technical Conference (San Antonio, Texas, June 2003), pp. 113–126.
12. EMC CORPORATION. EMC Centera: Content Addressed Storage System, Data Sheet, April 2002.
13. ESHGHI, K. A framework for analyzing and improving content-based chunking algorithms. Tech. Rep. HPL-2005-30(R.1), Hewlett Packard Laboratories, Palo Alto, 2005.
14. ESHGHI, K., LILLIBRIDGE, M., WILCOCK, L., BELROSE, G., AND HAWKES, R. Jumbo Store: Providing efficient incremental upload and versioning for a utility rendering service. In Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST) (San Jose, CA, February 2007), USENIX Association, pp. 123–138.
15. Federal Information Processing Standard (FIPS) 180–3: Secure Hash Standard (SHS). Tech. Rep. 180–3, National Institute of Standards and Technology (NIST), Gaithersburg, MD, Oct 2008.
16. FORMAN, G., ESHGHI, K., AND CHIOCCHETTI, S. Finding similar files in large document repositories. In Proceeding of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (Chicago, IL, USA, August 2005), ACM Press, pp. 394–400.
17. JAIN, N., DAHLIN, M., AND TEWARI, R. TAPER: Tiered approach for eliminating redundancy in replica synchronization. In Proceedings of the 4th USENIX Conference on File and Storage Technologies (FAST) (San Francisco, CA, USA, December 2005), pp. 281–294.
18. MANBER, U. Finding similar files in a large file system. In Proceedings of the Winter 1994 USENIX Technical Conference (San Fransisco, CA, USA, January 1994), pp. 1–10.
19. Microsoft exchange server 2003 load simulator. Download available at http://www.microsoft.com/downloads, February 2006.
20. MUTHITACHAROEN, A., CHEN, B., AND MAZIÈRES, D. A low-bandwidth network file system. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP) (Banff, Alberta, Canada, October 2001), ACM Press, pp. 174–187.
21. POLICRONIADES, C., AND PRATT, I. Alternatives for detecting redundancy in storage systems data. In Proc. of the General Track, 2004 USENIX Annual Technical Conference (Boston, MA, USA, June 2004), USENIX Association, pp. 73–86.
22. QUINLAN, S., AND DORWARD, S. Venti: A new approach to archival storage. In Proceedings of the FAST 2002 Conference on File and Storage Technologies (Monterey, CA, USA, January 2002), USENIX Association, pp. 89–101.
23. RHEA, S., COX, R., AND PESTEREV, A. Fast, inexpensive content-addressed storage in Foundation. In Proceedings of the 2008 USENIX Annual Technical Conference (Boston, Massachusetts, June 2008), pp. 143–156.
24. SHAPIRO, L. D. Join processing in database systems with large main memories. ACM Transactions on Database Systems 11, 3 (1986), 239–264.
25. TOLIA, N., KOZUCH, M., SATYANARAYANAN, M., KARP, B., BRESSOUD, T., AND PERRIG, A. Opportunistic use of content addressable storage for distributed file systems. In Proceedings of the General Track, 2003 USENIX Annual Technical Conference (San Antonio, Texas, June 2003), USENIX Association, pp. 127–140.
26. YOU, L. L., AND KARAMANOLIS, C. Evaluation of efficient archival storage techniques. In Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies (College Park, MD, April 2004).
27. YOU, L. L., POLLACK, K. T., AND LONG, D. D. E. Deep Store: An archival storage system architecture. In Proc. of the 21st International Conference on Data Engineering (ICDE’05) (Tokyo, Japan, April 2005), IEEE, pp. 804–815.
28. ZHU, B., LI, K., AND PATTERSON, H. Avoiding the disk bottleneck in the Data Domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST) (San Jose, CA, USA, February 2008), USENIX Association, pp. 269–282.
29. ZIV, J., AND LEMPEL, A. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory 23, 3 (1977), 337–343.