Extreme binning: Scalable, parallel deduplication for chunk-based file backup

Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS

Volumn , Issue , 2009, Pages 237-245

  Bhagwat, Deepavali ^a   Eshghi, Kave ^b   Long, Darrell D E ^a   Lillibridge, Mark ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BACK-UP SYSTEMS; BACKUP OPERATIONS; CRITICAL COMPONENT; DATA MANAGEMENT; DEDUPLICATION; FILE BACKUP; INPUT DATAS; LARGE DATA; LOOKUPS; LOW OVERHEAD; NON-TRADITIONAL; PARALLELIZATIONS; STATELESS ROUTING; STORAGE SPACES;

MULTIMEDIA SERVICES; TELECOMMUNICATION SYSTEMS; THROUGHPUT;

COMPUTER SIMULATION;

EID: 76349109236     PISSN: 15267539     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/MASCOT.2009.5366623     Document Type: Conference Paper

References (41)

1. J. F. Gantz, C. Chute, A. Manfrediz, S. Minton, D. Reinsel, W. Schlichting, and A. Toncheva, "The diverse and exploding digital universe: An updated forecast of worldwide information growth through 2011," IDC, An IDC White Paper - sponsored by EMC, March 2008.
2. "104th Congress, United States of America. Public Law 104-191: Health Insurance Portability and Accountability Act (HIPAA)," August 1996.
3. "107th Congress, United States of America. Public Law 107-204: Sarbanes-Oxley Act of 2002," July 2002.
4. H. Biggar, "Experiencing Data De-Duplication: Improving Efficiency and Reducing Capacity Requirements," The Enterprise Strategy Group, Feb. 2007.
5. G. Forman, K. Eshghi, and S. Chiocchetti, "Finding similar files in large document repositories," in KDD '05: Proceeding of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, 2005, pp. 394-400.
6. A. Muthitacharoen, B. Chen, and D. Mazìeres, "A low-bandwidth network file system," in Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP '01), 2001, pp. 174-187.
7. S. Quinlan and S. Dorward, "Venti: A new approach to archival storage," in Proceedings of the First USENIX Conference on File and Storage Technologies (FAST), 2002, pp. 89-101.
8. L. L. You and C. Karamanolis, "Evaluation of efficient archival storage techniques," in Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, Apr. 2004. [Online]. Available: you-mss04.pdf
9. B. Zhu, K. Li, and H. Patterson, "Avoiding the disk bottleneck in the data domain deduplication file system," in Proceedings of the Seventh USENIX Conference on File and Storage Technologies (FAST), 2008, pp. 269-282.
10. M. Lillibridge, K. Eshghi, D. Bhagwat, V. Deolalikar, G. Trezise, and P. Campbell, "Sparse Indexing: Large scale, inline deduplication using sampling and locality," in Proceedings of the Eighth USENIX Conference on File and Storage Technologies (FAST), Feb. 2009, pp. 111-123.
11. S. Rhea, R. Cox, and A. Pesterev, "Fast, inexpensive content-addressed storage in Foundation," in Proceedings of the 2008 USENIX Annual Technical Conference, 2008, pp. 143-156.
12. S. Annapureddy, M. J. Freedman, and D. Mazìeres, "Shark: Scaling file servers via cooperative caching," in Proceedings of the 2nd Symposium on Networked Systems Design and Implementation (NSDI), 2005, pp. 129-142.
13. K. Eshghi, M. Lillibridge, L. Wilcock, G. Belrose, and R. Hawkes, "Jumbo Store: Providing efficient incremental upload and versioning for a utility rendering service," in Proceedings of the Fifth USENIX Conference on File and Storage Technologies (FAST), 2007, pp. 123-138.
14. M. O. Rabin, "Fingerprinting by random polynomials," Center for Research in Computing Technology, Harvard University, Tech. Rep. TR-15-81, 1981.
15. R. Rivest, "The MD5 message-digest algorithm," IETF, Request For Comments (RFC) 1321, Apr. 1992. [Online]. Available: http: //www.ietf.org/rfc/rfc1321.txt
16. National Institute of Standards and Technology, "Secure hash standard," FIPS 180-1, Apr. 1995. [Online]. Available: http://www.itl.nist. gov/ fipspubs/fip180-1.htm
17. National Institute of Standards and Technology, "Secure hash standard," FIPS 180-2, Aug. 2002. [Online]. Available: http://csrc.nist. gov/ publications/fips/fips180-2/fips180-2.pdf
18. C. Policroniades and I. Pratt, "Alternatives for detecting redundancy in storage systems data," in Proceedings of the General Track: 2004 USENIX Annual Technical Conference, 2004, pp. 73-86.
19. M. Dutch, "Understanding data deduplication ratios," SNIA Data Management Forum, June 2008.
20. A. Z. Broder, "On the resemblance and containment of documents," in SEQUENCES '97: Proceedings of the Compression and Complexity of Sequences 1997, 1997, pp. 21-29.
21. P. Jaccard, " Étude comparative de la distribution orale dans une portion des Alpes et des Jura," In Bulletin del la Société Vaudoise des Sciences Naturelles, vol. 37, pp. 547-579, 1901.
22. D. Bhagwat, K. Eshghi, and P. Mehra, "Content-based document routing and index partitioning for scalable similarity-based searches in a large corpus," in KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, 2007, pp. 105-112.
23. R. J. Honicky and E. L. Miller, "Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution," in Proceedings of the 18th International Parallel & Distributed Processing Symposium (IPDPS 2004), vol. 1, 2004, pp. 96.
24. W. Litwin, M.-A. Neimat, and D. A. Schneider, "LH* - a scalable, distributed data structure," ACM Transactions on Database Systems, vol. 21, no. 4, pp. 480-525, 1996.
25. K. Eshghi, "A framework for analyzing and improving content-based chunking algorithms," Hewlett Packard Laboraties, Palo Alto, Tech. Rep. HPL-2005-30(R.1), 2005.
26. V. Henson, "An analysis of compare-by-hash," in Proceedings of the 9th Workshop on Hot Topics in Operating Systems (HotOS-IX), 2003, pp. 13-18.
27. J. Black, "Compare-by-hash: a reasoned analysis," in Proceedings of the Systems and Experience Track: 2006 USENIX Annual Technical Conference, 2006, pp. 85-90.
28. Oracle Corporation, "Oracle Berkeley DB (http://www.oracle.com/ technology/products/berkeley-db/index.html)."
29. B. H. Bloom, "Space/time trade-offs in hash coding with allowable errors," Communications of the ACM, vol. 13, no. 7, pp. 422-426, 1970.
30. A. T. Clements, I. Ahmad, M. Vilayannur, and J. Li, "Decentralized deduplication in SAN cluster file systems," in Proceedings of the 2009 USENIX Annual Technical Conference, Jan. 2009.
31. EMC Corporation, "EMC Centera: Content Addressed Storage System, Data Sheet," Apr. 2002.
32. L. L. You, K. T. Pollack, and D. D. E. Long, "Deep Store: An archival storage system architecture," in Proceedings of the 21st International Conference on Data Engineering (ICDE '05), Apr. 2005, pp. 804-815.
33. M. Ajtai, R. Burns, R. Fagin, D. D. E. Long, and L. Stockmeyer, "Compactly encoding unstructured inputs with differential compression," Journal of the Association for Computing Machinery, vol. 49, no. 3, pp. 318-367, May 2002.
34. F. Douglis and A. Iyengar, "Application-specific delta-encoding via resemblance detection," in Proceedings of the 2003 USENIX Annual Technical Conference, Jun. 2003, pp. 113-126.
35. L. Aronovich, R. A. Ron, E. Bachmat, H. Bitner, M. Hirsch, and S. T. Klein, "The design of a similarity based deduplication system," in SYSTOR '09: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference. New York, NY, USA: ACM, 2009.
36. L. P. Cox, C. D. Murray, and B. D. Noble, "Pastiche: Making backup cheap and easy," in Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI), 2002, pp. 285-298.
37. J. Kubiatowicz, D. Bindel, Y. Chen, P. Eaton, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoon, W. Weimer, C. Wells, and B. Zhao, "OceanStore: An architecture for global-scale persistent storage," in Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Nov. 2000, pp. 190-201.
38. P. Druschel and A. Rowstron, "PAST: A large-scale, persistent peer-to-peer storage utility," in Proceedings of the 8th IEEE Workshop on Hot Topics in Operating Systems (HotOS-VIII), 2001, p. 75.
39. I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan, "Chord: A scalable peer-to-peer lookup service for internet applications," in Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM '01), 2001, pp. 149-160.
40. A. Rowstron and P. Druschel, "Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems," in IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), November 2001, pp. 329-350.
41. B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, and J. D. Kubiatowicz, "Tapestry: A resilient global-scale overlay for service deployment," IEEE Journal on Selected Areas in Communications, vol. 22, no. 1, pp. 41-53, January 2004.