A performance evaluation and examination of open-source erasure coding libraries for storage

Proceedings of the 7th USENIX Conference on File and Storage Technologies, FAST 2009

Volumn , Issue , 2009, Pages 253-265

  Plank, James S ^a   Luo, Jianqiang ^b   Schuman, Catherine D ^a   Xu, Lihao ^b   Wilcox O’Hearn, Zooko ^c  

^a UNIVERSITY OF TENNESSEE   (United States)

^b WAYNE STATE UNIVERSITY   (United States)

^c AllMyData   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FAULT TOLERANT COMPUTER SYSTEMS;

CODING PERFORMANCE; CODING TECHNIQUES; ENCODING AND DECODING; FAULT PROTECTION; MULTIPLE DISK FAILURES; OPEN SOURCE IMPLEMENTATION; PARAMETER SELECTION; STORAGE SYSTEMS;

CODES (SYMBOLS);

EID: 85077327800     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. ALLMYDATA. Unlimited online backup, storage, and sharing. http://allmydata.com, 2008.
2. ANVIN, H. P. The mathematics of RAID-6. http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf, 2007.
3. BECK, M., ARNOLD, D., BASSI, A., BERMAN, F., CASANOVA, H., DONGARRA, J., MOORE, T., OBERTELLI, G., PLANK, J. S., SWANY, M., VADHIYAR, S., AND WOLSKI, R. Logistical computing and internetworking: Middleware for the use of storage in communication. In Third Annual International Workshop on Active Middleware Services (AMS) (San Francisco, August 2001).
4. BLAUM, M., BRADY, J., BRUCK, J., AND MENON, J. EVENODD: An efficient scheme for tolerating double disk failures in RAID architectures. IEEE Transactions on Computing 44, 2 (February 1995), 192– 202.
5. BLAUM, M., AND ROTH, R. M. On lowest density MDS codes. IEEE Transactions on Information Theory 45, 1 (January 1999), 46–59.
6. BLOMER, J., KALFANE, M., KARPINSKI, M., KARP, R., LUBY, M., AND ZUCKERMAN, D. An XOR-based erasure-resilient coding scheme. Tech. Rep. TR-95-048, International Computer Science Institute, August 1995.
7. CLEVERSAFE, INC. Cleversafe Dispersed Storage. Open source code distribution: http://www.cleversafe.org/downloads, 2008.
8. CORBETT, P., ENGLISH, B., GOEL, A., GRCANAC, T., KLEIMAN, S., LEONG, J., AND SANKAR, S. Row diagonal parity for double disk failure correction. In 3rd Usenix Conference on File and Storage Technologies (San Francisco, CA, March 2004).
9. FENG, G., DENG, R., BAO, F., AND SHEN, J. New efficient MDS array codes for RAID Part I: Reed-Solomon-like codes for tolerating three disk failures. IEEE Transactions on Computers 54, 9 (September 2005), 1071–1080.
10. FENG, G., DENG, R., BAO, F., AND SHEN, J. New efficient MDS array codes for RAID Part II: Rabin-like codes for tolerating multiple (≥ 4) disk failures. IEEE Transactions on Computers 54, 12 (Decemeber 2005), 1473–1483.
11. GREENAN, K., MILLER, E., AND SCHWARTZ, T. J. Optimizing Galois Field arithmetic for diverse processor architectures and applications. In MASCOTS 2008: 16th IEEE Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (Baltimore, MD, September 2008).
12. HAFNER, J. L. WEAVER Codes: Highly fault tolerant erasure codes for storage systems. In FAST-2005: 4th Usenix Conference on File and Storage Technologies (San Francisco, December 2005), pp. 211–224.
13. HAFNER, J. L. HoVer erasure codes for disk arrays. In DSN-2006: The International Conference on Dependable Systems and Networks (Philadelphia, June 2006), IEEE.
14. HAFNER, J. L., DEENADHAYALAN, V., RAO, K. K., AND TOMLIN, A. Matrix methods for lost data reconstruction in erasure codes. In FAST-2005: 4th Usenix Conference on File and Storage Technologies (San Francisco, December 2005), pp. 183–196.
15. HUANG, C., CHEN, M., AND LI, J. Pyramid codes: Flexible schemes to trade space for access efficienty in reliable data storage systems. In NCA-07: 6th IEEE International Symposium on Network Computing Applications (Cambridge, MA, July 2007).
16. HUANG, C., LI, J., AND CHEN, M. On optimizing XOR-based codes for fault-tolerant storage applications. In ITW’07, Information Theory Workshop (Tahoe City, CA, September 2007), IEEE, pp. 218–223.
17. HUANG, C., AND XU, L. STAR: An efficient coding scheme for correcting triple storage node failures. In FAST-2005: 4th Usenix Conference on File and Storage Technologies (San Francisco, December 2005), pp. 197–210.
18. KARN, P. Dsp and fec library. http://www.ka9q.net/code/fec/, 2007.
19. LUBY, M. Code for Cauchy Reed-Solomon coding. Uuencoded tar file: http://www.icsi.berkeley.edu/~luby/cauchy.tar.uu, 1997.
20. LUO, J., XU, L., AND PLANK, J. S. An efficient XOR-Scheduling algorithm for erasure code encoding. Tech. Rep. Computer Science, Wayne State University, December 2008.
21. MACWILLIAMS, F. J., AND SLOANE, N. J. A. The Theory of Error-Correcting Codes, Part I. North-Holland Publishing Company, Amsterdam, New York, Oxford, 1977.
22. NISBET, B. FAS storage systems: Laying the foundation for application availability. Network Appliance white paper: http://www.netapp.com/us/library/analyst-reports/ar1056.html, February 2008.
23. PARTOW, A. Schifra Reed-Solomon ECC Library. Open source code distribution: http://www.schifra.com/downloads.html, 2000-2007.
24. PLANK, J. S. A new minimum density RAID-6 code with a word size of eight. In NCA-08: 7th IEEE International Symposium on Network Computing Applications (Cambridge, MA, July 2008).
25. PLANK, J. S. The RAID-6 Liberation codes. In FAST-2008: 6th Usenix Conference on File and Storage Technologies (San Jose, February 2008), pp. 97–110.
26. PLANK, J. S., SIMMERMAN, S., AND SCHUMAN, C. D. Jerasure: A library in C/C++ facilitating erasure coding for storage applications - Version 1.2. Tech. Rep. CS-08-627, University of Tennessee, August 2008.
27. PLANK, J. S., AND XU, L. Optimizing Cauchy Reed-Solomon codes for fault-tolerant network storage applications. In NCA-06: 5th IEEE International Symposium on Network Computing Applications (Cambridge, MA, July 2006).
28. REED, I. S., AND SOLOMON, G. Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics 8 (1960), 300–304.
29. RHEA, S., WELLS, C., EATON, P., GEELS, D., ZHAO, B., WEATHERSPOON, H., AND KUBIATOWICZ, J. Maintenance-free global data storage. IEEE Internet Computing 5, 5 (2001), 40–49.
30. RIZZO, L. Effective erasure codes for reliable computer communication protocols. ACM SIGCOMM Computer Communication Review 27, 2 (1997), 24–36.
31. STORER, M. W., GREENAN, K. M., MILLER, E. L., AND VORUGANTI, K. Pergamum: Replacing tape with energy efficient, reliable, disk-based archival storage. In FAST-2008: 6th Usenix Conference on File and Storage Technologies (San Jose, February 2008), pp. 1–16.
32. WELCH, B., UNANGST, M., ABBASI, Z., GIBSON, G., MUELLER, B., SMALL, J., ZELENKA, J., AND ZHOU, B. Scalable performance of the Panasas parallel file system. In FAST-2008: 6th Usenix Conference on File and Storage Technologies (San Jose, February 2008), pp. 17–33.
33. WILCOX-O’HEARN, Z. Zfec 1.4.0. Open source code distribution: http://pypi.python.org/pypi/zfec, 2008.
34. WYLIE, J. J., AND SWAMINATHAN, R. Determining fault tolerance of XOR-based erasure codes efficiently. In DSN-2007: The International Conference on Dependable Systems and Networks (Edinburgh, Scotland, June 2007), IEEE.
35. XU, L., AND BRUCK, J. X-Code: MDS array codes with optimal encoding. IEEE Transactions on Information Theory 45, 1 (January 1999), 272–276.
36. ZHU, B., LI, K., AND PATTERSON, H. Avoiding the disk bottleneck in the Data Domain deduplication file system. In FAST-2008: 6th Usenix Conference on File and Storage Technologies (San Jose, February 2008), pp. 269–282.