Rethinking erasure codes for cloud file systems: Minimizing I/O for recovery and degraded reads

Proceedings of FAST 2012: 10th USENIX Conference on File and Storage Technologies

Volumn , Issue , 2012, Pages 251-264

  Khan, Osama ^a   Burns, Randal ^a   Plank, James ^b   Pierce, William ^b   Huang, Cheng ^c  

^a JOHNS HOPKINS UNIVERSITY   (United States)

^b Journal of Nursing Education   (United States)

^c MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BLOCK CODES; FILE ORGANIZATION; FORWARD ERROR CORRECTION; OPTIMAL SYSTEMS; RECOVERY; REED-SOLOMON CODES;

CLOUD SYSTEMS; CODING SCHEME; ERASURE CODES; NEW DIMENSIONS; OPTIMAL NUMBER; PERFORMANCE PROPERTIES; REED SOLOMON; STORAGE OVERHEAD;

DIGITAL STORAGE;

EID: 85077123991     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (53)

1. H. P. Anvin. The mathematics of RAID-6. http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf, 2009.
2. M. Blaum. A family of MDS array codes with minimal number of encoding operations. In IEEE International Symposium on Information Theory, September 2006.
3. M. Blaum, J. Brady, J. Bruck, and J. Menon. EVENODD: An efficient scheme for tolerating double disk failures in RAID architectures. IEEE Transactions on Computing, 44(2):192- 202, February 1995.
4. M. Blaum, J. Bruck, and A. Vardy. MDS array codes with independent parity symbols. IEEE Transactions on Information Theory, 42(2):529-542, February 1996.
5. M. Blaum and R. M. Roth. On lowest density MDS codes. IEEE Transactions on Information Theory, 45(1):46-59, January 1999.
6. J. Blomer, M. Kalfane, M. Karpinski, R. Karp, M. Luby, and D. Zuckerman. An XOR-based erasure-resilient coding scheme. Technical Report TR-95-048, International Computer Science Institute, August 1995.
7. V. Bohossian and J. Bruck. Shortening array codes and the perfect 1-Factorization conjecture. In IEEE International Symposium on Information Theory, pages 2799-2803, 2006.
8. D. Borthakur. The Hadoop distributed file system: Architecture and design. http://hadoop.apache.org/common/docs/current/hdfs-design.html, 2009.
9. E. Brewer. Lessons from giant-scale services. Internet Computing, 5(4), 2001.
10. B. Calder, J. Wang, A. Ogus, N. Nilakantan, A. Skjolsvold, S. McKelvie, Y. Xu, S. Srivastav, J. Wu, H. Simitci, J. Haridas, C. Uddaraju, H. Khatri, A. Edwards, V. Bedekar, S. Mainali, R. Abbasi, A. Agarwal, M. Fahim ul Haq, M. Ikram ul Haq, D. Bhardwaj, S. Dayanand, A. Adusumilli, M. McNett, S. Sankaran, K. Manivannan, and L. Rigas. Windows Azure storage: A highly available cloud storage service with strong consistency. In Symposium on Operating Systems Principles, 2011.
11. P. Corbett, B. English, A. Goel, T. Grcanac, S. Kleiman, J. Leong, and S. Sankar. Row diagonal parity for double disk failure correction. In Conference on File and Storage Technologies, March 2004.
12. A. G. Dimakis, P. B. Godfrey, Y. Wu, M. J. Wainwright, and K. Ramchandran. Network coding for distributed storage systems. IEEE Trans. Inf. Theor., 56(9):4539-4551, September 2010.
13. A. L. Drapeau et al. RAID-II: A high-bandwidth network file server. In International Symposium on Computer Architecture, 1994.
14. B. Fan, W Tanisiriroj, L. Xiao, and G. Gibson. DiskReduce: RAID for data-intensive scalable computing. In Parallel Data Storage Workshop, 2008.
15. D. Ford, F. Labelle, F. .I. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan. Availability in globally distributed file systems. In Operating Systems Design and Implementation, 2010.
16. S. Ghemawat, H. Gobioff, and S. Leung. The Google file system. In ACM SOSP, 2003.
17. K. Greenan, E. Miller, and T. J. Schwartz. Optimizing Galois Field arithmetic for diverse processor architectures and applications. In Modeling, Analysis and Simulation of Computer and Telecommunication Systems, September 2008.
18. K. M. Greenan, X. Li, and J. J. Wylie. Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs. Mass Storage Systems and Technologies, 2010.
19. J. L. Hafner. Weaver codes: Highly fault tolerant erasure codes for storage systems. In Conference on File and Storage Technologies, 2005.
20. J. L. Hafner, V. Deenadhayalan, K. K. Rao, and J. A. Tomlin. Matrix methods for lost data reconstruction in erasure codes. In Conference on File and Storage Technologies, 2005.
21. M. Holland and G. A. Gibson. Parity declustering for continuous operation in redundant disk arrays. In Architectural Support for Programming Languages and Operating Systems. ACM, 1992.
22. R. Y. Hou, J. Menon, and Y. N. Patt. Balancing I/O response time and disk rebuild time in a RAID5 disk array. In Hawai'i International Conference on System Sciences, 1993.
23. C. Huang, M. Chen, and J. Li. Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems. Network Computing and Applications, 2007.
24. C. Huang and L. Xu. STAR: An efficient coding scheme for correcting triple storage node failures. IEEE Transactions on Computers, 57(7):889-901, July 2008.
25. H. Jin, J. Zhang, and K. Hwang. A raid reconfiguration scheme for gracefully degraded operations. EuroMicro Conference on Parallel, Distributed, and Network-Based Processing, 0:66, 1999.
26. D. Kenchammana-Hosekote, D. He, and J. L. Hafner. REO: A generic RAID engine and optimizer. In Conference on File and Storage Technologies, pages 261-276, 2007.
27. O. Khan, R. Burns, J. S. Plank, and C. Huang. In search of I/O-optimal recovery from disk failures. In Workshop on Hot Topics in Storage Systems, 2011.
28. M. Luby, M. Mitzenmacher, A. Shokrollahi, D. Spielman, and V. Stemann. Practical loss-resilient codes. In 29th Annual ACM Symposium on Theory of Computing, pages 150-159, El Paso, TX, 1997. ACM.
29. F. J. MacWilliams and N. J. A. Sloane. The Theory of Error-Correcting Codes, Part I. North-Holland Publishing Company, 1977.
30. Onion Networks. Java FEC Library v1.0.3. Open source code distribution: http://onionnetworks.com/fec/javadoc/, 2001.
31. A. Partow. Schifra Reed-Solomon ECC Library. Open source code distribution: http://www.schifra.com/downloads.html, 2000-2007.
32. W. W. Peterson and E. J. Weldon, Jr. Error-Correcting Codes, Second Edition. The MIT Press, 1972.
33. J. S. Plank. A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems. Software—Practice & Experience, 27(9):995-1012, 1997.
34. J. S. Plank. The RAID-6 Liberation codes. In Conference on File and Storage Technologies, 2008.
35. J. S. Plank. The RAID-6 Liber8Tion code. Int. J. High Perform. Comput. Appl., 23:242-251, August 2009.
36. J. S. Plank and Y. Ding. Note: Correction to the 1997 tutorial on Reed-Solomon coding. Software - Practice & Experience, 35(2):189-194, February 2005.
37. J. S. Plank, J. Luo, C. D. Schuman, L. Xu, and Z. WilcoxOHearn. A performance evaluation and examination of open-source erasure coding libraries for storage. In Conference on File and Storage Technologies, 2009.
38. J. S. Plank, S. Simmerman, and C. D. Schuman. Jerasure: A library in C/C++ facilitating erasure coding for storage applications - Version 1.2. Technical Report CS-08-627, University of Tennessee, August 2008.
39. K. V. Rashmi, N. B. Shah, P. V. Kumar, and K. Ramachandran. Explicit construction of optimal exact regenerating codes for distributed storage. In Communication, Control, and Computing, 2009.
40. I. S. Reed and G. Solomon. Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics, 8:300-304, 1960.
41. L. Rizzo. Effective erasure codes for reliable computer communication protocols. ACM SIGCOMM Computer Communication Review, 27(2):24-36, 1997.
42. L. Rizzo. Erasure codes based on Vandermonde matrices. Gzipped tar file posted at http://planete-bcast.inrialpes.fr/rubrique.php3?id rubrique=10, 1998.
43. R. Rodrigues and B. Liskov. High availability in DHTS: Erasure coding vs. replication. In Workshop on Peer-to-Peer Systems, 2005.
44. B. Schroeder and G. Gibson. Disk failures in the real world: What does an MTTF of 1,000,000 mean to you? In Conference on File and Storage Technologies, 2007.
45. M. Sivathanu, V. Prabhakaran, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Improving storage system availability with D-GRAID. In Conference on File and Storage Technologies, 2004.
46. Apache Software. Pigmix. https://cwiki.apache.org/confluence/display/PIG/ PigMix, 2011.
47. A. Thusoo, D. Borthakur, R. Murthy, Z. Shao, N. Jain, H. Liu, S. Anthony, and J. S. Sarma. Data warehousing and analytics infrastructure at Facebook. In SIGMOD, 2010.
48. L. Tian, D. Feng, H. Jiang, K. Zhou, L. Zeng, J. Chen, Z. Wang, and Z. Song. PRO: a popularity-based multi-threaded reconstruction optimization for RAID-structured storage systems. In Conference on File and Storage Technologies, 2007.
49. Z. Wang, A. G. Dimakis, and J. Bruck. Rebuilding for array codes in distributed storage systems. CoRR, abs/1009.3291, 2010.
50. H. Weatherspoon and J. Kubiatowicz. Erasure coding vs. replication: A quantitative comparison. In Workshop on Peer-to-Peer Systems, 2002.
51. L. Xiang, Y. Xu, J. C. S. Lui, and Q. Chang. Optimal recovery of single disk failure in RDP code storage systems. In ACM SIGMETRICS, 2010.
52. L. Xu and J. Bruck. X-Code: MDS array codes with optimal encoding. IEEE Transactions on Information Theory, 45(1):272-276, January 1999.
53. Z. Zhang, A. Deshpande, X. Ma, E. Thereska, and D. Narayanan. Does erasure coding have a role to play in my data center? Microsoft Technical Report MSR-TR-2010-52, 2010.