A tale of two erasure codes in HDFS

Proceedings of the 13th USENIX Conference on File and Storage Technologies, FAST 2015

Volumn , Issue , 2015, Pages 213-226

  Xia, Mingyuan ^a   Saxena, Mohit ^b   Blaum, Mario ^b   Pease, David A ^b  

^a MCGILL UNIVERSITY   (Canada)

^b IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIGITAL STORAGE; FILE ORGANIZATION; FORWARD ERROR CORRECTION; MULTIPROCESSING SYSTEMS; PRODUCT DESIGN; RECOVERY;

CONVERSION MECHANISM; DATA REPLICATION; DESIGN TECHNIQUE; DISTRIBUTED STORAGE SYSTEM; HADOOP DISTRIBUTED FILE SYSTEM (HDFS); NETWORK TRANSFERS; RECOVERY PERFORMANCE; STORAGE OVERHEAD;

CODES (SYMBOLS);

EID: 84991977354     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Cleversafe object storage. http://www.cleversafe.com.
2. Colossus, successor to Google File System. http://static.googleusercontent.com/media/research.google.com/ en/us/university/relations/ facultysummit2010/storage_ architecture_and_challenges.pdf.
3. Facebook’s erasure coded hadoop distributed file system (HDFS-RAID). https://github.com/facebook/hadoop-20.
4. The Hadoop Distributed File System. http://wiki.apache.org/hadoop/HDFS, 2007.
5. Cloudera: What do real-life apache hadoop workloads look like? http://blog.cloudera.com/blog/2012/09/what-do-real-life-hadoop-workloads-look-like/, September 2012.
6. An introduction to GPFS version 3.5. http://www-03.ibm.com/systems/resources/introduction-to-gpfs-3-5.pdf, August 2012.
7. ABAD, C. L., ROBERTS, N., LU, Y., AND CAMPBELL, R. H. A storage-centric analysis of mapreduce workloads: File popularity, temporal locality and arrival patterns. In Proceedings of the 2012 IEEE International Symposium on Workload Characterization (IISWC) (Washington, DC, USA, 2012), IISWC’12, IEEE Computer Society, pp. 100–109.
8. BLAUM, M., HAFNER, J. L., AND HETZLER, S. Partial-MDS codes and their application to RAID type of architectures. IEEE Transactions on Information Theory 59, 7 (2013), 4510–4519.
9. CHEN, Y., ALSPAUGH, S., AND KATZ, R. Interactive analytical processing in big data systems: A cross-industry study of mapreduce workloads. Proc. VLDB Endow. 5, 12 (Aug. 2012), 1802–1813.
10. FAN, B., TANTISIRIROJ, W., XIAO, L., AND GIBSON, G. DiskReduce: RAID for data-intensive scalable computing. In Proceedings of the 4th Annual Workshop on Petascale Data Storage (New York, NY, USA, 2009), PDSW’09, ACM, pp. 6–10.
11. GHEMAWAT, S., GOBIOFF, H., AND LEUNG, S.T. The Google File System. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles (New York, NY, USA, 2003), SOSP’03, ACM, pp. 29–43.
12. HAFNER, J. L. HoVer erasure codes for disk arrays. In Proceedings of the International Conference on Dependable Systems and Networks (Washington, DC, USA, 2006), DSN’06, IEEE Computer Society, pp. 217–226.
13. HAFNER, J. L., AND RAO, K. Notes on reliability models for non-MDS erasure codes. IBM Tech Report RJ10391 (A0610-035), 2006.
14. HUANG, C., CHEN, M., AND LI, J. Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems. Trans. Storage 9, 1 (Mar. 2013), 3:1–3:28.
15. HUANG, C., SIMITCI, H., XU, Y., OGUS, A., CALDER, B., GOPALAN, P., LI, J., AND YEKHANIN, S. Erasure coding in Windows Azure Storage. In Proceedings of the 2012 USENIX Conference on Annual Technical Conference (2012), USENIX ATC’12, USENIX Association, pp. 2–2.
16. KHAN, O., BURNS, R., PLANK, J., PIERCE, W., AND HUANG, C. Rethinking erasure codes for cloud file systems: Minimizing I/O for recovery and degraded reads. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (2012), FAST’12, USENIX Association, pp. 20–20.
17. LUSE, P., AND GREENAN, K. Swift object storage: Adding erasure codes. http://www.snia.org/sites/default/files/Luse_ Kevin_SNIATutorialSwift_Object_ Storage2014_final.pdf, 2014.
18. PLANK, J. S., GREENAN, K. M., AND MILLER, E. L. Screaming fast Galois Field arithmetic using Intel SIMD instructions. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (2013), FAST’13, USENIX Association, pp. 299–306.
19. RASHMI, K. V., SHAH, N. B., GU, D., KUANG, H., BORTHAKUR, D., AND RAMCHANDRAN, K. A solution to the network challenges of data recovery in erasure-coded distributed storage systems: A study on the Facebook warehouse cluster. In Proceedings of the 5th USENIX Conference on Hot Topics in Storage and File Systems (2013), HotStorage’13, USENIX Association, pp. 8–8.
20. RASHMI, K. V., SHAH, N. B., GU, D., KUANG, H., BORTHAKUR, D., AND RAMCHANDRAN, K. A “hitchhiker’s” guide to fast and efficient data reconstruction in erasure-coded data centers. In Proceedings of ACM SIGCOMM (2014), SIGCOMM’14.
21. REED, I. S., AND SOLOMON, G. Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics 8, 2 (1960), 300–304.
22. REN, K., KWON, Y., BALAZINSKA, M., AND HOWE, B. Hadoop’s adolescence: An analysis of Hadoop usage in scientific workloads. Proc. VLDB Endow. 6, 10 (Aug. 2013), 853–864.
23. ROTH, R. Introduction to Coding Theory. Cambridge University Press, New York, NY, USA, 2006.
24. SATHIAMOORTHY, M., ASTERIS, M., PAPAILIOPOULOS, D., DIMAKIS, A. G., VADALI, R., CHEN, S., AND BORTHAKUR, D. XORing elephants: novel erasure codes for big data. In Proceedings of the 39th international conference on Very Large Data Bases (2013), PVLDB’13, VLDB Endowment, pp. 325–336.
25. SCHMUCK, F., AND HASKIN, R. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 1st USENIX Conference on File and Storage Technologies (2002), FAST’02, USENIX Association.
26. TAMO, I., AND BARG, A. A family of optimal locally recoverable codes. IEEE Transactions on Information Theory 60, 8 (2014), 4661–4676.
27. WILKES, J., GOLDING, R., STAELIN, C., AND SULLIVAN, T. The HP AutoRAID hierarchical storage system. ACM Trans. Comput. Syst. 14, 1 (Feb. 1996), 108–136.