A "hitchhiker's" guide to fast and efficient data reconstruction in erasure-coded data centers

SIGCOMM 2014 - Proceedings of the 2014 ACM Conference on Special Interest Group on Data Communication

Volumn , Issue , 2014, Pages 331-342

  Rashmi, K V ^a   Shah, Nihar B ^a   Gu, Dikang ^b   Kuang, Hairong ^b   Borthakur, Dhruba ^b   Ramchandran, Kannan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b FACEBOOK   (United States)

Author keywords

degraded reads; disk IO; distributed storage; erasure codes; fault tolerance; network traffic; recovery

Indexed keywords

CONVOLUTIONAL CODES; DIGITAL STORAGE; FAULT TOLERANCE; FORWARD ERROR CORRECTION; RECOVERY;

DEGRADED READS; DISK IO; DISTRIBUTED STORAGE; ERASURE CODES; NETWORK TRAFFIC;

DATA REDUCTION;

EID: 84907310763     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2619239.2626325     Document Type: Conference Paper

References (32)

1. HDFS-RAID. http://wiki.apache.org/hadoop/HDFS-RAID.
2. Seamless reliability. http://www.cleversafe.com/overview/reliable, Feb. 2014.
3. R. Bhagwan, K. Tati, Y. C. Cheng, S. Savage, and G. Voelker. Total recall: System support for automated availability management. In NSDI, 2004.
4. D. Borthakur. HDFS and Erasure Codes (HDFS-RAID). http://hadoopblog. blogspot.com/2009/08/hdfs-and-erasure-codes-hdfs-raid.html, Aug. 2009.
5. B.-G. Chun, F. Dabek, A. Haeberlen, E. Sit, H. Weatherspoon, M. F. Kaashoek, J. Kubiatowicz, and R. Morris. Efficient replica maintenance for distributed storage systems. In NSDI, 2006.
6. A. G. Dimakis, P. B. Godfrey, Y. Wu, M. Wainwright, and K. Ramchandran. Network coding for distributed storage systems. IEEE Trans. Inf. Th., Sept. 2010.
7. K. Esmaili, L. Pamies-Juarez, and A. Datta. CORE: Cross-object redundancy for efficient data repair in storage systems. In IEEE International Conf. on Big data, 2013.
8. B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson. Diskreduce: RAID for data-intensive scalable computing. In Proceedings of the 4th Annual Workshop on Petascale Data Storage, pages 6-10. ACM, 2009.
9. B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson. Diskreduce: RAID for data-intensive scalable computing. In ACM Workshop on Petascale Data Storage, 2009.
10. S. Ghemawat, H. Gobioff, and S. Leung. The Google file system. In ACM SOSP, 2003.
11. Y. Hu, H. C. Chen, P. P. Lee, and Y. Tang. Nccloud: Applying network coding for the storage repair in a cloud-of-clouds. In USENIX FAST, 2012.
12. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin. Erasure coding in Windows Azure storage. In USENIX ATC, 2012.
13. S. Jiekak, A. Kermarrec, N. Scouarnec, G. Straub, and A. Van Kempen. Regenerating codes: A system perspective. arXiv:1204.5028, 2012.
14. G. Kamath, N. Silberstein, N. Prakash, A. Rawat, V. Lalitha, O. Koyluoglu, P. Kumar, and S. Vishwanath. Explicit MBR all-symbol locality codes. In ISIT, 2013.
15. O. Khan, R. Burns, J. Plank, W. Pierce, and C. Huang. Rethinking erasure codes for cloud file systems: minimizing I/O for recovery and degraded reads. In FAST, 2012.
16. S. Lin and D. Costello. Error control coding. Prentice-hall Englewood Cliffs, 2004.
17. S. Mahesh, M. Asteris, D. Papailiopoulos, A. G. Dimakis, R. Vadali, S. Chen, and D. Borthakur. Xoring elephants: Novel erasure codes for big data. In VLDB, 2013.
18. J. Mickens and B. Noble. Exploiting availability prediction in distributed systems. In NSDI, 2006.
19. D. Papailiopoulos, A. Dimakis, and V. Cadambe. Repair optimal erasure codes through hadamard designs. IEEE Trans. Inf. Th., May 2013.
20. K. V. Rashmi, N. B. Shah, D. Gu, H. Kuang, D. Borthakur, and K. Ramchandran. A solution to the network challenges of data recovery in erasure-coded distributed storage systems: A study on the Facebook warehouse cluster. In Proc. USENIX HotStorage, June 2013.
21. K. V. Rashmi, N. B. Shah, and P. V. Kumar. Optimal exact-regenerating codes for the MSR and MBR points via a product-matrix construction. IEEE Trans. Inf. Th., 2011.
22. K. V. Rashmi, N. B. Shah, and K. Ramchandran. A piggybacking design framework for read-and download-efficient distributed storage codes. In IEEE International Symposium on Information Theory, 2013.
23. I. Reed and G. Solomon. Polynomial codes over certain finite fields. Journal of SIAM, 1960.
24. S. Rhea, P. Eaton, D. Geels, H. Weatherspoon, B. Zhao, and J. Kubiatowicz. Pond: The OceanStore prototype. In USENIX FAST, 2003.
25. R. Rodrigues and B. Liskov. High availability in DHTs: Erasure coding vs. replication. In IPTPS, 2005.
26. N. Shah, K. Rashmi, P. Kumar, and K. Ramchandran. Distributed storage codes with repair-by-transfer and non-achievability of interior points on the storage-bandwidth tradeoff. IEEE Trans. Inf. Theory, 2012.
27. N. B. Shah. On minimizing data-read and download for storage-node recovery. IEEE Communications Letters, 2013.
28. K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The Hadoop distributed file system. In IEEE MSST, 2010.
29. I. Tamo, Z. Wang, and J. Bruck. Zigzag codes: MDS array codes with optimal rebuilding. IEEE Trans. Inf. Th., 2013.
30. Z. Wang, A. Dimakis, and J. Bruck. Rebuilding for array codes in distributed storage systems. In ACTEMT, 2010.
31. H. Weatherspoon and J. D. Kubiatowicz. Erasure coding vs. replication: A quantitative comparison. In IPTPS, 2002.
32. L. Xiang, Y. Xu, J. Lui, and Q. Chang. Optimal recovery of single disk failure in RDP code storage systems. In ACM SIGMETRICS, 2010.