A solution to the network challenges of data recovery in erasure-coded distributed storage systems: A study on the Facebook warehouse cluster

5th USENIX Workshop on Hot Topics in Storage and File Systems, HotStorage 2013

Volumn , Issue , 2013, Pages

  Rashmi, K V ^a   Shah, Nihar B ^a   Gu, Dikang ^b   Kuang, Hairong ^b   Borthakur, Dhruba ^b   Ramchandran, Kannan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b FACEBOOK   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CODES (SYMBOLS); COMPUTER SYSTEM RECOVERY; FILE ORGANIZATION; MULTIPROCESSING SYSTEMS; NETWORK CODING; OPTIMAL SYSTEMS; RECOVERY; SOCIAL NETWORKING (ONLINE); WAREHOUSES;

DATA CENTER NETWORKS; DATA RECOVERY; DESIGN PARAMETERS; DISTRIBUTED STORAGE SYSTEM; ERASURE CODES; HADOOP DISTRIBUTED FILE SYSTEM (HDFS); LARGE AMOUNTS OF DATA; STORAGE EFFICIENCY;

DIGITAL STORAGE;

EID: 85084160461     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. S. Ghemawat, H. Gobioff, and S. Leung, “The google file system,” in ACM SOSP, 2003.
2. D. Borthakur, “HDFS architecture guide,” 2008. [Online]. Available: http://hadoop. apache. org/common/docs/current/hdfs design. pdf
3. ——, “HDFS and Erasure Codes (HDFS-RAID),” 2009. [Online]. Available: http://hadoopblog.blogspot.com/2009/08/hdfs-anderasure-codes-hdfs-raid.html
4. I. Reed and G. Solomon, “Polynomial codes over certain finite fields,” Journal of SIAM, 1960.
5. K. V. Rashmi, N. B. Shah, and K. Ramchandran, “A piggybacking design framework for read-and download-efficient distributed storage codes,” in Proc. IEEE ISIT (to appear), Jul. 2013.
6. B. Schroeder and G. Gibson, “Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?” in Proc. FAST, 2007.
7. D. Ford, F. Labelle, F. Popovici, M. Stokely, V. Truong, L. Barroso, C. Grimes, and S. Quinlan, “Availability in globally distributed storage systems,” in OSDI, 2010.
8. H. Weatherspoon and J. D. Kubiatowicz, “Erasure coding vs. replication: A quantitative comparison,” in IPTPS, 2002.
9. A. G. Dimakis, P. B. Godfrey, Y. Wu, M. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” IEEE Trans. Inf. Th., Sep. 2010.
10. K. V. Rashmi, N. B. Shah, and P. V. Kumar, “Optimal exact-regenerating codes for the MSR and MBR points via a product-matrix construction,” IEEE Trans. Inf. Th., Aug. 2011.
11. I. Tamo, Z. Wang, and J. Bruck, “MDS array codes with optimal rebuilding,” in IEEE ISIT, Jul. 2011.
12. D. Papailiopoulos, A. Dimakis, and V. Cadambe, “Repair optimal erasure codes through Hadamard designs,” in Proc. Allerton Conf., 2011.
13. V. R. Cadambe, C. Huang, S. A. Jafar, and J. Li, “Optimal repair of mds codes in distributed storage via subspace interference alignment,” 2011. [Online]. Available: arXiv:1106.1250
14. O. Khan, R. Burns, J. Plank, W. Pierce, and C. Huang, “Rethinking erasure codes for cloud file systems: minimizing I/O for recovery and degraded reads,” in USENIX FAST, 2012.
15. Z. Wang, A. G. Dimakis, and J. Bruck, “Rebuilding for array codes in distributed storage systems,” in ACTEMT, 2010.
16. L. Xiang, Y. Xu, J. Lui, and Q. Chang, “Optimal recovery of single disk failure in RDP code storage systems,” in ACM SIGMETRICS, 2010.
17. Z. Wang, A. G. Dimakis, and J. Bruck, “Rebuilding for array codes in distributed storage systems,” in ACTEMT, Dec. 2010.
18. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin, “Erasure coding in Windows Azure storage,” in USENIX ATC, 2012.
19. S. Mahesh, M. Asteris, D. Papailiopoulos, A. G. Dimakis, R. Vadali, S. Chen, and D. Borthakur, “Xoring elephants: Novel erasure codes for big data,” in VLDB Endowment (to appear), 2013.