Coding techniques for repairability in networked distributed storage systems

Foundations and Trends in Communications and Information Theory

Volumn 9, Issue 4, 2012, Pages 383-466

  Oggier, Fréedérique ^a   Datta, Anwitaman ^b  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

Distributed storage systems; Erasure codes; Repair

Indexed keywords

CODING TECHNIQUES; DISTRIBUTED STORAGE SYSTEM; ERASURE CODES; REPAIRABILITY;

FORWARD ERROR CORRECTION; REPAIR; SURVEYS;

MULTIPROCESSING SYSTEMS;

EID: 84888220275     PISSN: 15672190     EISSN: 15672328     Source Type: Journal    
DOI: 10.1561/0100000068     Document Type: Article

References (43)

1. R. Alshwede, N. Cai, S.-Y. R. Li, and R. W. Yeung, "Network information flow," IEEE Transactions on Information Theory, Vol. 46, no. 4, 2000.
2. D. Beaver, S. Kumar, H. C. Li, J. Sobel, and P. Vajgel, "Finding a needle in Haystack: Facebooks photo storage," OSDI, 2010.
3. R. Bhagwan, K. Tati, Y.-C. Cheng, S. Savage, and G. M. Voelker, "Total recall: System support for automated availability management," USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2004.
4. R. E. Blahut, Algebraic Codes for Data Transmission. Cambridge.
5. A. Datta and F. Oggier, "Redundantly grouped cross-object coding for repairable storage," APSys, 2012.
6. A. Datta and F. Oggier, "An overview of codes tailor-made for better repairability in networked distributed storage systems," SIGACT News, Vol. 44, no. 1, Available at http://arxiv.org/abs/1109.2317, March 2013.
7. A. Datta, L. Pamies-Juarez, and F. Oggier, "On data insertion and migration in erasure-coding based large-scaled storage systems," to appear in ICDCIT 2013.
8. A. G. Dimakis, P. B. Godfrey, Y. Wu, M. Wainwright, and K. Ramchandran, "Network coding for distributed storage systems," IEEE Transactions on Information Theory, Vol. 56, no. 9, September 2010.
9. A. G. Dimakis, K. Ramchandran, Y. Wu, and C. Suh, "A survey on network codes for distributed storage," The Proceedings of the IEEE, Vol. 99, no. 3, March 2011.
10. A. Duminuco and E. Biersack, "Hierarchical codes: How to make erasure codes attractive for peer-to-peer storage systems," Eighth International Conference on In Peer-to-Peer Computing, P2P, 2008.
11. P. Elias, "Error-free coding," Transactions on Information Theory, Vol. 4, no. 4, September 1954.
12. K. S. Esmaili, L. Pamies-Juarez, and A. Datta, "The CORE storage primitive: Cross-object redundancy for efficient data repair & access in erasure coded storage," preprint, available at http://arxiv.org/abs/1302. 5192.
13. P. Gopalan, C. Huang, H. Simitci, and S. Yekhanin, "On the locality of codewords symbols," Electronic Colloquium on Computational Complexity (ECCC), Vol. 18, 2011.
14. hadoop.apache.org/.
15. H. D. L. Hollmann, "Storage codes - coding rate and repair locality," International Conference on Computing, Networking and Communications, available at http://arxiv.org/abs/1301.4300 (ICNC 2013).
16. http://wiki.apache.org/hadoop/HDFS-RAID.
17. Y. Hu, Y. Xu, X. Wang, C. Zhan, and P. Li, "Cooperative recovery of distributed storage systems from multiple losses with network coding," IEEE Journal on Selected Areas in Communications, Vol. 28, no. 2, February 2010.
18. C. Huang, M. Chen, and J. Li, "Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems," IEEE International Symposium on Network Computing and Applications, NCA 2007, 2007.
19. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Lin, and S. Yekhanin, "Erasure coding in windows azure storage,".
20. Y. H. K. W. Shum, "Cooperative regenerating codes," preprint, available at http://arxiv.org/abs/1101.5257.
21. A.-M. Kermarrec, N. Le Scouarnec, and G. Straub, "Repairing multiple failures with coordinated and adaptive regenerating codes," in the Proceedings of the 2011 International Symposium on Network Coding (NetCod 2011).
22. R. Koetter and M. Medard, "An algebraic approach to network coding," IEEE/ACM Transactions on Networking, Vol. 11, 2001.
23. D. Leong, A. G. Dimakis, and T. Ho, "Distributed storage allocations," IEEE Transactions on Information Theory, Vol. 58, no. 7, pp. 4733-4752, July 2012.
24. W. K. Lin, D. M. Chiu, and Y. B. Lee, "Erasure code replication revisited," P2P, 2004.
25. F. Oggier and A. Datta, "Homomorphic self-repairing codes for agile maintenance of distributed storage systems," http://arxiv.org/abs/1107. 3129.
26. F. Oggier and A. Datta, "Byzantine fault tolerance of regenerating codes," P2P, 2011.
27. F. Oggier and A. Datta, "Self-repairing codes for distributed storage - a projective geometric construction," ITW, 2011.
28. F. Oggier and A. Datta, "Self-repairing homomorphic codes for distributed storage systems," INFOCOM, 2011.
29. L. Pamies-Juarez, A. Datta, and F. Oggier, "RapidRAID: Pipelined erasure codes for fast data archival in distributed storage systems," http://arxiv.org/ abs/1207.6744.
30. L. Pamies-Juarez, F. Oggier, and A. Datta, "An empirical study of the repair performance of novel coding schemes for networked distributed storage systems," http://arxiv.org/abs/1206.2187.
31. D. S. Papailiopoulos and A. G. Dimakis, "Locally repairable codes," IEEE International Symposium on Information Theory (ISIT), Available at http: //arxiv.org/abs/1206.3804, 2012.
32. D. A. Patterson, G. Gibson, and R. H. Katz, "A case for redundant arrays of inexpensive disks (RAID)," ACM SIGMOD International Conference on Management of Data, 1988.
33. S. Pawar, S. El Rouayheb, and K. Ramchandran, "Securing dynamic distributed storage systems against eavesdropping and adversarial attacks," IEEE Transactions on Information Theory (Special Issue on Facets of Coding Theory: from Algorithms to Networks), Vol. 57, no. 9, September 2011.
34. K. V. Rashmi, N. B. Shah, P. Vijay Kumar, and K. Ramchandran, "Explicit construction of optimal exact regenerating codes for distributed storage," Allerton, 2009.
35. A. S. Rawat and S. Vishwanath, "On locality in distributed storage systems," ITW, 2012.
36. I. Reed and G. Solomon, "Polynomial codes over certain finite fields," Journal of the Society of Industrial and Applied Mathematics, 1960.
37. K. W. Shum, "Cooperative regenerating codes for distributed storage systems," ICC, available at http://arxiv.org/abs/1101.5257, 2011.
38. E. Stefanov, M. van Dijk, A. Oprea, and A. Juels, "Iris: A scalable cloud file system with efficient integrity checks," Cryptology ePrint Archive, Report 2011/585, 2011.
39. C. Tian, "Rate region of the (4, 3, 3) exact-repair regenerating codes," preprint.
40. Y. Wu and A. G. Dimakis, "Reducing repair traffic for erasure coding-based storage via interference alignment," ISIT, 2009.
41. www.cleversafe.com.
42. www.wuala.com/.
43. R. W. Yeung, S.-Y. R. Li, N. Cai, and Z. Zhang, "Network coding theory: Single sources," Foundations and Trends in Communication and Information Theory, Vol. 2, no. 4.