RapidRAID: Pipelined erasure codes for fast data archival in distributed storage systems

Proceedings - IEEE INFOCOM

Volumn , Issue , 2013, Pages 1294-1302

  Pamies Juarez, Lluis ^a   Datta, Anwitaman ^a   Oggier, Frederique ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

archival; distributed storage; erasure codes; migration

Indexed keywords

ARCHIVAL; COMPUTING CAPACITY; DATA RELIABILITY; DISTRIBUTED STORAGE; DISTRIBUTED STORAGE SYSTEM; ERASURE CODES; INDIVIDUAL OBJECTS; MIGRATION;

BENCHMARKING; ENCODING (SYMBOLS); FORWARD ERROR CORRECTION; MULTIPROCESSING SYSTEMS;

DIGITAL STORAGE;

EID: 84883128621     PISSN: 0743166X     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/INFCOM.2013.6566922     Document Type: Conference Paper

References (23)

1. S. Ghemawat, H. Gobioff, and S. Leung, The google file system, in Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2003.
2. Amazon.com, Amazon S3, http://aws.amazon.com/s3.
3. Apache.org, HDFS, http://hadoop.apache.org/hdfs/.
4. R. Rodrigues and B. Liskov, High availability in dhts: Erasure coding vs. replication, in Proceedings of the 4th International Workshop on Peer-To-Peer Systems (IPTPS), 2005.
5. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin, Erasure coding in windows azure storage, in Proceedings of the USENIX Annual Technical Conference (ATC), 2012.
6. A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sen Sarma, R. Murthy, and H. Liu, Data warehousing and analytics infrastructure at facebook, in Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, ser. SIGMOD 10, 2010.
7. L. X. Bin Fan, Wittawat Tantisiriroj and G. Gibson, Diskreduce: Replication as a prelude to erasure coding in data-intensive scalable computing, Carnegie Mellon Univsersity, Parallel Data Laboratory, Tech. Rep. Technical Report CMU-PDL-11-112, 2011.
8. D. Ford, F. Labelle, F. I. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan, Availability in globally distributed storage systems, in The 9th USENIX conference on Operating Systems Design and Implementation (OSDI), 2010.
9. Apache.org, HDFSRAID, http://wiki.apache.org/hadoop/HDFS-RAID.
10. J. Dean and S. Ghemawat, Mapreduce: simplified data processing on large clusters, Commun. ACM, vol. 51, no. 1, 2008.
11. A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sen Sarma, R. Murthy, and H. Liu, Data warehousing and analytics infrastructure at facebook, in Proceedings of the 2010 international conference on Management of data (SIGMOD), 2010.
12. Y. Li, D. D. E. Long, and E. L. Miller, Understanding data survivability in archival storage systems, 2012.
13. L. Pamies-Juarez, P. Garc?a-Lopez, M. Sanchez-Artigas, and B. Herrera, Towards the design of optimal data redundancy schemes for heterogeneous cloud storage infrastructures, Computer Networks, vol. 55, no. 5, pp. 1100-1113, 2011.
14. J. Li, S. Yang, X. Wang, and B. Li, Tree-structured data regeneration in distributed storage systems with regenerating codes, in The 29th IEEE Intl. Conference on Computer Communications (INFOCOM), 2010.
15. F. Oggier and A. Datta, Self-repairing homomorphic codes for distributed storage systems, in The 30th IEEE Intl. Conference on Computer Communications (INFOCOM), 2011.
16. A.-M. Kermarrec, N. LeScouarnec, and G. Straub, Repairing multiple failures with coordinated and adaptive regenerating codes, in Intl. Symp. on Network Coding (NetCod), 2011.
17. X. Li, M. Lillibridge, and M. Uysal, Reliability analysis of deduplicated and erasure-coded storage, in Proceedings of the Workshop on Hot Topics in Measurement Modeling of Computer Systems (HotMetrics), 2010.
18. S. Acedanski, S. Deb, M. Medard, and R. Koetter, How good is random linear coding based distributed networked storage, in Workshop on Network Coding, Theory, and Applications (NetCod), 2005.
19. B. Shroeder and G. A. Gibson, Disk failures in the real world: What does an mttf of 1,000,000 hours mean to you? in The 5th USENIX Conference on File and Storage Technologies (FAST), 2007.
20. E. Pinheiro, W.-D. Weber, and L. A. Barroso, Failure trends in a large disk drive population, in The 5th USENIX Conference on File and Storage Technologies (FAST), 2007.
21. J. S. Plank, S. Simmerman, and C. D. Schuman, Jerasure: A library in c/c++ facilitating erasure coding for storage applications, University of Tennessee, Tech. Rep. Technical Report CS-08-627, 2008.
22. J. S. Plank, J. Luo, C. D. Schuman, L. Xu, and Z. Wilcox-OHearn, A performance evaluation and examination of open-source erasure coding libraries for storage, in Proccedings of the 7th conference on File and storage technologies (FAST), 2009.
23. A. Dimakis, V. Prabhakaran, and K. Ramchandran, Decentralized erasure codes for distributed networked storage, IEEE/ACM Transactions on Networking, vol. 14, 2006.