ERMS: An elastic replication management system for HDFS

Proceedings - 2012 IEEE International Conference on Cluster Computing Workshops, Cluster Workshops 2012

Volumn , Issue , 2012, Pages 32-40

  Cheng, Zhendong ^a   Luan, Zhongzhi ^a   Meng, You ^a   Xu, Yijing ^a   Qian, Depei ^a   Roy, Alain ^b   Zhang, Ning ^c   Guan, Gang ^c  

^a BEIHANG UNIVERSITY   (China)

^b University of Wisconsin   (United States)

^c TENCENT AI LAB   (China)

Author keywords

Elastic; HDFS; Replication Management

Indexed keywords

CLUSTER COMPUTING; COMPUTER ARCHITECTURE; FILE ORGANIZATION; MULTIPROCESSING SYSTEMS;

COMPLEX EVENT PROCESSING; DISTRIBUTED STORAGE SYSTEM; ELASTIC; HADOOP DISTRIBUTED FILE SYSTEM (HDFS); HDFS; LARGE SCALE DATA SETS; RELIABILITY AND AVAILABILITY; REPLICA PLACEMENT STRATEGIES;

DIGITAL STORAGE;

EID: 84872516809     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ClusterW.2012.25     Document Type: Conference Paper

References (26)

1. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System, " in Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST 10), 2010, pp. 1-10.
2. J. Dean and S. Ghemawat, "MapReduce: Simplied Data Processing on Large Clusters, " in Proceedings of the 6th conference on Symposium on Opearting Systems Design Implementation, 2004.
3. S. Cheng, Z. Cheng, Z. Luan, and D. Qian, "NEPnet: A scalable monitoring system for anomaly detection of network service, " in Proceedings of the 7th International Conference on Network and Service Management (CNSM 11), 2011, pp. 1-5.
4. Thain, T. Tannenbaum, and M. Livny, "Distributed computing in practice: the Condor experience: Research Articles, " Concurrency and Computation: Practice and Experience, vol. 17, pp. 323-356, 2005. (Pubitemid 40285781)
5. HDFS Architecture Guide http://hadoop. apache. org/common/docs/stable/ hdfs-design. ht ml
6. Q. Wei, B. Veeravalli, B. Gong, L. Zeng, and D. Feng, "CDRM: A Cost-Effective Dynamic Replication Management Scheme for Cloud Storage Cluster, " in Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER 10), 2010, pp. 188-196.
7. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris, "Scarlett: coping with skewed content popularity in mapreduce clusters, " presented at the Proceedings of the sixth conference on Computer systems, EuroSys '11 Salzburg, Austria, 2011.
8. L. Abad, Y. Lu, and R. H. Campbell, "DARE: Adaptive Data Replication for Efficient Cluster Scheduling, " in Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER 11), 2011, pp. 159-168.
9. B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, "DiskReduce: RAID for data-intensive scalable computing, " in Proceedings of the 4th Annual Workshop on Petascale Data Storage, Portland, Oregon, 2009.
10. O. Khan, R. Burns, J. Plank, W. Pierce and C. Huang, "Rethinking Erasure Codes for Cloud File Systems: Minimizing I/O for Recovery and Degraded Reads. " Conference on File and Storage Technologies (FAST), USENIX, 2012.
11. D. Borthakur, J. Gray, J. S. Sarma, K. Muthukkaruppan, N. Spiegelberg, H. Kuang, K. Ranganathan, D. Molkov, A. Menon, S. Rash, R. Schmidt, and A. Aiyer, "Apache hadoop goes realtime at Facebook, " in Proceedings of the 2011 international conference on Management of data, Athens, Greece, 2011.
12. H. C. Lim, S. Babu, and J. S. Chase, "Automated control for elastic storage, " in Proceedings of the 7th international conference on Autonomic computing, Washington, DC, USA, 2010.
13. F. Leibert, J. Mannix, J. Lin, and B. Hamadani, "Automatic management of partitioned, replicated search services, " in Proceedings of the 2nd ACM Symposium on Cloud Computing, Cascais, Portugal, 2011.
14. D. Ford, F. Labelle, F. I. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan, "Availability in globally distributed storage systems, " in Proceedings of the 9th USENIX conference on Operating systems design and implementation, Vancouver, BC, Canada, 2010.
15. Y. Shi, X. Meng, J. Zhao, X. Hu, B. Liu, and H. Wang, "Benchmarking cloud-based data management systems, " in Proceedings of the 2nd international workshop on Cloud data management, Toronto, ON, Canada, 2010.
16. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber, "Bigtable: a distributed storage system for structured data, " in Proceedings of the 7th symposium on Operating systems design and implementation, Seattle, Washington, 2006.
17. Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, "The Case for Evaluating MapReduce Performance Using Workload Suites, " in Proceedings of the 2011 IEEE 19th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, 2011.
18. Lakshman and P. Malik, "Cassandra: a decentralized structured storage system, " SIGOPS Oper. Syst. Rev., vol. 44, pp. 35-40, 2010.
19. M. Y. Eltabakh, Y. Tian, F. Ozcan, R. Gemulla, A. Krettek, and J. McPherson, "CoHadoop: flexible data placement and its exploitation in Hadoop, " Proc. VLDB Endow., vol. 4, pp. 575-585, 2011.
20. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, and H. Liu, "Data warehousing and analytics infrastructure at facebook, " in Proceedings of the 2010 international conference on Management of data, Indianapolis, Indiana, USA, 2010.
21. M. K. McKusick and S. Quinlan, "GFS: Evolution on Fastforward, " Queue, vol. 7, pp. 10-20, 2009.
22. S. Ghemawat, H. Gobioff, and S. T. Leung, "The Google file system, " in Proceedings of the 9th ACM Symposium on Operating Systems Principles (SOSP 03), Bolton Landing, NY, USA, 2003.
23. J. Shafer, S. Rixner, and A. L. Cox, "The Hadoop distributed filesystem: Balancing portability and performance, " in Proceedings of the IEEE International Symposium on Performance Analysis of Systems & Software (ISPASS 10), 2010, pp. 122-133.
24. E. Wu, Y. Diao, and S. Rizvi, "High-performance complex event processing over streams, " in Proceedings of the 2006 ACM SIGMOD international conference on Management of data, Chicago, IL, USA, 2006.
25. W. Tantisiriroj, S. W. Son, S. Patil, S. J. Lang, G. Gibson, and R. B. Ross, "On the duality of data-intensive file system design: reconciling HDFS and PVFS, " in Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, Seattle, Washington, 2011.
26. Calder, J. Wang, A. Ogus, N. Nilakantan, A. Skjolsvold, S. McKelvie, Y. Xu, S. Srivastav, J. Wu, H. Simitci, J. Haridas, C. Uddaraju, H. Khatri, A. Edwards, V. Bedekar, S. Mainali, R. Abbasi, A. Agarwal, M. F. u. Haq, M. I. u. Haq, D. Bhardwaj, S. Dayanand, A. Adusumilli, M. McNett, S. Sankaran, K. Manivannan, and L. Rigas, "Windows Azure Storage: a highly available cloud storage service with strong consistency, " in Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, Cascais, Portugal, 2011.