A load balance algorithm based on nodes performance in Hadoop cluster

APNOMS 2014 - 16th Asia-Pacific Network Operations and Management Symposium

Volumn , Issue , 2014, Pages

  Gao, Zhipeng ^a   Liu, Dangpeng ^a   Yang, Yang ^a   Zheng, Jingchen ^b   Hao, Yuwen ^b  

^a BEIJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS   (China)

^b GENERAL HOSPITAL OF CHINESE PEOPLE'S ARMED POLICE FORCES   (China)

Author keywords

Hadoop; Heterogeneous environment; Load balance; MapReduce; Nodes performance

Indexed keywords

OPEN SOURCE SOFTWARE;

HADOOP; HETEROGENEOUS ENVIRONMENTS; LOAD BALANCE; MAP-REDUCE; NODES PERFORMANCE;

DATA MINING;

EID: 84941085366     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/APNOMS.2014.6996555     Document Type: Conference Paper

References (14)

1. The Apache Hadoop project, http://Hadoop. apache. org
2. Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters, " In Communications of the ACM, Volume 51, Issue 1, pp. 107-113, 2008.
3. J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp. 68-73.
4. WHITE, T. Hadoop: The Definitive Guide. O'Relly Mdeia, Inc., 2009.
5. Rafique M M, Rose B, Butt A R, et al. Supporting MapReduce on largescale asymmetric multi-core clusters[J] ACM SIGOPS Operating Systems Review, 2009, 43(2): 25-34.
6. Xie J, Yin S, Ruan X, et al. Improving MapReduce performance through data placement in heterogeneous Hadoop clusters[C]//Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), 2010 IEEE International Symposium on. IEEE, 2010: 1-9.
7. Zaharia M, Konwinski A, Joseph A D, et al. Improving MapReduce Performance in Heterogeneous Environments[C]//OSDI. 2008, 8(4): 7.
8. Guo Z, Fox G. Improving MapReduce performance in heterogeneous network environments and resource utilization[C]//Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012). IEEE Computer Society, 2012: 714-716.
9. DeWitt D J, Naughton J F, Schneider D A, et al. Practical skew handling in parallel joins[M] University of Wisconsin-Madison, Computer Sciences Department, 1992.
10. Ramakrishnan S R, Swart G, Urmanov A. Balancing Reducer skew in MapReduce workloads using progressive sampling[C]//Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 2012: 16.
11. [Martha V S, Zhao W, Xu X. h-MapReduce: A Framework for Workload Balancing in MapReduce[C]//Advanced Information Networking and Applications (AINA), 2013 IEEE 27th International Conference on. IEEE, 2013: 637-644.
12. Venner J, Cyrus S. Pro Hadoop[M] New York, NY: Apress, 2009.
13. Rao B T, Sridevi N V, Reddy V K, et al. Performance issues of heterogeneous Hadoop clusters in cloud computing[J] arXiv preprint arXiv:1207. 0894, 2012.
14. Gufler B, Augsten N, Reiser A, et al. Handing Data Skew in MapReduce[C]//Proceedings of the 1st International Conference on Cloud Computing and Services Science. 2011, 146: 574-583.