Matchmaking: A new MapReduce scheduling technique

Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011

Volumn , Issue , 2011, Pages 40-47

  He, Chen ^a   Lu, Ying ^a   Swanson, David ^a  

^a UNIVERSITY OF NEBRASKA LINCOLN   (United States)

Author keywords

Data locality; Hadoop; MapReduce; Scheduling technique

Indexed keywords

DATA LOCALITY; ENHANCEMENT TECHNIQUES; FACEBOOK; HADOOP; INPUT DATAS; MAP TASK; MAP-REDUCE; PARAMETER-TUNING; SCHEDULING TECHNIQUE; SCHEDULING TECHNIQUES;

ALGORITHMS; CLOUD COMPUTING; DATA PROCESSING;

SCHEDULING;

EID: 84863180724     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CloudCom.2011.16     Document Type: Conference Paper

References (15)

1. J. Dean and S. Ghemawat. "MapReduce: Simplified Data Processing on Large Clusters". Commun. ACM, 51(1):107-113, 2008.
2. Apache Hadoop. http://hadoop.apache.org.
3. Amazon EC2. http://aws.amazon.com/ec2/
4. M.C. Schatz, "BlastReduce: high performance short read mapping with MapReduce". http://www.cbcb.umd.edu/software/blastreduce/.
5. M. Zaharia et al. "Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling". In EuroSys, 2010.
6. HDFS. http://hadoop.apache.org/hdfs/
7. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Job scheduling for multi-user mapreduce clusters," EECS Department, University of California, Berkeley, Tech. Rep., Apr 2009.
8. X. Zhang, Z. Zhong, S. Feng, B. Tu, J. Fan, "Improving Data Locality of MapReduce by Scheduling in Homogeneous Computing Environments", in 9th IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA), pp. 120-126, 2011.
9. M. Zaharia, A. Konwinski, A.D. Joseph, R. Katz, I. Stoica, "Improving MapReduce performance in heterogeneous environments", in: Proc. 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008, San Diego,USA, Dec. 2008.
10. Q. Chen, D. Zhang, M. Guo, Q. Deng, and S. Guo, "SAMR: A selfadaptive MapReduce scheduling algorithm in heterogeneous environment", in 10th IEEE International Conference on Computer and Information Technology (CIT'10), pp. 2736-2743, 2010.
11. Capacity Scheduler http://hadoop.apache.org/common/docs/r0.19.2/capacity- scheduler.h tml
12. J. Polo, D. Carrera, Y. Becerra, J. Torres, E. Ayguade and, M. Steinder, and I. Whalley, "Performance-driven task co-scheduling for mapreduce environments," in Network Operations and Management Symposium (NOMS), 2010 IEEE, 2010, pp. 373-380.
13. K. Kc and K. Anyanwu, "Scheduling hadoop jobs to meet deadlines," in 2nd IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 388 -392, 2010.
14. Fair Scheduler, http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair- scheduler.html
15. S. Seo, I. Jang, K. Woo, I. Kim, J.-S. Kim, and S. Maeng. "HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment". In Proc. CLUSTER'10, pp. 1-8, 2009.