Center-of-gravity reduce task scheduling to lower MapReduce network traffic

Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012

Volumn , Issue , 2012, Pages 49-58

  Hammoud, Mohammad ^a   Rehman, M Suhail ^a   Sakr, Majd F ^a  

^a CARNEGIE MELLON UNIVERSITY   (Qatar)

Author keywords

Hadoop; MapReduce; Reduce Task Scheduling

Indexed keywords

AMAZON EC2; CENTER OF GRAVITY; CLUSTER NETWORKS; COMPUTING PLATFORM; DATA LOCALITY; DISTRIBUTED DATA; HADOOP; JOB EXECUTION; MAP-REDUCE; MULTIPLE MACHINE; NETWORK TRAFFIC; OPEN SOURCE IMPLEMENTATION; PRIVATE CLOUDS; TASK-SCHEDULING;

CLOUD COMPUTING; SCHEDULING ALGORITHMS;

MULTITASKING;

EID: 84866769125     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLOUD.2012.92     Document Type: Conference Paper

References (35)

1. Amazon Elastic Compute Cloud, "http://aws.amazon.com/ec2/."
2. P. C. Chen, Y. L. Su, J. B. Chang, and C. K. Shieh, "Variable-Sized Map and Locality-Aware Reduce on Public-Resource Grids," GPC, 2010.
3. J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing On Large Clusters," OSDI, 2004.
4. Fedora, "http://fedoraproject.org/."
5. S. Ghemawat, H. Gobioff, and S. T. Leung, "The Google File System," SOSP, 2003.
6. Hadoop, "http://hadoop.apache.org/."
7. Tom White, "Hadoop: The Definitive Guide" 1st Edition, O'REILLY, 2009.
8. Hadoop Tutorial, "http://developer.yahoo.com/hadoop/tutorial/."
9. M. Hammoud and M. Sakr, "Locality-Aware Reduce Task Scheduling for MapReduce," CloudCom, 2010.
10. S. Huang, J. Huang, J. Dai, T. Xie, B. Huang, "The HiBench Benchmark Suite: Characterization of the MapReduce-Based Data Analysis," ICDEW, 2010.
11. B. He, W. Fang, Q. Luo, N.K. Govindaraju, T. Wang, "Mars: a MapReduce Framework on Graphics Processors," PACT, 2008.
12. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, L. Qi, "CLOUDLET: Towards Mapreduce Implementation on Virtual Machines," HPDC, 2009.
13. S. Ibrahim, H. Jin, L. Lu, S. Wu, B. He, and L. Qi, "LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud," CloudCom, 2010.
14. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks," EuroSys, 2007.
15. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: Fair Scheduling for Distributed Computing Clusters," SOSP, 2009.
16. JDK, "http://download.oracle.com/javase/6/docs/."
17. K. Kc and K. Anyanwu, "Scheduling Hadoop Jobs to Meet Deadlines," CloudCom, 2010.
18. Y. C. Kwon, M. Balazinska, B. Howe, and J. Rolia, "Skew-Resistant Parallel Processing of Feature-Extracting Scientific User-Defined Functions," SOCC, 2010.
19. Y. C. Kwon, M. Balazinska, B. Howe, and J. Rolia, "A Study of Skew in MapReduce Applications," Open Cirrus Summit, 2011.
20. J. Lin, "The Curse of Zipf and Limits to Parallelization: A Look at the Stragglers Problem in MapReduce," LSDS-IR, 2009.
21. Mahout Homepage, "http://mahout.apache.org/."
22. Netperf, "http://www.netperf.org/."
23. L.T.X. Phan, Z. Zhang, B.T. Loo, and I. Lee, "Real-Time MapReduce Scheduling," Tech. R No. MS-CIS-10-32, UPenn, 2010.
24. J. Polo, D. Carrera, Y. Becerra, V. Beltran, J. Torres, and E. Ayguadé, "Performance Management of Accelerated Mapreduce Workloads in Heterogeneous Clusters," ICPP, 2010.
25. J. Polo, D. Carrera, Y. Becerra, M. Steinder, and I. Whalley, "Performance-Driven Task Co-Scheduling for MapReduce Environments," NOMS, 2010.
26. X. Qiu, J. Ekanayake, S. Beason, T. Gunarathne, G. Fox, R. Barga, and D. Gannon, "Cloud Technologies for Bioinformatics Applications," MTAGS, 2009.
27. M. Rafique, B. Rose, A. Butt, and D. Nikolopoulos, "Supporting MapReduce on Large-Scale Asymmetric Multi-Core Clusters," SIGOPS Operating Systems Review 43, 2009.
28. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, C. Kozyrakis, "MapReduce for Multi-Core and Multiprocessor Systems," HPCA, 2007.
29. T. Sandholm and K. Lai, "Dynamic Proportional Share Scheduling in Hadoop," Workshop on Job Scheduling Strategies for Parallel Processing, 2010.
30. S. Seo, I. Jang, K. Woo, I. Kim, J. Kim, S. Maeng, "HPMR: Prefetching and Pre-Shuffling in Shared MapReduce Computation Environment," CLUSTER, 2009.
31. C. Tian, H. Zhou, Y. He, and L. Zha, "A Dynamic MapReduce Scheduler for Heterogeneous Workloads," GCC, 2009.
32. VMware, "http://www.vmware.com/."
33. J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh, K.L. Wu, and A. Balmin, "FLEX: A Slot Allocation Scheduling Optimizer for MapReduceWorkloads," Middleware, 2010.
34. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling," EuroSys, 2010.
35. M. Zaharia, A. Konwinski, A. Joseph, R. Katz, I. Stoica, "Improving Mapreduce Performance in Heterogeneous Environments," OSDI, 2008.