Coupling task progress for MapReduce resource-aware scheduling

Proceedings - IEEE INFOCOM

Volumn , Issue , 2013, Pages 1618-1626

  Tan, Jian ^a   Meng, Xiaoqiao ^a   Zhang, Li ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA LOCALITY; DESIGN AND IMPLEMENTS; MAP TASK; MAP-REDUCE; RESOURCE AWARE; RESOURCE-AWARE SCHEDULING; STRONG DEPENDENCES;

OPTIMIZATION;

SCHEDULING;

EID: 84883112222     PISSN: 0743166X     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/INFCOM.2013.6566958     Document Type: Conference Paper

References (25)

1. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," Commun. ACM, vol. 51, pp. 107-113, January 2008.
2. Hadoop, http://hadoop.apache.org.
3. Fair Scheduler, http://hadoop.apache.org/mapreduce/docs/fair-scheduler. html.
4. Capacity Scheduler, http://hadoop.apache.org/mapreduce/docs/r0.21.0/ capacity-scheduler.html.
5. J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh, K.- L. Wu, and A. Balmin, "FLEX: A slot allocation scheduling optimizer for MapReduce workloads," in Middleware 2010, ser. Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2010, vol. 6452, pp. 1-20.
6. J. Polo, C. Castillo, D. Carrera, Y. Becerra, I. Whalley, M. Steinder, J. Torres, and E. Ayguadé, "Resource-aware adaptive scheduling for MapReduce clusters," in Middleware, ser. Lecture Notes in Computer Science, vol. 7049. Springer, 2011, pp. 187-207.
7. T. Sandholm and K. Lai, "Dynamic proportional share scheduling in hadoop," in Proceedings of the 15th international conference on job scheduling strategies for parallel processing, ser. JSSPP'10. Berlin, Heidelberg: Springer-Verlag, 2010, pp. 110-131.
8. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving MapReduce performance in heterogeneous environments," in Proceedings of the 8th USENIX conference on Operating systems design and implementation (OSDI'08), 2008, pp. 29-42.
9. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris, "Reining in the outliers in map-reduce clusters using mantri," in Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation (OSDI'10), 2010, pp. 1-16.
10. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: fair scheduling for distributed computing clusters," in Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (SOSP '09). ACM, 2009, pp. 261-276.
11. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica, "Mesos: a platform for fine-grained resource sharing in the data center," in Proceedings of the 8th USENIX conference on Networked systems design and implementation (NSDI'11), 2011, pp. 22-22.
12. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Job scheduling for multi-user MapReduce clusters," EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS- 2009-55, April 2009.
13. B. Palanisamy, A. Singh, L. Liu, and B. Jain, "Purlieus: locality-aware resource allocation for MapReduce in a cloud," in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC '11, 2011, pp. 58:1-58:11.
14. M. Hammoud and M. Sakr, "Locality-aware reduce task scheduling for MapReduce," in Proceedings of 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), 2011, pp. 570-576.
15. Y. Wang, J. Tan, W. Yu, L. Zhang, X. Meng, and X. Li, "Preemptive ReduceTask scheduling for fair and fast job completion," 2013, under review.
16. S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan, "An analysis of traces from a production MapReduce cluster," in Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, ser. CCGRID '10, Washington, DC, USA, 2010, pp. 94-103.
17. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," in Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 (EuroSys '07), 2007.
18. Y. Chen, A. Ganapathi, R. Griffith, and R. H. Katz, "The case for evaluating MapReduce performance using workload suites," in Proceedings of the 19th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), July 2011.
19. J. Tan, X. Meng, and L. Zhang, "Performance analysis of Coupling Scheduler for MapReduce/Hadoop," in Proceedings of the 31th IEEE International Conference on Computer Communications (INFOCOM'12),, March 2012, pp. 2586 -2590, mini conference.
20. -, "Delay tails in MapReduce scheduling," in Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE joint international conference on Measurement and Modeling of Computer Systems (SIGMETRICS '12). New York, NY, USA: ACM, 2012, pp. 5-16.
21. -, "Coupling scheduler for MapReduce/Hadoop," in Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing (HPDC'12), 2012, pp. 129-130, poster.
22. M. Hammoud, M. S. Rehman, and M. F. Sakr, "Center-of-gravity reduce task scheduling to lower MapReduce network traffic," in Proceedings of 2012 IEEE Fifth International Conference on Cloud Computing (IEEE CLOUD). IEEE, June 2012.
23. J. Tan, X. Meng, and L. Zhang, "Improving ReduceTask data locality for sequential MapReduce jobs," in Proceedings of the 32th IEEE International Conference on Computer Communications (INFOCOM'13), April 2013.
24. Hadoop distributed filesystem, http://hadoop.apache.org/hdfs/.
25. A. Verma, L. Cherkasova, and R. H. Campbell, "ARIA: Automatic Resource Inference and Allocation for MapReduce Environments," in Proc. of Intl. Conference on Autonomic Computing (ICAC), 2011.