Performance-driven task co-scheduling for mapreduce environments

Proceedings of the 2010 IEEE/IFIP Network Operations and Management Symposium, NOMS 2010

Volumn , Issue , 2010, Pages 373-380

  Polo, Jordà ^a   Carrera, David ^a   Becerra, Yolanda ^a   Torres, Jordi ^a   Ayguadé, Eduard ^a   Steinder, Malgorzata ^b   Whalley, Ian ^b  

^a BARCELONA SUPERCOMPUTING CENTER   (Spain)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

MapReduce; Performance management; Task scheduling

Indexed keywords

ENERGY CONSERVATION; MULTITASKING; SCHEDULING;

DATA CENTER MANAGEMENT; DISTRIBUTED DATA ANALYTICS; MAP-REDUCE; MAPREDUCE FRAMEWORKS; MULTIPLE APPLICATIONS; PERFORMANCE MANAGEMENT; PERFORMANCE OBJECTIVE; TASK-SCHEDULING;

MULTIPROCESSING SYSTEMS;

EID: 77957760781     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/NOMS.2010.5488494     Document Type: Conference Paper

References (16)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in OSDI '04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December 2004, pp. 137-150. [Online], Available: http://labs.google.com/papers/mapreduce.html
2. Apache Software Foundation. Hadoop MapReduce Tutorial. [Online]. Available: http://hadoop.apache.org/mapreduce/
3. -. Hadoop Distributed File System (HDFS) Architecture. [Online]. Available: http://hadoop.apache.org/core/docs/current/hdfs-design.html
4. D. Carrera, M. Steinder, I. Whalley, J. Torres, and E. Ayguadé, "Enabling resource sharing between transactional and batch workloads using dynamic application placement," in Middleware '08: Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware, 2008, pp. 203-222.
5. C. K. Jacob Leverich, "On the energy (in)efficiency of Hadoop clusters," in HotPower'09, 2009.
6. J. Polo, "Deadline scheduler for Hadoop," 2009. [Online]. Available: http://www.bsc.es/autonomic
7. M. Zaharia, "Hadoop Fair Scheduler Design Document," 2009. [Online]. Available: http://developer.yahoo.net/blogs/hadoop/FairSharePres.ppt
8. D. G. Feitelson and L. Rudolph, "Parallel job scheduling: Issues and approaches," in JSSPP, 1995, pp. 1-18.
9. D. G. Feitelson, "Job scheduling in multiprogrammed parallel systems," Tech. Rep. RC 19790 (87657), August 1997.
10. C. E. Volker, V. Hamscher, and R. Yahyapour, "Economic scheduling in grid computing," in Scheduling Strategies for Parallel Processing. Springer, 2002, pp. 128-152.
11. P.R. Jelenkovic, X. Kang, and J. Tan, "Adaptive and scalable comparison scheduling," in SIGMETRICS '07: Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems. New York, NY, USA: ACM, 2007, pp. 215-226.
12. A. Wierman and M. Nuyens, "Scheduling despite inexact job-size information," in SIGMETRICS '08: Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems. New York, NY, USA: ACM, 2008, pp. 25-36.
13. Apache Software Foundation. Hadoop on demand. [Online]. Available: http://hadoop.apache.org/core/docs/r0.20.0/hod-user-guide.html
14. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Job scheduling for multi-user MapReduce clusters," EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2009-55, Apr 2009. [Online]. Available: http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/ EECS-2009-55.html
15. T. Sandholm and K. Lai, "MapReduce optimization using regulated dynamic prioritization," in SIGMETRICS '09: Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems. New York, NY, USA: ACM, 2009, pp. 299-310.
16. Autonomic Systems and eBusiness Platforms research line. Barcelona Supercomputing Center (BSC). Emotive cloud. [Online]. Available: http://www.emotivecloud.net