A dynamic MapReduce Scheduler for heterogeneous workloads

8th International Conference on Grid and Cooperative Computing, GCC 2009

Volumn , Issue , 2009, Pages 218-224

  Tian, Chao ^a,b   Zhou, Haojie ^a   He, Yongqiang ^a,b   Zha, Li ^a  

^a INSTITUTE OF COMPUTING TECHNOLOGY   (China)

^b UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

Component; Heterogeneous workloads; MapReduce; Schdule

Indexed keywords

CLASSIFICATION ,; DATA CENTERS; DISK I/O; ON THE FLIES; PARALLELIZATIONS; PROGRAMMING MODELS;

COMPUTER SCIENCE; MACHINE DESIGN; SATELLITE COMMUNICATION SYSTEMS; SCHEDULING;

GRID COMPUTING;

EID: 72349090439     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/GCC.2009.19     Document Type: Conference Paper

References (18)

1. Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," In Communications of the ACM, Volume 51, Issue 1, pp. 107-113, 2008.J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68-73.
2. Matei Zaharia, Andy Konwinski, Anthony D. Joseph, and Randy Katz Ion Stoica, "Improving MapReduce Performance in Heterogeneous Environments," Proceedings of the 8th conference on Symposium on Opearting Systems Design & Implementation
3. Yahoo! Launches World's Largest Hadoop Production Application, Yahoo! Developer Network, http://developer.yahoo.com/blogs/hadoop/2008/02/yahoo-worlds- largest-production-hadoop.html.
4. Hadoop's Capacity Scheduler http://hadoop.apache.org/core/docs/current/ capacity-scheduler.html.
5. Matei Zaharia, "The Hadoop Fair Scheduler" http://developer. yahoo.net/blogs/hadoop/FairSharePres.ppt
6. E. Rosti, G. Serazzi, E. Smirni, and M.S. Squillante, "The Impact of I/O on Program Behavior and Parallel Scheduling," Proc. SIGMETRICS Conf. Measurement and Modeling of Computing Systems, 1998, pp. 56-65,
7. Yair Wiseman and Dror G. Feitelson, "Paired Gang Scheduling," IEEE Transactions on Parallel and Distributed System, vol. 14, no. 6, June 2003
8. Tera Sort Benchmark, http://www.hpl.hp.com/hosted/sortbenchmark/.
9. Parag Agrawal, Daniel Kifer, and Christopher Olston, "Scheduling Shared Scans of Large Data Files," PVLDB '08, August 2008, pp.23-28
10. E. Rosti, G. Serazzi, E. Smirni, and M.S. Squillante, "Models of Parallel Applications with Large Computation and I/O Requirements," IEEE Trans. Software Eng., vol. 28, no. 3, Mar.2002, pp. 286-307
11. M.J. Atallah, C.L. Black, D.C. Marinescu, H.J. Siegel and T.L. Casavant, "Models and algorithms for co-scheduling compute-intensive asks on a network of workstations," Journal of Parallel and Distributed Computing 16, 1992, pp.319-327
12. D.G. Feitelson and L. Rudolph, "Gang scheduling performance benefitsfor fine-grained synchronization," Journal of Parallel and Distributed Computing 16(4),December 1992, pp.306-318
13. J.K. Ousterhout, "Scheduling techniques for concurrent systems," in Proc. of 3rd Int. Conf. on Distributed Computing Systems, May 1982, pp.22-30.
14. W. Lee, M. Frank, V. Lee, K. Mackenzie, and L. Rudolph, "Implications of I/O for Gang Scheduled Workloads," Job Scheduling Strategies for Parallel Processing, 1997, pp. 215-237
15. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, "The Google file system," In Proceedings of 19th Symposium on Operating Systems Principles, 2003, pp. 29-43
16. H. Lee, D. Lee and R.S. Ramakrishna, "An Enhanced Grid Scheduling with Job Priority and Equitable Interval Job Distribution," The first International Conference on Grid and Pervasive Computing, Lecture Notes in Computer Science, vol. 3947, May 2006, pp. 53-62
17. A.J. Page and T.J. Naughton, "Dynamic task scheduling using genetic algorithms for heterogeneous distributed computing," in 19th IEEE International Parallel and Distributed Processing Symposium, 2005.
18. M. Pinedo, "Scheduling: Theory, Algorithms, and Systems," Prentice Hall, Englewood Cliffs, NJ, 1995.