Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling

EuroSys'10 - Proceedings of the EuroSys 2010 Conference

Volumn , Issue , 2010, Pages 265-278

  Zaharia, Matei ^a   Borthakur, Dhruba ^b   Sen Sarma, Joydeep ^b   Elmeleegy, Khaled ^c   Shenker, Scott ^a   Stoica, Ion ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b FACEBOOK   (United States)

^c YAHOO RESEARCH   (United States)

Author keywords

cluster computing; fair sharing; mapreduce; scheduling

Indexed keywords

CLUSTER COMPUTING SYSTEM; CLUSTER SCHEDULING; DATA LOCALITY; DELAY SCHEDULING; FACEBOOK; FAIR SHARING; INPUT DATAS; MAPREDUCE; OPTIMAL DATA; SCHEDULING POLICIES; SIMPLE ALGORITHM;

COMPUTER SYSTEMS; LAUNCHING; SCHEDULING ALGORITHMS;

CLUSTER COMPUTING;

EID: 77954636142     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1755913.1755940     Document Type: Conference Paper

References (30)

1. Amazon EC2. http://aws.amazon.com/ec2.
2. Apache Hadoop. http://hadoop.apache.org.
3. Apache Hive. http://hadoop.apache.org/hive.
4. Hadoop Map/Reduce tutorial. http://hadoop.apache.org/common/docs/current/ mapred tutorial.html.
5. Hive performance benchmarks. http://issues.apache.org/jira/browse/HIVE- 396.
6. HP Neoview Workload Management Services Guide. http://www.docs.hp.com/en/ 544806-001/Neoview-WMS-Guide-R2.3.pdf.
7. Max-Min Fairness (Wikipedia). http://en.wikipedia.org/wiki/Max-min- fairness.
8. NSF Cluster Exploratory (CluE) Program Solicitation. http://nsf.gov/pubs/ 2008/nsf08560/nsf08560.htm.
9. Official Google Blog: Sorting 1PB with MapReduce. http://googleblog. blogspot.com/2008/11/sorting-1pb-with-mapreduce.html.
10. Open Cirrus. http://www.opencirrus.org.
11. Personal communication with Hans Zeller of HP.
12. Personal communication with Owen O'Malley of the Yahoo! Hadoop team.
13. TORQUE Resource Manager. http://www.clusterresources.com/pages/products/ torque-resource-manager.php.
14. Yahoo! Launches New Program to Advance Open-Source Software for Internet Computing. http://research.yahoo.com/node/1879.
15. J. Bennett and H. Zhang. WF2Q: Worst-case fair weighted fair queueing. In IEEE INFOCOM'96, pages 120-128, 1996.
16. J. Bent, D. Thain, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, and M. Livny. Explicit control in a batch-aware distributed file system. In NSDI'04, 2004.
17. A. Chervenak, E. Deelman, M. Livny, M.-H. Su, R. Schuler, S. Bharathi, G. Mehta, and K. Vahi. Data Placement for Scientific Applications in Distributed Environments. In Proc. 8th IEEE/ACM International Conference on Grid Computing (Grid 2007), September 2007.
18. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1):107-113, 2008.
19. A. Demers, S. Keshav, and S. Shenker. Analysis and simulation of a fair queueing algorithm. In Journal of Internetworking Research and Experience, pages 3-26, 1990.
20. S. Floyd and V. Jacobson. Link-sharing and resource management models for packet networks. IEEE/ACM Transactions on Networking, 3(4):365-386, 1995.
21. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In Proc. SOSP 2003, pages 29-43, 2003.
22. B. Hindman, A. Konwinski, M. Zaharia, and I. Stoica. A common substrate for cluster computing. In Workshop on Hot Topics in Cloud Computing (HotCloud) 2009, 2009.
23. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys 2007, pages 59-72, 2007.
24. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair scheduling for distributed computing clusters. In SOSP 2009, 2009.
25. J. Nieh and M. S. Lam. A SMART scheduler for multimedia applications. ACM TOCS, 21(2):117-163, 2003.
26. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In SIGMOD '09, 2009.
27. I. Stoica, H. Zhang, and T. Ng. A hierarchical fair service curve algorithm for link-sharing, real-time and priority service. In SIGCOMM'97, pages 162-173, Sept. 1997.
28. D. Thain, T. Tannenbaum, and M. Livny. Distributed computing in practice: the Condor experience. Concurrency and Computation Practice and Experience, 17(2-4):323-356, 2005.
29. C. A. Waldspurger. Lottery and Stride Scheduling: Flexible Proportional-Share Resource Management. PhD thesis, MIT, Laboratory of Computer Science, 1995. MIT/LCS/TR-667.
30. C. A. Waldspurger and W. E. Weihl. Lottery scheduling: Flexible proportional-share resource management. In Proc. OSDI 94, 1994.