Efficient queue management for cluster scheduling

Proceedings of the 11th European Conference on Computer Systems, EuroSys 2016

Volumn , Issue , 2016, Pages

  Rasley, Jeff ^a,b   Karanasos, Konstantinos ^a   Kandula, Srikanth ^a   Fonseca, Rodrigo ^b   Vojnovic, Milan ^a   Rao, Sriram ^a  

^a MICROSOFT   (United States)

^b BROWN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BIG DATA; DISTRIBUTED COMPUTER SYSTEMS; MERCURY (METAL); QUEUEING THEORY; WOOL; YARN;

CENTRALIZED APPROACHES; CLUSTER OPERATOR; CLUSTER SCHEDULING; CLUSTER UTILIZATION; DISTRIBUTED SCHEDULERS; HETEROGENEOUS WORKLOADS; PRODUCTION WORKLOADS; STARVATION-FREEDOM;

SCHEDULING;

EID: 84971497586     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2901318.2901354     Document Type: Conference Paper

References (34)

1. M. Alizadeh, S. Yang, M. Sharif, S. Katti, N. McKeown, B. Prabhakar, and S. Shenker. pFabric: Minimal Near-optimal Datacenter Transport. In SIGCOMM, 2013.
2. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in mapreduce clusters using mantri. In OSDI, 2010.
3. T. Anderson and M. Dahlin. Operating Systems: Principles and Practice. Recursive Books, second edition, 2014.
4. Apache Hadoop Capacity Scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html.
5. Apache Hadoop Fair Scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html.
6. Apache Hadoop Project. http://hadoop.apache.org/.
7. Apache Tez. https://tez.apache.org.
8. E. Boutin, J. Ekanayake, W. Lin, B. Shi, J. Zhou, Z. Qian, M. Wu, and L. Zhou. Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing. In OSDI, 2014.
9. J. Dean and L. A. Barroso. The tail at scale. Communications of the ACM, 56:74-80, 2013. URL http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scale/fulltext.
10. P. Delgado, F. Dinu, A. Kermarrec, and W. Zwaenepoel. Hawk: Hybrid datacenter scheduling. In USENIX ATC, 2015.
11. C. Delimitrou, D. Sanchez, and C. Kozyrakis. Tarcil: Reconciling scheduling speed and quality in large shared clusters. In SoCC, 2015.
12. A. D. Ferguson, P. Bodik, S. Kandula, E. Boutin, and R. Fonseca. Jockey: Guaranteed job latency in data parallel clusters. In EuroSys, 2012.
13. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica. Dominant Resource Fairness: Fair Allocation of Multiple Resource Types. In NSDI, 2011.
14. A. Goder, A. Spiridonov, and Y. Wang. Bistro: Scheduling data-parallel jobs against live production systems. In USENIX ATC, 2015.
15. R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella. Multi-Resource Packing for Cluster Schedulers. In SIGCOMM, 2014.
16. M. P. Grosvenor, M. Schwarzkopf, I. Gog, R. N. M. Watson, A. W. Moore, S. Hand, and J. Crowcroft. Queues Don't Matter When You Can JUMP Them! In NSDI, 2015.
17. Hadoop GridMix. http://hadoop.apache.org/docs/r1.2.1/gridmix.html.
18. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In NSDI, 2011.
19. C. Hong, M. Caesar, and B. Godfrey. Finishing Flows Quickly with Preemptive Scheduling. In SIGCOMM, 2012.
20. K. Karanasos, S. Rao, C. Curino, C. Douglas, K. Chaliparambil, G. M. Fumarola, S. Heddaya, R. Ramakrishnan, and S. Sakalanaga. Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters. In USENIX ATC, 2015.
21. K. Ousterhout, A. Panda, J. Rosen, S. Venkataraman, R. Xin, S. Ratnasamy, S. Shenker, and I. Stoica. The case for tiny tasks in compute clusters. In HotOS, 2013.
22. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. Sparrow: Distributed, low latency scheduling. In SOSP, 2013.
23. J. Rasley, K. Karanasos, S. Kandula, R. Fonseca, M. Vojnovic, and S. Rao. Efficient queue management for cluster scheduling. Technical Report MSR-TR-2016-12, 2016. URL http://research.microsoft.com/apps/pubs/default.aspx?id=263820.
24. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes. Omega: Flexible, Scalable Schedulers for Large Compute Clusters. In EuroSys, 2013.
25. M. Shreedhar and G. Varghese. Efficient fair queueing using deficit round-robin. IEEE/ACM Trans. Netw., 1996.
26. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: a warehousing solution over a map-reduce framework. PVLDB, 2(2), 2009.
27. Under the Hood: Scheduling MapReduce jobs more efficiently with Corona. http://tinyurl.com/fbcorona.
28. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache Hadoop YARN: Yet Another Resource Negotiator. In SoCC, 2013.
29. A. Verma, L. Pedrosa, M. R. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes. Large-scale cluster management at Google with Borg. In EuroSys, 2015.
30. YARN-2877. Extend YARN to support distributed scheduling. https://issues.apache.org/jira/browse/YARN-2877.
31. YARN-2883. Queuing of container requests in the NM. https://issues.apache.org/jira/browse/YARN-2883.
32. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In OSDI, 2008.
33. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster computing with working sets. In HotCloud, 2010.
34. D. Zats, T. Das, P. Mohan, D. Borthakur, and R. H. Katz. DeTail: Reducing The Flow Completion Time Tail In Datacenter Networks. In SIGCOMM, 2012.