Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters

Proceedings of the 2015 USENIX Annual Technical Conference, USENIX ATC 2015

Volumn , Issue , 2015, Pages 485-497

  Karanasos, Konstantinos ^a   Raol, Sriram ^a   Curino, Carlo ^a   Douglas, Chris ^a   Chaliparambil, Kishore ^a   Fumarola, Giovanni Matteo ^a   Ramakrishnan, Solom Heddaya Raghu ^a   Sakalanaga, Sarvesh ^a  

^a MICROSOFT   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SOFTWARE; ECONOMIC AND SOCIAL EFFECTS; MERCURY (METAL); THROUGHPUT;

BLENDED APPROACH; DISTRIBUTED SOLUTIONS; ECONOMY OF SCALE; EFFICIENT SCHEDULING; FULL SPECTRUM; HYBRID CENTRALIZED; HYBRID RESOURCE MANAGEMENTS; JOB THROUGHPUT;

SCHEDULING;

EID: 85077131857     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Apache Hadoop Capacity Scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html.
2. Apache Hadoop Fair Scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html.
3. Apache Hadoop Project. http://hadoop.apache.org/.
4. Apache Tez. https://tez.apache.org.
5. Enable YARN RM scale out via federation using multiple RM's. https://issues.apache.org/jira/browse/YARN-2915.
6. Extend YARN to support distributed scheduling. https://issues.apache.org/jira/browse/YARN-2877.
7. Hadoop Distributed Filesystem. http://hadoop.apache.org/hdfs.
8. Hadoop Gridmix. http://hadoop.apache.org/docs/r1.2.1/gridmix.html.
9. BOUTIN, E., EKANAYAKE, J., LIN, W., SHI, B., ZHOU, J., QIAN, Z., WU, M., AND ZHOU, L. Apollo: Scalable and coordinated scheduling for cloud-scale computing. In OSDI (2014).
10. CARVALHO, M., CIRNE, W., BRASILEIRO, F., AND WILKES, J. Long-term SLOs for reclaimed cloud computing resources. In SoCC (2014).
11. CURINO, C., DIFALLAH, D. E., DOUGLAS, C., KRISHNAN, S., RAMAKRISHNAN, R., AND RAO, S. Reservation-based scheduling: If youre late dont blame us! In SoCC (2014).
12. DEAN, J., AND BARROSO, L. A. The tail at scale. Communications of the ACM 56 (2013), 74-80.
13. DELIMITROU, C., AND KOZYRAKIS, C. Quasar: Resource-efficient and qos-aware cluster management. In ASPLOS (2014).
14. GHODSI, A., ZAHARIA, M., HINDMAN, B., KONWINSKI, A., SHENKER, S., AND STOICA, I. Dominant resource fairness: Fair allocation of multiple resource types. In NSDI (2011).
15. GRANDL, R., ANANTHANARAYANAN, G., KANDULA, S., RAO, S., AND AKELLA, A. Multiresource packing for cluster schedulers. In SIGCOMM (2014).
16. HINDMAN, B., KONWINSKI, A., ZAHARIA, M., GHODSI, A., JOSEPH, A. D., KATZ, R. H., SHENKER, S., AND STOICA, I. Mesos: A platform for fine-grained resource sharing in the data center. In NSDI (2011).
17. ISARD, M., PRABHAKARAN, V., CURREY, J., WIEDER, U., TALWAR, K., AND GOLDBERG, A. Quincy: Fair scheduling for distributed computing clusters. In SOSP (2009).
18. KARANASOS, K., RAO, S., CURINO, C., DOUGLAS, C., CHALIPARAMBIL, K., FUMAROLA, G., HEDDAYA, S., RAMAKRISHNAN, R., AND SAKALANAGA, S. Mercury: Hybrid centralized and distributed scheduling in large shared clusters. Tech. Rep. MSR-TR-2015-6, 2015.
19. KORNACKER, M., BEHM, A., BITTORF, V., BOBROVYTSKY, T., CHING, C., CHOI, A., ERICKSON, J., GRUND, M., HECHT, D., JACOBS, M., JOSHI, I., KUFF, L., KUMAR, D., LEBLANG, A., LI, N., PANDIS, I., ROBINSON, H., RORKE, D., RUS, S., RUSSELL, J., TSIROGIANNIS, D., WANDERMAN-MILNE, S., AND YODER, M. Impala: A modern, open-source SQL engine for hadoop. In CIDR (2015).
20. LI, H., GHODSI, A., ZAHARIA, M., SHENKER, S., AND STOICA, I. Tachyon: Reliable, memory speed storage for cluster computing frameworks. In SoCC (2014).
21. MELNIK, S., GUBAREV, A., LONG, J. J., ROMER, G., SHIVAKUMAR, S., TOLTON, M., AND VASSILAKIS, T. Dremel: Interactive analysis of web-scale datasets. In PVLDB (2010).
22. OUSTERHOUT, K., WENDELL, P., ZAHARIA, M., AND STOICA, I. Sparrow: Distributed, low latency scheduling. In SOSP (2013).
23. SANDHOLM, T., AND LAI, K. Dynamic proportional share scheduling in hadoop. In Job scheduling strategies for parallel processing (2010).
24. SCHWARZKOPF, M., KONWINSKI, A., ABD-ELMALEK, M., AND WILKES, J. Omega: Flexible, scalable schedulers for large compute clusters. In EuroSys (2013).
25. SHARMA, B., PRABHAKAR, R., LIM, S., KANDEMIR, M. T., AND DAS, C. R. MROrchestrator: A fine-grained resource orchestration framework for mapreduce clusters. In Cloud Computing (CLOUD) (2012).
26. STAPLES, G. TORQUE resourcemanager. In IEEE SC (2006).
27. VAVILAPALLI, V. K., MURTHY, A. C., DOUGLAS, C., AGARWAL, S., KONAR, M., EVANS, R., GRAVES, T., LOWE, J., SHAH, H., SETH, S., SAHA, B., CURINO, C., O'MALLEY, O., RADIA, S., REED, B., AND BALDESCHWIELER, E. Apache hadoop YARN: Yet another resource negotiator. In SoCC (2013).
28. VERMA, A., PEDROSA, L., KORUPOLU, M. R., OPPENHEIMER, D., TUNE, E., AND WILKES, J. Large-scale cluster management at Google with Borg. In EuroSys (2015).
29. WEIMER, M., CHEN, Y., CHUN, B.-G., CONDIE, T., CURINO, C., DOUGLAS, C., LEE, Y., MAJESTRO, T., MALKHI, D., MATUSEVYCH, S., MYERS, B., NARAYANAMURTHY, S., RAMAKRISHNAN, R., RAO, S., SEARS, R., SEZGIN, B., AND WANG, J. REEF: Retainable evaluator execution framework. In SIGMOD (2015).
30. YOO, A. B., JETTE, M. A., AND GRONDONA, M. SLURM: Simple linux utility for resource management. In Job Scheduling Strategies for Parallel Processing (2003).
31. ZAHARIA, M., BORTHAKUR, D., SEN SARMA, J., ELMELEEGY, K., SHENKER, S., AND STOICA, I. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In EuroSys (2010).
32. ZAHARIA, M., CHOWDHURY, M., DAS, T., DAVE, A., MA, J., MCCAULEY, M., FRANKLIN, M. J., SHENKER, S., AND STOICA, I. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI (2012).
33. ZAHARIA, M., CHOWDHURY, M., FRANKLIN, M. J., SHENKER, S., AND STOICA, I. Spark: Cluster computing with working sets. In HotCloud (2010).
34. ZHOU, J., BRUNO, N., WU, M.-C., LARSON, P.-A., CHAIKEN, R., AND SHAKIB, D. Scope: Parallel databases meet mapreduce. VLDB J. 21, 5 (2012), 611-636.