Omega: Flexible, scalable schedulers for large compute clusters

Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys 2013

Volumn , Issue , 2013, Pages 351-364

  Schwarzkopf, Malte ^a   Konwinski, Andy ^b   Abd El Malek, Michael ^c   Wilkes, John ^c  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c GOOGLE INC   (United States)

Author keywords

cluster scheduling; optimistic concurrency control

Indexed keywords

SCHEDULING;

CLUSTER GROWTH; CLUSTER SCHEDULING; LOCK-FREE; OPTIMISTIC CONCURRENCY CONTROLS; PRODUCTION WORKLOADS; RAPID RESPONSE; SHARED STATE; USE-CASE;

CONCURRENCY CONTROL;

EID: 84877711648     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2465351.2465386     Document Type: Conference Paper

References (29)

1. ADAPTIVE COMPUTING ENTERPRISES INC. Maui Scheduler Administrator's Guide, 3.2 ed. Provo, UT, 2011.
2. ADL-TABATABAI, A.-R., LEWIS, B. T., MENON, V., MURPHY, B. R., SAHA, B., AND SHPEISMAN, T. Compiler and runtime support for efficient software transactional memory. In Proceedings of PLDI (2006), pp. 26-37.
3. ANANTHANARAYANAN, G., DOUGLAS, C., RAMAKRISHNAN, R., RAO, S., AND STOICA, I. True elasticity in multi-tenant data-intensive compute clusters. In Proceedings of SoCC (2012), p. 24.
4. APACHE. Hadoop On Demand. http://goo.gl/px8Yd, 2007. Accessed 20/06/2012.
5. CHANG, F., DEAN, J., GHEMAWAT, S., HSIEH, W. C., WALLACH, D. A., BURROWS, M., CHANDRA, T., FIKES, A., AND GRUBER, R. E. Bigtable: A Distributed Storage System for Structured Data. ACM Transactions on Computer Systems 26, 2 (June 2008), 4:1-4:26.
6. CHEN, Y., ALSPAUGH, S., BORTHAKUR, D., AND KATZ, R. Energy efficiency for large-scale MapReduce workloads with significant interactive analysis. In Proceedings of EuroSys (2012).
7. CHEN, Y., GANAPATHI, A. S., GRIFFITH, R., AND KATZ, R. H. Design insights for MapReduce from diverse production workloads. Tech. Rep. UCB/EECS-2012-17, UC Berkeley, Jan. 2012.
8. DEAN, J., AND GHEMAWAT, S. MapReduce: Simplified data processing on large clusters. CACM 51, 1 (2008), 107-113.
9. ENGLER, D. R., KAASHOEK, M. F., AND O'TOOLE, JR., J. Exokernel: an operating system architecture for application-level resource management. In Proceedings of SOSP (1995), pp. 251-266.
10. FERGUSON, A. D., BODIK, P., KANDULA, S., BOUTIN, E., AND FONSECA, R. Jockey: guaranteed job latency in data parallel clusters. In Proceedings of EuroSys (2012), pp. 99-112.
11. GHODSI, A., ZAHARIA, M., HINDMAN, B., KONWINSKI, A., SHENKER, S., AND STOICA, I. Dominant resource fairness: fair allocation of multiple resource types. In Proceedings of NSDI (2011), pp. 323-336.
12. HERODOTOU, H., DONG, F., AND BABU, S. No one (cluster) size fits all: automatic cluster sizing for data-intensive analytics. In Proceedings of SoCC (2011).
13. HINDMAN, B., KONWINSKI, A., ZAHARIA, M., GHODSI, A., JOSEPH, A., KATZ, R., SHENKER, S., AND STOICA, I. Mesos: a platform for fine-grained resource sharing in the data center. In Proceedings of NSDI (2011).
14. IQBAL, S., GUPTA, R., AND FANG, Y.-C. Planning considerations for job scheduling in HPC clusters. Dell Power Solutions (Feb. 2005).
15. ISARD, M., PRABHAKARAN, V., CURREY, J., WIEDER, U., TALWAR, K., AND GOLDBERG, A. Quincy: fair scheduling for distributed computing clusters. In Proceedings of SOSP (2009).
16. JACKSON, D. AND SNELL, Q. AND CLEMENT, M. Core algorithms of the Maui scheduler. In Job Scheduling Strategies for Parallel Processing. 2001, pp. 87-102.
17. KAVULYA, S., TAN, J., GANDHI, R., AND NARASIMHAN, P. An analysis of traces from a production MapReduce cluster. In Proceedings of CCGrid (2010), pp. 94-103.
18. KUNG, H. T., AND ROBINSON, J. T. On optimistic methods for concurrency control. ACM Transactions on Database Systems 6, 2 (June 1981), 213-226.
19. MALEWICZ, G., AUSTERN, M., BIK, A., DEHNERT, J., HORN, I., LEISER, N., AND CZAJKOWSKI, G. Pregel: a system for large-scale graph processing. In Proceedings of SIGMOD (2010), pp. 135-146.
20. MISHRA, A. K., HELLERSTEIN, J. L., CIRNE, W., AND DAS, C. R. Towards characterizing cloud backend workloads: insights from Google compute clusters. SIGMETRICS Performance Evaluation Review 37 (Mar. 2010), 34-41.
21. MURTHY, A. C., DOUGLAS, C., KONAR, M., O'MALLEY, O., RADIA, S., AGARWAL, S., AND K V, V. Architecture of next generation Apache Hadoop MapReduce framework. Tech. rep., Apache Hadoop, 2011.
22. PAN, H., HINDMAN, B., AND ASANOVIĆ, K. Lithe: enabling efficient composition of parallel libraries. In Proceedings of HotPar (2009).
23. PENG, D., AND DABEK, F. Large-scale incremental processing using distributed transactions and notifications. In Proceedings of OSDI (2010).
24. REISS, C., TUMANOV, A., GANGER, G. R., KATZ, R. H., AND KOZUCH, M. A. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In Proceedings of SoCC (2012).
25. SHARMA, B., CHUDNOVSKY, V., HELLERSTEIN, J., RIFAAT, R., AND DAS, C. Modeling and synthesizing task placement constraints in Google compute clusters. In Proceedings of SoCC (2011).
26. VERMA, A., CHERKASOVA, L., AND CAMPBELL, R. SLO-driven right-sizing and resource provisioning of MapReduce jobs. In Proceedings of LADIS (2011).
27. WILKES, J. More Google cluster data. Google research blog, Nov. 2011. Posted at http://goo.gl/9B7PA.
28. ZAHARIA, M., BORTHAKUR, D., SEN SARMA, J., ELMELEEGY, K., SHENKER, S., AND STOICA, I. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of EuroSys (2010), pp. 265-278.
29. ZHANG, Q., HELLERSTEIN, J., AND BOUTABA, R. Characterizing task usage shapes in Google's compute clusters. In Proceedings of LADIS (2011).