Dependency-aware and resource-efficient scheduling for heterogeneous jobs in clouds

Proceedings of the International Conference on Cloud Computing Technology and Science, CloudCom

Volumn 0, Issue , 2016, Pages 110-117

  Liu, Jinwei ^a   Shen, Haiying ^b  

^a South   (United States)

^b University of Virginia   (United States)

Author keywords

Clouds; Reinforcement learning; Resource efficiency; Response time; Scheduling; Throughput

Indexed keywords

CLOUD COMPUTING; CLOUDS; DATA ANALYTICS; MACHINE LEARNING; REINFORCEMENT LEARNING; RESPONSE TIME (COMPUTER SYSTEMS); THROUGHPUT;

EFFICIENT SCHEDULING; MULTIPLE RESOURCES; MUTUAL REINFORCEMENT; RESOURCE EFFICIENCIES; RESOURCE REQUIREMENTS; RESOURCE UTILIZATIONS; RESOURCE-EFFICIENT; TASK DEPENDENCIES;

SCHEDULING;

EID: 85012952393     PISSN: 23302194     EISSN: 23302186     Source Type: Conference Proceeding    
DOI: 10.1109/CloudCom.2016.0032     Document Type: Conference Paper

References (48)

1. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. Batch sampling: Low overhead scheduling for sub-second prallel jobs. University of California, Berkeley, 2012
2. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. Sparrow: Distributed, low latency scheduling. In Proc. of ACM SOSP, 2013
3. P. Delgado, F. Dinu, A.-M. Kermarrec, and W. Zwaenepoel. Hawk: Hybrid datacenter scheduling. In Proc. of USENIX ATC, 2015
4. M. Zaharia, A. Konwinski, A. Joseph, R. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In OSDI, 2008
5. M. Harchol-Balter. Task assignment with unknown duration. Journal of the ACM, 49(2): 260-288, March 2002
6. R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella. Multi-resource packing for cluster schedulers. In SIGCOMM, 2014
7. B. Sharma, V. Chudnovsky, J. L. Hellerstein, R. Rifaat, and C. R. Das. Modeling, and synthesizing task placement constraints in google compute clusters. In Proc. of SoCC, 2011
8. A. Ghodsi, M. Zaharia, S. Shenker, and I. Stoica. Choosy: Max-min fair sharing for datacenter jobs with constraints. In Eurosys, 2013
9. A. Tumanov, T. Zhu, J. W. Park, M. A. Kozuch, M. Harchol-Balter, and G. R. Ganger. Tetrisched: global rescheduling with adaptive plan-ahead in dynamic heterogeneous clusters. In Proc. of EuroSys, 2016
10. J. Rasley, K. Karanasos, S. Kandula, R. Fonseca, M. Vojnovic, and S. Rao. Efficient queue management for cluster scheduling. In Proc. of ACM EuroSys, London, 2016
11. J. Liu, H. Shen, and L. Chen. CORP: Cooperative opportunistic resource provisioning for short-lived jobs in cloud systems. In Proc. of IEEE CLUSTER, 2016
12. P. Xiong, Y. Chi, S. Zhu, J. Tatemura, C. Pu, and H. Hacigms. Activesla: a profit-oriented admission control framework for database-as-a-service providers. In Proc. of SoCC, 2011
13. C. Qiu, H. Shen, and L. Chen. Probabilistic demand allocation for cloud service brokerage. In Proc. of INFOCOM, 2016
14. E. Boutin, J. Ekanayake, W. Lin, B. Shi, and J. Zhou. Apollo: Scalable, and coordinated scheduling for cloud-scale computing. In OSDI, 2014
15. I. Pietri, G. Juve, E. Deelman, and R. Sakellariou. A performance model to estimate execution time of scientific workflows on the cloud. In Proc. of SC, 2014
16. J. Dean, and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In Proc. of OSDI, 2004
17. R. Buyya, M. Murshed, D. Abramson, and S. Venugopal. Scheduling parameter sweep applications on global grids: A deadline, and budget constrained cost-time optimisation algorithm. Software: Practice, and Experience (SPE), 35(5): 491-512, April 2005
18. P. Xiong, Y. Chi, S. Zhu, H. Moon, C. Pu, and H. Hacigms. Intelligent management of virtualized resources for database systems in cloud environment. In Proc. of ICDE, 2011
19. J. Liu, H. Shen, and X. Zhang. A survey of mobile crowdsensing techniques: A critical component for the internet of things. In Proc. of ICCCN Workshop on ContextQoS, 2016
20. P. Xiong, Z. Wang, S. Malkowski, Q. Wang, D. Jayasinghe, and C. Pu. Economical, and robust provisioning of n-tier cloud workloads: A multilevel control approach. In Proc. of ICDCS, 2011
21. J. Liu, H. Shen, and H. Hu. Load-aware, and congestion-free state management in network function virtualization. In Proc. of ICNC, 2017
22. Cplex linear program solver. http://www-01.ibm.com/software/integration/optimization/cplex-optimizer/[accessed in Jul. 2016
23. H. Shen, J. Liu, K. Chen, J. Liu, and S. Moyer. SCPS: A socialaware distributed cyber-physical human-centric search engine. IEEE Transactions on Computers (TC), 64: 518-532, 2015
24. C. Qiu, H. Shen, and L. Yu. Energy-efficient cooperative broadcast in fading wireless networks. In Proc. of INFOCOM, 2014
25. D. Shah. Gossip Algorithms. Foundations, and Trends in Networking, 2009
26. J. Liu, L. Yu, H. Shen, Y. He, and J. Hallstrom. Characterizing data deliverability of greedy routing in wireless sensor networks. In Proc. of SECON, Seattle, June 2015
27. K. Jagannathan, M. Markakis, E. Modiano, and J. N. Tsitsiklis. Queuelength asymptotics for generalized max-weight scheduling in the presence of heavy-tailed traffic. TON, 20(4): 1096-1111, 2012
28. J. Liu, H. Shen, and L. Yu. Question quality analysis, and prediction in community question answering services with coupled mutual reinforcement. TSC, PP(99): 1-14, 2015
29. Palmetto cluster. http://citi.clemson.edu/palmetto/ [accessed in Jul. 2016
30. Amazon EC2. http://aws.amazon.com/ec2 [accessed in Jul. 2016
31. Google trace. https://code.google.com/p/googleclusterdata/[accessed in Jul. 2016
32. G. Ananthanarayanan, C. Douglas, R. Ramakrishnan, S. Rao, and I. Stoica. True elasticity in multi-tenant data-intensive compute clusters. In Proc. of SoCC, 2012
33. B. Cho, M. Rahman, T. Chajed, I. Gupta, C. Abad, N. Roberts, and P. Lin. Natjam: Design, and evaluation of eviction policies for supporting priorities, and deadlines in mapreduce clusters. In Proc. of SoCC, 2013
34. A. Balasubramanian, A. Sussman, and N. Sadeh. Decentralized preemptive scheduling across heterogeneous multi-core grid resources. In Proc. of JSSPP, 2013
35. M. Harchol-Balter, B. Schroeder, N. Bansal, and M. Agrawal. Size-based scheduling to improve web performance. ACM Trans. on Computer Systems, 21(2): 207-233, 2003
36. Z. Li, H. Shen, W. Ligon, and J. Denton. An exploration of designing a hybrid scale-up/out hadoop architecture based on performance measurements. TPDS, PP(99): 1-1, 2016
37. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: A simple technique for achieving locality, and fairness in cluster scheduling. In Proc. of EuroSys, 2010
38. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde. Falkon: a fast, and light-weight task execution framework. In Proc. of SC, 2007
39. W. Wang, K. Zhu, L. Ying, J. Tan, and L. Zhang. A throughput optimal algorithm for map task scheduling in mapreduce with data locality. SIGMETRICS Performance Evaluation Review, 40(4): 33-42, 2013
40. A. Bar-Noy, S. Guha, Y. Katz, J. Naor, B. Schieber, and H. Shachnai. Throughput maximization of real-time scheduling with batching. ACM Trans. Algorithms, 5(2): 1-17, 2009
41. J. Dean, and L. A. Barroso. The tail at scale. Communications of the ACM, 56(2): 74-80, February 2013
42. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair scheduling for distributed computing clusters. In Proc. of SOSP, 2009
43. D. Bertsekas, and R. Gallager. Data Networks. Prentice Hall, second edition edition, 1992
44. D. Raz, H. Levy, and B. Avi-Itzhak. A resource-allocation queueing fairness measure. In Proc. of SIGMETRICS, pages 130-141, June 2004
45. C. A. Waldspurger, and W. E. Weihl. Lottery scheduling: flexible proportional-share resource management. In Proc. of OSDI, 1994
46. A. Wierman, and M. Harchol-Balter. Classifying scheduling policies with respect to unfairness in an m/gi/1. In Proc. of SIGMETRICS, 2003
47. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, I. Stoica, and S. Shenker. Dominant resource fairness: Fair allocation of multiple resource types. In Proc. of NSDI, 2011
48. J. Liu, and H. Shen. A low-cost multi-failure resilient replication scheme for high data availability in cloud storage. In Proc. of HiPC, 2016