Scheduling stochastic multi-stage jobs to elastic hybrid cloud resources

IEEE Transactions on Parallel and Distributed Systems

Volumn 29, Issue 6, 2018, Pages 1401-1415

  Zhu, Jie ^a   Li, Xiaoping ^b   Ruiz, Ruben ^c   Xu, Xiaolong ^a  

^a NANJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS   (China)

^b Ministry of Education   (China)

^c INSTITUTO TECNOLÓGICO DE INFORMÁTICA   (Spain)

Author keywords

cloud computing; deadline constraint; elastic; linearly dependent tasks; Multi stage job scheduling; stochastic

Indexed keywords

CALIBRATION; CLOUD COMPUTING; GENETIC ALGORITHMS; HEURISTIC ALGORITHMS; JOB SHOP SCHEDULING; OPTIMIZATION; STOCHASTIC SYSTEMS; WORK SIMPLIFICATION;

DEADLINE CONSTRAINT; DEPENDENT TASKS; DYNAMIC SCHEDULING; ELASTIC; JOB SCHEDULING; PROCESSOR SCHEDULING; STOCHASTIC;

SCHEDULING;

EID: 85041230973     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2018.2793254     Document Type: Article

References (65)

1. X. Zhang, L. T. Yang, C. Liu, and J. Chen, "A scalable two-phase top-down specialization approach for data anonymization using MapReduce on cloud," IEEE Trans. Parallel Distrib. Syst., vol. 25, no. 2, pp. 363-373, Feb. 2014.
2. X. Li, T. Jiang, and R. Ruiz, "Heuristics for periodical batch job scheduling in a MapReduce computing framework," Inf. Sci., vol. 326, no. C, pp. 119-133, 2016.
3. T. Y. Wu, C. Y. Chen, L. S. Kuo, W. T. Lee, and H. C. Chao, "Cloud-based image processing system with priority-based data distribution mechanism," Comput. Commun., vol. 35, no. 15, pp. 1809-1818, 2012.
4. M. D. D. Assuncao, C. H. Cardonha, M. A. S. Netto, and R. L. F. Cunha, "Impact of user patience on auto-scaling resource capacity for cloud services," Future Generation Comput. Syst., vol. 55, pp. 41-50, 2015.
5. M. Masdari, S. Valikardan, Z. Shahi, and S. I. Azar, "Towards workflow scheduling in cloud computing: A comprehensive analysis," J. Netw. Comput. Appl., vol. 66, pp. 64-82, 2016.
6. W. N. Chen and J. Zhang, "An ant colony optimization approach to a grid workflow scheduling problem with various QoS requirements," IEEE Trans. Syst. Man Cybern. Part C, vol. 39, no. 1, pp. 29-43, Jan. 2009.
7. K. Bochenina, N. Butakov, and A. Boukhanovsky, "Static scheduling of multiple workflows with soft deadlines in non-dedicated heterogeneous environments," Future Generation Comput. Syst., vol. 55, pp. 51-61, 2016.
8. M. Malawski, G. Juve, E. Deelman, and J. Nabrzyski, "Algorithms for cost-and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds," Future Generation Comput. Syst., vol. 48, pp. 1-18, 2015.
9. M. Mao and M. Humphrey, "Auto-scaling to minimize cost and meet application deadlines in cloud workflows," in Proc. Int. Conf. High Perform. Comput. Netw. Storage Anal., 2011, pp. 1-12.
10. D. Yun, C. Q. Wu, and Y. Gu, "An integrated approach to workflow mapping and task scheduling for delay minimization in distributed environments," J. Parallel Distrib. Comput., vol. 84, pp. 51-64, 2015.
11. E. N. Alkhanak, S. P. Lee, R. Rezaei, and R. M. Parizi, "Cost optimization approaches for scientific workflow scheduling in cloud and grid computing: A review, classifications, and open issues," J. Syst. Softw., vol. 113, pp. 1-26, 2016.
12. X. Wang, C. S. Yeo, R. Buyya, and J. Su, "Optimizing the makespan and reliability for workflow applications with reputation and a look-ahead genetic algorithm," Future Generation Comput. Syst., vol. 27, no. 8, pp. 1124-1134, 2011.
13. J. J. Durillo, V. Nae, and R. Prodan, "Multi-objective workflow scheduling: An analysis of the energy efficiency and makespan tradeoff," in Proc. IEEE/ACM Int. Symp. Cluster Cloud Grid Comput., 2013, pp. 203-210.
14. A. Berl, et al., "Energy-efficient cloud computing," Comput. J., vol. 53, no. 7, pp. 1045-1051, 2010.
15. E. Gelenbe and S. Timotheou, "Random neural networks with synchronized interactions," Neural Comput., vol. 20, no. 9, pp. 2308-2324, 2008.
16. E. Gelenbe, "Sensible decisions based on QoS," Comput. Manage. Sci., vol. 1, no. 1, pp. 1-14, 2003.
17. L. Wang and E. Gelenbe, "Adaptive dispatching of tasks in the cloud," IEEE Trans. Cloud Comput., 2015, doi: 10.1109/TCC.2015.2474406.
18. Z. Wang, M. M. Hayat, N. Ghani, and K. B. Shanban, "Optimizing cloud-service performance: Efficient resource provisioning via optimal workload allocation," IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 6, pp. 1689-1702, Jun. 2017.
19. I. Al-Azzoni and D. G. Down, "Linear programming-based affinity scheduling of independent tasks on heterogeneous computing systems," IEEE Trans. Parallel Distrib. Syst., vol. 19, no. 12, pp. 1671-1682, Dec. 2008.
20. X. Zuo, G. Zhang, and W. Tan, "Self-adaptive learning PSO-based deadline constrained task scheduling for hybrid IaaS cloud," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 2, pp. 564-573, Apr. 2014.
21. Y. Xu, K. Li, T. T. Khac, and M. Qiu, "A multiple priority queueing genetic algorithm for task scheduling on heterogeneous computing systems," Inf. Sci., vol. 270, no. 4, pp. 639-646, 2012.
22. C. Delimitrou and C. Kozyrakis, "QoS-aware scheduling in heterogeneous datacenters with paragon," ACM Trans. Comput. Syst., vol. 31, no. 4, pp. 879-889, 2013.
23. S. Kanev, et al., "Profiling a warehouse-scale computer," in Proc. 42nd Annu. Int. Symp. Comput. Archit., 2015, pp. 158-169.
24. M. Kozlovszky, K. Karoczkai, I. Marton, A. Balasko, A. Marosi, and P. Kacsuk, "Enabling generic distributed computing infrastructure compatibility for workflow management systems," Comput. Sci., vol. 13, no. 3, pp. 61-78, 2012.
25. F. Jrad, J. Tao, and A. Streit, "A broker-based framework for multi-cloud workflows," in Proc. Int. Workshop Multi-Cloud Appl. Federated Clouds, 2013, pp. 61-68.
26. J. Wang, P. Korambath, I. Altintas, J. Davis, and D. Crawl, "Workflow as a service in the cloud: Architecture and scheduling algorithms," Procedia Comput. Sci., vol. 29, pp. 546-556, 2014.
27. G. Papuzzo and G. Spezzano, "Autonomic management of workflows on hybrid grid-cloud infrastructure," in Proc. Int. Conf. Netw. Serv. Manage., 2011, pp. 1-4.
28. Q. Tao, H. Y. Chang, Y. Yi, C. Q. Gu, and W. J. Li, "A rotary chaotic PSO algorithm for trustworthy scheduling of a grid workflow," Comput. Operations Res., vol. 38, no. 5, pp. 824-836, 2011.
29. R. K. Jena, "Multi objective task scheduling in cloud environment using nested PSO framework," Procedia Comput. Sci., vol. 57, pp. 1219-1227, 2015.
30. H. Liu, A. Abraham, V. Snasel, and S. Mcloone, "Swarm scheduling approaches for work-flow applications with security constraints in distributed data-intensive computing environments," Inf. Sci., vol. 192, no. 6, pp. 228-243, 2013.
31. L. Cui, et al., "A genetic algorithm based data replica placement strategy for scientific applications in clouds," IEEE Trans. Serv. Comput., 2015, doi: 10.1109/TSC.2015.2481421.
32. S. G. Ahmad, C. S. Liew, E. U. Munir, F. A. Tan, and S. U. Khan, "A hybrid genetic algorithm for optimization of scheduling workflow applications in heterogeneous computing systems," J. Parallel Distrib. Comput., vol. 87, no. C, pp. 80-90, 2016.
33. P. Lopez-Garcia , E. Onieva, E. Osaba, A. D. Masegosa, and A. Perallos, "GACE: A meta-heuristic based in the hybridization of genetic algorithms and cross entropy methods for continuous optimization," Expert Syst. Appl., vol. 55, pp. 508-519, 2016.
34. M. H. Eawna, S. H. Mohammed, and E. S. M. El-Horbaty, "Hybrid algorithmfor resource provisioning ofmulti-tier cloud computing," Procedia Comput. Sci., vol. 65, pp. 682-690, 2015.
35. S. Kianpisheh, N. M. Charkari, and M. Kargahi, "Reliabilitydriven scheduling of time/cost-constrained grid workflows," Future Generation Comput. Syst., vol. 55, pp. 1-16, 2015.
36. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica, "Sparrow: Distributed, low latency scheduling," in Proc. 24th ACM Symp. Operating Syst. Principles, 2013, pp. 69-84.
37. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes, "Omega: Flexible, scalable schedulers for large compute clusters," in Proc. ACM Eur. Conf. Comput. Syst., 2013, pp. 351-364.
38. I. R. A. Rajendran, "Matrix: Many-task computing execution fabric for extreme scales," MS Thesis, Dept. Comput. Sci., Illinois Inst. Technol., Chicago, IL, USA, 2013.
39. I. Sadooghi, et al., "Achieving efficient distributed scheduling with message queues in the cloud for many-task computing and high-performance computing," in Proc. IEEE/ACM Int. Symp. Cluster Cloud Grid Comput., 2014, pp. 404-413.
40. L. Wang and E. Gelenbe, "Experiments with smart workload allocation to cloud servers," in Proc. IEEE Symp. Netw. Cloud Comput. Appl., 2015, pp. 31-35.
41. E. Gelenbe and L. Wang, "TAP: A task allocation platform for the EU FP7 PANACEA project," in Proc. Eur. Conf. Service-Oriented Cloud Comput., 2016, Art. no. 425.
42. L. Wang, O. Brun and E. Gelenbe, "Adaptive workload distribution for local and remote Clouds," in Proc. IEEE Int. Conf. Syst. Man Cybern., 2017, pp. 3984-3988.
43. R. Bertin, S. Hunold, A. Legrand, and C. Touati, "Fair scheduling of bag-of-tasks applications using distributed lagrangian optimization," J. ParallelDistrib. Comput., vol. 74, no. 1, pp. 1914-1929, 2014.
44. A. Benoit, L. Marchal, J. F. Pineau, Y. Robert, and F. Vivien, "Scheduling concurrent bag-of-tasks applications on heterogeneous platforms," IEEE Trans. Comput., vol. 59, no. 2, pp. 202-217, Feb. 2010.
45. H. Topcuouglu, S. Hariri, and M. Y.Wu, "Performance-effective and low-complexity task scheduling for heterogeneous computing," IEEE Trans. Parallel Distrib. Syst., vol. 13, no. 3, pp. 260-274, Mar. 2002.
46. M. D. De Assuncao, A. Di Costanzo, and R. Buyya, "Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters," in Proc. 18th ACM Int. Symp. High Perform. Distrib. Comput., 2009, pp. 141-150.
47. C.-T. Lu, C.-W. Chang, and J.-S. Li, "VM scaling based on hurst exponent and Markov transition with empirical cloud data," J. Syst. Softw., vol. 99, pp. 199-207, 2015.
48. L. F. Bittencourt and E. R. M. Madeira, "HCOC: A cost optimization algorithm for workflow scheduling in hybrid clouds," J. Internet Serv. Appl., vol. 2, no. 3, pp. 207-227, 2011.
49. J. Liu, E. Pacitti, P. Valduriez, D. DeOliveira, and M. Mattoso,"Multiobjective scheduling of scientific workflows in multisite clouds," Future Generation Comput. Syst., vol. 63, no. C, pp. 76-95, 2016.
50. H. M. Fard, R. Prodan, and T. Fahringer, "A truthful dynamic workflow scheduling mechanism for commercial multicloud environments," IEEE Trans. Parallel Distrib. Syst., vol. 24, no. 6, pp. 1203-1212, Jun. 2013.
51. E. Gelenbe and R. Lent, "Energy-QoS trade-offs in mobile service selection," Future Internet, vol. 5, no. 2, pp. 128-139, 2013.
52. R. Duan, R. Prodan, and X. Li, "Multi-objective game theoretic scheduling of bag-of-tasks workflows on hybrid clouds," IEEE Trans. Cloud Comput., vol. 2, no. 1, pp. 29-42, Jan.-Mar. 2014.
53. J. Zhu and X. Li, "Scheduling for multi-stage applications with scalable virtual resources in cloud computing," Int. J. Mach. Learn. Cybern., vol. 8, no. 5, pp. 1633-1641, 2017.
54. J. Zhu and X. Li, "Elastic and flexible multi-stage task scheduling with deadline-constraint in clouds," in Proc. 20th IEEE Int. Conf. Comput. Supported Cooperative Work Des., 2016, pp. 286-291.
55. J. Zhu, X. Li, R. Ruiz, X. Xu, and Y. Zhang, "Scheduling stochastic multi-stage jobs on elastic computing services in hybrid clouds," in Proc. IEEE Int. Conf. Web Serv., 2016, pp. 678-681.
56. TOSCA Specification. (2016). [Online]. Available: Http://docs. oasis-open.org/tosca/TOSCA/v1.0/os/TOSCA-v1.0-os.html
57. G. E. Vieira, J. W. Herrmann, and E. Lin, "Rescheduling manufacturing systems: A framework of strategies, policies, and methods," J. Scheduling, vol. 6, no. 1, pp. 39-62, 2003.
58. X. Liu, et al., "A novel statistical time-series pattern based interval forecasting strategy for activity durations in workflow systems," J. Syst. Softw., vol. 84, no. 3, pp. 354-376, 2011.
59. D. Ouelhadj and S. Petrovic, "A survey of dynamic scheduling in manufacturing systems," J. Scheduling, vol. 12, no. 4, pp. 417-431, 2009.
60. H. M. Goldberg, "Analysis of the earliest due date scheduling rule in queueing systems," Math. Operations Res., vol. 2, no. 2, pp. 145-154, 1977.
61. E. Figielska, "A heuristic for scheduling in a two-stage hybrid flowshop with renewable resources shared among the stages," Eur. J. Oper. Res., vol. 236, no. 2, pp. 433-444, 2014.
62. B. Naderi, S. Gohari, and M. Yazdani, "Hybrid flexible flowshop problems: Models and solution methods," Appl. Math. Modelling, vol. 38, no. 24, pp. 5767-5780, 2014.
63. J. Brimberg, P. Hansen, N. Mlandinovic, and E. D. Taillard, "Improvement and comparison of heuristics for solving the uncapacitated multisource weber problem," Operations Res., vol. 48, no. 3, pp. 444-460, 2000.
64. J. Zhu, X. Li, and Q. Wang, "Complete local search with limited memory algorithm for no-wait job shops to minimize makespan," Eur. J. Oper. Res., vol. 198, no. 2, pp. 378-386, 2009.
65. E. Taillard, "Benchmarks for basic scheduling problems," Eur. J. Oper. Res., vol. 64, no. 2, pp. 278-285, 1993.