Algorithms for cost-and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds

Future Generation Computer Systems

Volumn 48, Issue , 2015, Pages 1-18

  Malawski, Maciej ^a   Juve, Gideon ^b   Deelman, Ewa ^b   Nabrzyski, Jarek ^c  

^a AGH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Poland)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

Cloud computing; Resource provisioning; Scientific workflows; Workflow ensembles

Indexed keywords

BUDGET CONTROL; CLOUD COMPUTING;

DEADLINE CONSTRAINT; EFFICIENT MANAGEMENTS; LARGE-SCALE APPLICATIONS; ON-DEMAND RESOURCE PROVISIONING; QUALITY OF SOLUTION; RESOURCE PROVISIONING; SCIENTIFIC WORKFLOWS; WORKFLOW ENSEMBLES;

INFRASTRUCTURE AS A SERVICE (IAAS);

EID: 84923348674     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2015.01.004     Document Type: Article

References (55)

1. S. Callaghan, P. Maechling, P. Small, K. Milner, G. Juve, T. Jordan, E. Deelman, G. Mehta, K. Vahi, D. Gunter, K. Beattie, and C.X. Brooks Metrics for heterogeneous scientific workflows: a case study of an earthquake science application Int. J. High Perform. Comput. Appl. 25 3 2011 274 285
2. CyberShake Study 13.4. URL http://scec.usc.edu/scecpedia/CyberShake-Study-13.4.
3. E. Deelman, G. Singh, M. Livny, B. Berriman, J. Good, The cost of doing science on the cloud: the montage example, in: 2008 ACM/IEEE Conference on Supercomputing, SC 08, 2008.
4. Montage galactic plane. URL http://pegasus.isi.edu/applications/galactic-plane.
5. J. Vöckler, G. Juve, E. Deelman, M. Rynge, G.B. Berriman, Experiences using cloud computing for a scientific workflow application, in: 2nd Workshop on Scientific Cloud Computing, ScienceCloud'11, 2011.
6. S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema A performance analysis of EC2 cloud computing services for scientific computing O. Akan, Cloud Computing Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering vol. 34 2010 Springer Berlin, Heidelberg 115 131
7. K. Keahey, M. Tsugawa, A. Matsunaga, and J. Fortes Sky computing IEEE Internet Comput. 13 5 2009 43 51
8. D. Durkee Why cloud computing will never be free Commun. ACM 53 5 2010 62 69
9. R.N. Calheiros, R. Ranjan, A. Beloglazov, C.A.F. De Rose, and R. Buyya CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms Softw.-Pract. Exp. 41 1 2011 23 50
10. M. Malawski, G. Juve, E. Deelman, and J. Nabrzyski Cost-and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis SC'12 2012 IEEE Computer Society Press URL http://portal.acm.org/citation.cfm?id=2389026
11. Amazon auto scaling. URL http://aws.amazon.com/autoscaling.
12. RightScale. URL http://www.rightscale.com.
13. P. Marshall, K. Keahey, T. Freeman, Elastic site: using clouds to elastically extend site resources, in: 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2010, 2010.
14. H. Kim, Y. el-Khamra, I. Rodero, S. Jha, and M. Parashar Autonomic management of application workflows on hybrid computing infrastructure Sci. Program. 19 2011 75 89
15. J. Chen, C. Wang, B.B. Zhou, L. Sun, Y.C. Lee, and A.Y. Zomaya Tradeoffs between profit and customer satisfaction for service provisioning in the cloud Proceedings of the 20th International Symposium on High Performance Distributed Computing HPDC'11 2011 ACM New York, NY, USA 229 238
16. J. Yu, R. Buyya, C. Tham, Cost-based scheduling of scientific workflow application on utility grids, in: First International Conference on e-Science and Grid Computing, 2005.
17. S. Abrishami, M. Naghibzadeh, and D.H. Epema Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds Future Gener. Comput. Syst. 29 1 2013 158 169 10.1016/j.future.2012.05.004 URL http://www.sciencedirect.com/science/article/pii/S0167739X12001008
18. Y.C. Lee, and A.Y. Zomaya Stretch out and compact: workflow scheduling with resource abundance 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2013 IEEE 219 226 10.1109/ccgrid.2013.55
19. M. Mao, M. Humphrey, Auto-scaling to minimize cost and meet application deadlines in cloud workflows, in: 2011 ACM/IEEE Conference on Supercomputing, SC'11, 2011.
20. H. Arabnejad, and J. Barbosa List scheduling algorithm for heterogeneous systems by an optimistic cost table IEEE Trans. Parallel Distrib. Syst. 25 3 2014 682 694 10.1109/TPDS.2013.57
21. M. Wieczorek, A. Hoheisel, and R. Prodan Towards a general model of the multi-criteria workflow scheduling on the grid Future Gener. Comput. Syst. 25 3 2009 237 256
22. R. Prodan, and M. Wieczorek Bi-criteria scheduling of scientific grid workflows IEEE Trans. Autom. Sci. Engrg. 7 2 2010 364 376
23. J.J. Dongarra, E. Jeannot, E. Saule, Z. Shi, Bi-objective scheduling algorithms for optimizing makespan and reliability on heterogeneous systems, in: 19th Annual ACM Symposium on Parallel Algorithms and Architectures, SPAA'07, 2007.
24. J.J. Durillo, V. Nae, and R. Prodan Multi-objective workflow scheduling: an analysis of the energy efficiency and makespan tradeoff 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2013 IEEE 203 210 10.1109/ccgrid.2013.62
25. J.J. Durillo, H.M. Fard, R. Prodan, Moheft: a multi-objective list-based method for workflow scheduling, in: 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings, CloudCom 2012, Taipei, Taiwan, December 3-6, 2012, pp. 185-192.
26. R. Sakellariou, H. Zhao, E. Tsiakkouri, and M.D. Dikaiakos Scheduling workflows with budget constraints Integrated Research in GRID Computing COREGrid Series 2007 Springer-Verlag
27. W. Zheng, and R. Sakellariou Stochastic DAG scheduling using a Monte Carlo approach J. Parallel Distrib. Comput. 73 12 2013 1673 1689 10.1016/j.jpdc.2013.07.019 Heterogeneity in parallel and distributed computing. URL http://www.sciencedirect.com/science/article/pii/S0743731513001573
28. H. Zhao, and R. Sakellariou Scheduling multiple dags onto heterogeneous systems 20th International Parallel and Distributed Processing Symposium, 2006 IPDPS 2006 14 10.1109/IPDPS.2006.1639387
29. A. Hirales-Carbajal, A. Tchernykh, R. Yahyapour, J. González-García, T. Röblitz, and J. Ramírez-Alcaraz Multiple workflow scheduling strategies with user run time estimates on a grid J. Grid Comput. 10 2 2012 325 346 10.1007/s10723-012-9215-6
30. R. Duan, R. Prodan, and X. Li A sequential cooperative game theoretic approach to Storage-Aware scheduling of multiple Large-Scale workflow applications in grids 2012 ACM/IEEE 13th International Conference on Grid Computing (GRID) 2012 IEEE 31 39 10.1109/Grid.2012.14
31. R. Duan, R. Prodan, and X. Li Multi-objective game theoretic schedulingof bag-of-tasks workflows on hybrid clouds IEEE Trans. Cloud Comput. 2 1 2014 29 42 10.1109/TCC.2014.2303077
32. R. Tolosana-Calasanz, J.A. Banares, C. Pham, and O.F. Rana Enforcing qos in scientific workflow systems enacted over cloud infrastructures J. Comput. System Sci. 78 5 2012 1300 1315 10.1016/j.jcss.2011.12.015 URL http://www.sciencedirect.com/science/article/pii/S0022000011001607
33. W. Zheng, and R. Sakellariou Budget-deadline constrained workflow planning for admission control J. Grid Comput. 11 4 2013 633 651 10.1007/s10723-013-9257-4
34. A.K.M.K.A. Talukder, M. Kirley, and R. Buyya Multiobjective differential evolution for scheduling workflow applications on global grids Concurr. Comput. Pract. Exp. 21 13 2009 1742 1756
35. S. Pandey, L. Wu, S.M. Guru, R. Buyya, A particle swarm optimization-based heuristic for scheduling workflow applications in cloud computing environments, in: International Conference on Advanced Information Networking and Applications, 2010.
36. J. Durillo, and R. Prodan Multi-objective workflow scheduling in Amazon EC2 Clust. Comput. 17 2 2014 169 189 10.1007/s10586-013-0325-0
37. T.A.L. Genez, L.F. Bittencourt, and E.R.M. Madeira Using time discretization to schedule scientific workflows in multiple cloud providers 2013 IEEE Sixth International Conference on Cloud Computing 2013 IEEE 123 130 10.1109/CLOUD.2013.141 URL http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6676686
38. R. Van den Bossche, K. Vanmechelen, and J. Broeckhove Cost-optimal scheduling in hybrid iaas clouds for deadline constrained workloads 2010 IEEE 3rd International Conference on Cloud Computing 2010 IEEE 228 235 10.1109/CLOUD.2010.58 URL http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5557990
39. R.V. den Bossche, K. Vanmechelen, and J. Broeckhove Online cost-efficient scheduling of deadline-constrained workloads on hybrid clouds Future Gener. Comput. Syst. 29 4 2013 973 985 10.1016/j.future.2012.12.012 URL http://www.sciencedirect.com/science/article/pii/S0167739X12002324
40. L.F. Bittencourt, and E.R.M. Madeira Hcoc: a cost optimization algorithm for workflow scheduling in hybrid clouds J. Internet Serv. Appl. 2 3 2011 207 227
41. M. Malawski, K. Figiela, M. Bubak, E. Deelman, and J. Nabrzyski Cost optimization of execution of multi-level deadline-constrained scientific workflows on clouds Parallel Processing and Applied Mathematics-10th International Conference PPAM 2013, Warsaw, Poland, September 8-11 Lecture Notes in Computer Science vol. 8384 2014 Springer 251 260 Revised selected papers, Part I
42. I. Sfiligoi glideinWMS-a generic pilot-based workload management system J. Phys. Conf. Ser. 119 6 2008 062044 URL http://stacks.iop.org/1742-6596/119/i=6/a=062044
43. G. Juve, E. Deelman, K. Vahi, G. Mehta, B. Berriman, B.P. Berman, P. Maechling, Scientific workflow applications on amazon EC2, in: 2009 5th IEEE International Conference on E-Science Workshops, 2009.
44. S. Bharathi, A. Chervenak, E. Deelman, G. Mehta, M. Su, K. Vahi, Characterization of scientific workflows, in: 3rd Workshop on Workflows in Support of Large Scale Science, WORKS 08, 2008.
45. Workflow generator. URL https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator.
46. Cloud workflow simulator. URL https://github.com/malawski/cloudworkflowsimulator.
47. E. Deelman, G. Singh, M. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob, and D.S. Katz Pegasus: a framework for mapping complex scientific workflows onto distributed systems Sci. Program. 13 3 2005 219 237
48. R.F. da Silva, W. Chen, G. Juve, K. Vahi, E. Deelman, Community resources for enabling research in distributed scientific workflows, in: 10th IEEE International Conference on e-Science, eScience 2014, 2014.
49. M. Malawski, K. Figiela, J. Nabrzyski, Cost minimization for computational applications on hybrid cloud infrastructures, Future Gener. Comput. Syst. http://dx.doi.org/10.1016/j.future.2013.01.004. URL http://dx.doi.org/10.1016/j.future.2013.01.004.
50. W. Chen, and E. Deelman Fault tolerant clustering in scientific workflows Eighth IEEE World Congress on Services SERVICES 2012, Honolulu, HI, USA, June 24-29, 2012 2012 IEEE 9 16
51. G. Juve, E. Deelman, K. Vahi, G. Mehta, B.P. Berman, B. Berriman, P. Maechling, Data sharing options for scientific workflows on amazon EC2, in: 2010 ACM/IEEE conference on Supercomputing, SC10, 2010.
52. D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov Eucalyptus: A technical report on an elastic utility computing architecture linking your programs to useful systems, UCSB Computer Science Technical Report 2008-10 2008
53. G. Juve, E. Deelman, Automating application deployment in infrastructure clouds, in: 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011, 2011.
54. National Energy Research Scientific Computing Center, NERSC, Magellan. URL http://magellan.nersc.gov.
55. FutureGrid. URL http://futuregrid.org/.