A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments

Concurrency and Computation: Practice and Experience

Volumn 29, Issue 8, 2017, Pages

  Rodriguez, Maria Alejandra ^a   Buyya, Rajkumar ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

IaaS cloud, resource provisioning, scientific workflow, scheduling, survey, taxonomy

Indexed keywords

COSTS; DISTRIBUTED DATABASE SYSTEMS; SCHEDULING; SURVEYS; TAXONOMIES;

APPLICATION MODELING; DISTRIBUTED ENVIRONMENTS; DISTRIBUTED SYSTEMS; IAAS CLOUDS; PAY-PER-USE; SCHEDULING MODELS; SCIENTIFIC WORKFLOWS; WORK-FLOWS;

INFRASTRUCTURE AS A SERVICE (IAAS);

EID: 85007137336     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.4041     Document Type: Article

References (81)

1. Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I. Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Future Gener Comput Syst. 2009;25(6):599–616.
2. Elastic compute cloud EC2. Available from: http://aws.amazon.com/ec2. Accessed June 2016.
3. Microsoft Azure. Available from: https://azure.microsoft.com. Accessed June 2016.
4. Yildiz U, Guabtni A, Ngu AH. Business versus scientific workflows: a comparative study. Proceedings of the Fourth World Conference on Services (SERVICES). IEEE, Los Angeles, CA; 2009:340–343.
5. Barker A, Van Hemert J. Scientific workflow: a survey and research directions. Parallel Processing and Applied Mathematics. Springer, Gdansk, Poland; 2008:746–753.
6. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K. Characterizing and profiling scientific workflows. Future Gener Comput Syst. 2013;29(3):682–692.
7. Berriman GB, Laity AC, Good JC, Jacob JC, Katz DS, Deelman E, Singh G, Su MH, Prince TA. Montage: The architecture and scientific applications of a national virtual observatory service for computing astronomical image mosaics. Proceedings of Earth Sciences Technology Conference, College Park, Maryland; 2006.
8. Graves R, Jordan TH, Callaghan S, Deelman E, Field E, Juve G, Kesselman C, Maechling P, Mehta G, Milner K, et al. Cybershake: a physics-based seismic hazard model for southern california. Pure Appl Geophys. 2011;168(3-4):367–381.
9. Abramovici A, Althouse WE, Drever RWP, Gürsel Y, Kawamura S, Raab FJ, Shoemaker D, Sievers L, Spero RE, Thorne KS, et al. Ligo: the laser interferometer gravitational-wave observatory. Science. 1992;256(5055):325–333.
10. Livny J, Teonadi H, Livny M, Waldor M. K. High-throughput, kingdom-wide prediction and annotation of bacterial non-coding rnas. PloS one. 2008;3(9):e3197.
11. USC epigenome center. Available from: http://epigenome.usc.edu. Accessed June 2016.
12. National Centre for Biotechnology Information. Available from: http://www.ncbi.nlm.nih.gov. Accessed June 2016.
13. Bharathi S, Chervenak A, Deelman E, Mehta G, Su Mei-Hui, Vahi K. Characterization of scientific workflows. Proceedings of the Third Workshop on Workflows in Support of Large-Scale Science (WORKS), Austin, TX, USA; 2008:1–10.
14. Deelman E, Singh G, Su MH, Blythe J, Gil Y, Kesselman C, Mehta G, Vahi K, Berriman GB, Good J, et al. Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci Program. 2005;13(3):219–237.
15. Pandey S, Karunamoorthy D, Buyya R. Workflow engine for clouds. Cloud Computing: Principles and Paradigms, Hoboken, New Jersey, United States; 2011;87:321–344.
16. Fahringer T, Prodan R, Duan R, Nerieri F, Podlipnig S, Qin J, Siddiqui M, Truong HL, Villazon A, Wieczorek M. Askalon: a grid application development and computing environment. Proceedings of the Sixth IEEE/ACM International Workshop on Grid Computing. IEEE Computer Society, Seattle, Washington, USA; 2005:122–131.
17. Couvares P, Kosar T, Roy A, Weber J, Wenger K. Workflow management in condor. Workflows for e-Science. Springer; 2007:357–375.
18. Pautasso C, Alonso G. Parallel computing patterns for grid workflows. Workshop on Workflows in Support of Large-Scale Science, 2006. WORKS'06., IEEE, Paris; 2006:1–10.
19. Ullman JD. NP-complete scheduling problems. J Comput Syst Sci. 1975;10(3):384–393.
20. Hu TC. Parallel sequencing and assembly line problems. Oper Res. 1961;9(6):841–848.
21. Mayr EW, Stadtherr H. Optimal parallel algorithms for two processor scheduling with tree precedence constraints. 1995.
22. Fishburn P. C. Interval Orders and Interval Graphs: A Study of Partially Ordered Sets: John Wiley & Sons; 1985.
23. Yu J, Buyya R. A novel architecture for realizing grid workflow using tuple spaces. Proceedings of the Fifth IEEE/ACM International Workshop on Grid Computing. IEEE, Pittsburgh, USA; 2004:119–128.
24. Apache Jclouds. Available from: http://jclouds.apache.org. Accessed June 2016.
25. Gutierrez-Garcia JO, Sim KM. A family of heuristics for agent-based elastic cloud bag-of-tasks concurrent scheduling. Future Gener Comput Syst. 2013;29(7):1682–1699.
26. Michon E, Gossa J, Genaud S, et al. Free elasticity and free CPU power for scientific workloads on IaaS clouds. Proceedings of the Eighteen IEEE International Conference on Parallel and Distributed Systems (ICPADS). IEEE, Singapore; 2012:85–92.
27. Villegas D, Antoniou A, Sadjadi SM, Iosup A. An analysis of provisioning and allocation policies for infrastructure-as-a-service clouds. Proceedings of the Twelfth IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). IEEE, Ottawa, Canada; 2012:612–619.
28. Frincu ME, Genaud S, Gossa J, et al. Comparing provisioning and scheduling strategies for workflows on clouds. Proceedings of the Second International Workshop on Workflow Models, Systems, Services and Applications in the Cloud (CloudFlow). IEEE, Boston, MA; 2013.
29. Schad J, Dittrich J, Quiané-Ruiz JA. Runtime measurements in the cloud: observing, analyzing, and reducing variance. Proc of the VLDB Endowment. 2010;3(1-2):460–471.
30. Ostermann S, Iosup A, Yigitbasi N, Prodan R, Fahringer T, Epema D. A performance analysis of EC2 cloud computing services for scientific computing. Cloud Computing. Springer, Munich, Germany; 2010:115–131.
31. Gupta A, Milojicic D. Evaluation of HPC applications on cloud. Open Cirrus Summit (OCS), 2011 Sixth, Atlanta, Georgia; 2011:22–26.
32. Iosup A, Ostermann S, Yigitbasi MN, Prodan R, Fahringer T, Epema D. Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans Parallel Distrib Syst. 2011;22(6):931–945.
33. Jackson KR, Ramakrishnan L, Muriki K, Canon S, Cholia S, Shalf J, Wasserman HJ, Wright NJ. Performance analysis of high performance computing applications on the amazon web services cloud. Proceedings of the IEEE Second International Conference on Cloud Computing Technology and Science (CLOUDCOM). IEEE, Athens, Greece; 2010:159–168.
34. Nabrzyski J, Schopf JM, Weglarz J. Grid Resource Management: State of the Art and Future Trends, vol. 64: Berlin, Germany: Springer Science & Business Media; 2012.
35. Valentin C, Ciprian D, Corina S, Florin P, Alexandru C. Large-scale distributed computing and applications: models and trends. 2010.
36. Berman F, Fox G, Hey AnthonyJG. Grid Computing: Making the Global Infrastructure a Reality, vol. 2: Hoboken, New Jersey, United States: John Wiley and sons; 2003.
37. Mao M, Humphrey M. A performance study on the VM startup time in the cloud. Proceedings of the Fifth IEEE International Conference on Cloud Computing (CLOUD), IEEE, Honolulu, Hawaii, USA; 2012:423–430.
38. Amazon EC2 Spot Instances. Available from: https://aws.amazon.com/ec2/purchasing-options/spot-instances/. Accessed June 2016.
39. Maechling P, Deelman E, Zhao L, Graves R, Mehta G, Gupta N, Mehringer J, Kesselman C, Callaghan S, Okaya D, et al. SCEC CyberShake Workflows—Automating Probabilistic Seismic Hazard Analysis Calculations. Workflows for e-Science. Springer; 2007:143–163.
40. Deelman E, Singh G, Livny M, Berriman B, Good J. The cost of doing science on the cloud: the montage example. Proceedings of the ACM/IEEE Conference on Supercomputing (SC). IEEE Press, Austin, Texas; 2008:Article No. 50.
41. Vöckler JS, Juve G, Deelman E, Rynge M, Berriman B. Experiences using cloud computing for a scientific workflow application. Proceedings of the Second International Workshop on Scientific Cloud Computing (ScienceCloud).ACM, San Jose, California; 2011:15–24.
42. Malawski M, Juve G, Deelman E, Nabrzyski J. Cost-and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC). EEE, Salt Lake City, Utah; 2012:Article No. 22.
43. Tolosana-Calasanz R, BañAres JÁ, Pham C, Rana OF. Enforcing qos in scientific workflow systems enacted over cloud infrastructures. J Comput Syst Sci. 2012;78(5):1300–1315.
44. Casavant TL, Kuhl JG. A taxonomy of scheduling in general-purpose distributed computing systems. IEEE Trans Softw Eng. 1988;14(2):141–154.
45. Kwok YK, Ahmad I. Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Comput Surv (CSUR). 1999;31(4):406–471.
46. Yu J, Buyya R, Ramamohanarao K. Workflow scheduling algorithms for grid computing. Metaheuristics for Scheduling in Distributed Computing Environments. Springer; 2008:173–214.
47. Zeng L, Veeravalli B, Li X. Saba: a security-aware and budget-aware workflow scheduling strategy in clouds. J Parallel Distrib Comput. 2015;75:141–151.
48. Genez TA, Bittencourt LF, Madeira ER. Workflow scheduling for SaaS/PaaS cloud providers considering two sla levels. Proceedings of the IEEE Network Operations and Management Symposium (NOMS). IEEE, Maui, Hawaii, USA; 2012:906–912.
49. Rodriguez MA, Buyya R. A responsive knapsack-based algorithm for resource provisioning and scheduling of scientific workflows in clouds. Proceedings of the Fourty-Fourth International Conference on Parallel Processing (ICPP), vol. 1. IEEE; 2015:839–848.
50. Malawski M, Figiela K, Bubak M, Deelman E, Nabrzyski J. Scheduling multilevel deadline-constrained scientific workflows on clouds based on cost optimization. Sci Program. 2015;2015.
51. Talbi EG. Metaheuristics: From Design to Implementation, vol. 74: Hoboken, New Jersey, United States: John Wiley & Sons; 2009.
52. Amazon Simple Storage Service. Available from: http://aws.amazon.com/s3/. Accessed June 2016.
53. Amazon EFS. Available from: https://aws.amazon.com/efs/. Accessed June 2016. Accessed June 2016.
54. Google Cloud Storage. Available from: https://cloud.google.com/storage/. Accessed June 2016.
55. Rackspace Block Storage. Availeble from: http://www.rackspace.com.au/cloud/block-storage. Accessed June 2016.
56. Zhou A, He B, Liu C. Monetary cost optimizations for hosting workflow-as-a-service in iaas clouds. IEEE Trans Cloud Comput. 2015;PP(99):1–1.
57. Poola D, Ramamohanarao K, Buyya R. Fault-tolerant workflow scheduling using spot instances on clouds. Procedia Comput Sci. 2014;29:523–533.
58. Google Compute Engine. Available from: https://cloud.google.com/products/compute-engine/. Accessed June 2016.
59. Google Compute Engine Pricing. Available from: https://developers.google.com/compute/pricing. Accessed June 2016.
60. Cloudsigma. Available from: https://www.cloudsigma.com. Accessed June 2016.
61. Arabnejad H, Barbosa JG, Prodan R. Low-time complexity budget–deadline constrained workflow scheduling on heterogeneous resources. Future Gener Comput Syst. 2016;55:29–40.
62. Zhu Q, Zhu J, Agrawal G. Power-aware consolidation of scientific workflows in virtualized environments. Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, New Orleans, Louisiana, USA; 2010:1–12.
63. Rodriguez MA, Buyya R. Deadline based resource provisioning and scheduling algorithm for scientific workflows on clouds. IEEE Trans Cloud Comput. 2014April;2(2):222–235.
64. Durillo JJ, Prodan R. Multi-objective workflow scheduling in amazon ec2. Cluster Comput. 2014;17(2):169–189.
65. Topcuoglu H, Hariri S, Wu Min-you. Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst. 2002;13(3):260–274.
66. Abrishami S, Naghibzadeh M, Epema DickHJ. Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Gener Comput Syst. 2013;29(1):158–169.
67. Calheiros RN, Buyya R. Meeting deadlines of scientific workflows in public clouds with tasks replication. IEEE Trans Parallel Distrib Syst. 2014;25(7):1787–1796.
68. Byun EK, Kee YS, Kim JS, Maeng S. Cost optimized provisioning of elastic resources for application workflows. Future Gener Comput Syst. 2011;27(8):1011–1026.
69. Byun EK, Kee YS, Kim JS, Deelman E, Maeng S. Bts: Resource capacity estimate for time-targeted science workflows. J Parallel Distrib Comput. 2011;71(6):848–862.
70. Pietri I, Malawski M, Juve G, Deelman E, Nabrzyski J, Sakellariou R. Energy-constrained provisioning for scientific workflow ensembles. Proceedings of The Third International Conference on Cloud and Green Computing (CGC), IEEE, Karlsruhe, Germany; 2013:34–41.
71. Mao M, Humphrey M. Auto-scaling to minimize cost and meet application deadlines in cloud workflows. Proceedings of The International Conference for High Performance Computing, Networking, Storage and Analysis (SC), ACM, Seattle, WA, USA; 2011:Article No. 49.
72. Li HH, Fu YW, Zhan ZH, Li JJ. Renumber strategy enhanced particle swarm optimization for cloud computing resource scheduling. Proceedings of the IEEE Congress on Evolutionary Computation (CEC). IEEE; 2015:870–876.
73. Poola D, Garg SK, Buyya R, Yang Y, Ramamohanarao K. Robust scheduling of scientific workflows with deadline and budget constraints in clouds. Proceedings of the Twenty-Eighth IEEE International Conference on Advanced Information Networking and Applications (AINA), Victoria, Canada; 2014:1–8.
74. Lee YC, Zomaya AY. Stretch out and compact: workflow scheduling with resource abundance. Proceedings of the Thirteenth IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). IEEE, Delft, Netherlands; 2013:219–226.
75. de Oliveira D, Ocaña KA, Baião F, Mattoso M. A provenance-based adaptive scheduling heuristic for parallel scientific workflows in clouds. J Grid Comput. 2012;10(3):521–552.
76. Zhu Z, Zhang G, Li M, Liu X. Evolutionary multi-objective workflow scheduling in cloud. IEEE Transactions on Parallel and Distributed Systems. 2016;27(5):1344–1357.
77. Wang J, Korambath P, Altintas I, Davis J, Crawl D. Workflow as a service in the cloud: architecture and scheduling algorithms. Procedia Comput Sci. 2014;29:546–556.
78. Zhou AC, He B. Transformation-based monetary cost optimizations for workflows in the cloud. IEEE Trans Cloud Comput. 2014:1–1.
79. Lee YC, Han H, Zomaya AY, Yousif M. Resource-efficient workflow scheduling in clouds. Knowl-Based Syst. 2015;80:153–162.
80. Altintas I, Berkley C, Jaeger E, Jones M, Ludascher B, Mock S. Kepler: an extensible system for design and execution of scientific workflows. 16th International Conference on Scientific and Statistical Database Management, 2004. Proceedings. IEEE; 2004:423–424.
81. Esteves S, Veiga L. Waas: workflow-as-a-service for the cloud with scheduling of continuous and data-intensive workflows. Comput J. 2016;59(3):371–383.