Using imbalance metrics to optimize task clustering in scientific workflow executions

Future Generation Computer Systems

Volumn 46, Issue , 2015, Pages 69-84

  Chen, Weiwei ^a   Ferreira Da Silva, Rafael ^a   Deelman, Ewa ^a   Sakellariou, Rizos ^b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Load balancing; Performance analysis; Scheduling; Scientific workflows; Task clustering; Workflow simulation

Indexed keywords

RESOURCE ALLOCATION; SOFTWARE ENGINEERING;

CLOUD INFRASTRUCTURES; COARSE-GRAINED APPROACHES; COMPUTATIONAL GRANULARITY; PERFORMANCE ANALYSIS; SCIENTIFIC WORKFLOWS; TASK CLUSTERING; TRACE-BASED SIMULATION; WORKFLOW SIMULATION;

SCHEDULING;

EID: 84923323706     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2014.09.014     Document Type: Article

References (48)

1. R. Ferreira da Silva, G. Juve, E. Deelman, T. Glatard, F. Desprez, D. Thain, B. Tovar, and M. Livny Toward fine-grained online task characteristics estimation in scientific workflows Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science WORKS'13 2013 ACM 58 67
2. G. Juve, A. Chervenak, E. Deelman, S. Bharathi, G. Mehta, K. Vahi, Characterizing and profiling scientific workflows, 29 (2013) 682-692. Special Section: Recent Developments in High Performance Computing and Security.
3. W. Chen, E. Deelman, Workflow overhead analysis and optimizations, in: Proceedings of the 6th Workshop on Workflows in Support of Large-scale Science, WORKS'11, 2011, pp. 11-20.
4. N. Muthuvelu, J. Liu, N.L. Soe, S. Venugopal, A. Sulistio, R. Buyya, A dynamic job grouping-based scheduling for deploying applications with fine-grained tasks on global grids, in: Proceedings of the 2005 Australasian Workshop on Grid Computing and e-Research, vol. 44, 2005, pp. 41-48.
5. N. Muthuvelu, I. Chai, C. Eswaran, An adaptive and parameterized job grouping algorithm for scheduling grid jobs, in: Advanced Communication Technology, 2008. ICACT 2008. 10th International Conference on, vol. 2, 2008, pp. 975-980.
6. N. Muthuvelu, I. Chai, E. Chikkannan, and R. Buyya On-line task granularity adaptation for dynamic grid applications Algorithms and Architectures for Parallel Processing Lecture Notes in Computer Science vol. 6081 2010 266 277
7. N. Muthuvelu, C. Vecchiolab, I. Chai, E. Chikkannana, and R. Buyya Task granularity policies for deploying bag-of-task applications on global grids Future Gener. Comput. Syst. 29 1 2012 170 181
8. W.K. Ng, T. Ang, T. Ling, and C. Liew Scheduling framework for bandwidth-aware job grouping-based scheduling in grid computing Malays. J. Comput. Sci. 19 2 2006 117 126
9. T. Ang, W. Ng, T. Ling, L. Por, and C. Lieu A bandwidth-aware job grouping-based scheduling on grid environment Inf. Technol. J. 8 2009 372 377
10. Q. Liu, Y. Liao, Grouping-based fine-grained job scheduling in grid computing, in: First International Workshop on Education Technology and Computer Science, vol. 1, 2009, pp. 556-559.
11. G. Singh, M.-H. Su, K. Vahi, E. Deelman, B. Berriman, J. Good, D.S. Katz, G. Mehta, Workflow task clustering for best effort systems with pegasus, in: 15th ACM Mardi Gras Conference, 2008, pp. 9:1-9:8.
12. R. Ferreira da Silva, T. Glatard, and F. Desprez On-line, non-clairvoyant optimization of workflow activity granularity on grids F. Wolf, B. Mohr, D. Mey, Euro-Par 2013 Parallel Processing Lecture Notes in Computer Science vol. 8097 2013 Springer Berlin, Heidelberg 255 266
13. W. Chen, E. Deelman, R. Sakellariou, Imbalance optimization in scientific workflows, in: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing, ICS'13, 2013, pp. 461-462.
14. J. Lifflander, S. Krishnamoorthy, L.V. Kale, Work stealing and persistence-based load balancers for iterative overdecomposed applications, in: Proceedings of the 21st International Symposium on High-Performance Parallel and Distributed Computing, HPDC'12, 2012, pp. 137-148.
15. W. Chen, E. Deelman, Integration of workflow partitioning and resource provisioning, in: Cluster, Cloud and Grid Computing, CCGrid, 2012 12th IEEE/ACM International Symposium on, 2012, pp. 764-768.
16. W. Chen, R. Ferreira da Silva, E. Deelman, R. Sakellariou, Balanced task clustering in scientific workflows, in: eScience (eScience), 2013 IEEE 9th International Conference on, 2013, pp. 188-195.
17. P.-O. Östberg, E. Elmroth, Mediation of service overhead in service-oriented grid architectures, in: Grid Computing, GRID, 2011 12th IEEE/ACM International Conference on, 2011, pp. 9-18.
18. R. Prodan, and T. Fabringer Overhead analysis of scientific workflows in grid environments IEEE Trans. Parallel Distrib. Syst. 19 2008
19. C. Stratan, A. Iosup, and D.H. Epema A performance study of grid workflow engines Grid Computing, 2008 9th IEEE/ACM International Conference on 2008 IEEE 25 32
20. R. Ferreira da Silva, T. Glatard, and F. Desprez Controlling fairness and task granularity in distributed, online, non-clairvoyant workflow executions Concurr. Comput.: Pract. Exper. 26 14 2014 2347 2366
21. G. Zheng, A. Bhatelé, E. Meneses, and L.V. Kalé Periodic hierarchical load balancing for large supercomputers Int. J. High Perform. Comput. Appl. 25 4 2011 371 385
22. L. Canon, E. Jeannot, R. Sakellariou, and W. Zheng Comparative evaluation of the robustness of DAG scheduling heuristics Grid Comput. 2008 73 84
23. J. Yu, and R. Buyya A taxonomy of workflow management systems for grid computing J. Grid Comput., 3 2005
24. X. Liu, J. Chen, K. Liu, Y. Yang, Forecasting duration intervals of scientific workflow activities based on time-series patterns, in: eScience, 2008. eScience'08. IEEE Fourth International Conference on, 2008, pp. 23-30.
25. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, S. Patil, M. Su, K. Vahi, M. Livny, Pegasus: Mapping scientific workflows onto the grid, in: Across Grid Conference, 2004.
26. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob, and D.S. Katz Pegasus: A framework for mapping complex scientific workflows onto distributed systems Sci. Program. 13 3 2005 219 237
27. E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P. Maechling, R. Mayani, W. Chen, R. Ferreira da Silva, M. Livny, and K. Wenger Pegasus: A workflow management system for science automation Future Gener. Comput. Syst. 2014 (submitted for publication)
28. E. Deelman Pegasus: A framework for mapping complex scientific workflows onto distributed systems Sci. Program. 13 2005 219 237 Deelman:2005:PFM:1239649.1239653
29. S. Ali, A. Maciejewski, H. Siegel, and J.-K. Kim Measuring the robustness of a resource allocation IEEE Trans. Parallel Distrib. Syst. 15 7 2004 630 641
30. R. Tolosana-Calasanz, M. Lackovic, O.F. Rana, J.Á. Bañares, and D. Talia Characterizing quality of resilience in scientific workflows Proceedings of the 6th Workshop on Workflows in Support of Large-scale Science 2011 ACM 117 126
31. Y. Ma, X. Zhang, and K. Lu A graph distance based metric for data oriented workflow retrieval with variable time constraints Expert Syst. Appl. 41 4 2014 1377 1388
32. T. Fahringer, A. Jugravu, S. Pllana, R. Prodan, C. Seragiotto Jr., and H.-L. Truong Askalon: A tool set for cluster and grid computing: Research articles Concurr. Comput.: Pract. Exper. 17 2-4 2005 143 169
33. T. Oinn, M. Greenwood, M. Addis, M.N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M.R. Pocock, M. Senger, R. Stevens, A. Wipat, and C. Wroe Taverna: Lessons in creating a workflow environment for the life sciences: Research articles Concurr. Comput.: Pract. Exper. 18 10 2006 1067 1100
34. M. Abouelhoda, S. Issa, and M. Ghanem Tavaxy: Integrating taverna and galaxy workflows with cloud computing support BMC Bioinformatics 13 2012 77
35. D. Brown, P. Brady, A. Dietz, J. Cao, B. Johnson, and J. McNabb A case study on the use of workflow technologies for scientific analysis: Gravitational wave data analysis I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows for e-Science 2007 Springer London 39 59
36. G.B. Berriman, E. Deelman, J.C. Good, J.C. Jacob, D.S. Katz, C. Kesselman, A.C. Laity, T.A. Prince, G. Singh, M. Su, Montage: A grid-enabled engine for delivering custom science-grade mosaics on demand, in: SPIE Conference on Astronomical Telescopes and Instrumentation, vol. 5493, 2004, pp. 221-232.
37. R. Graves, T. Jordan, S. Callaghan, E. Deelman, E. Field, G. Juve, C. Kesselman, P. Maechling, G. Mehta, K. Milner, D. Okaya, P. Small, and K. Vahi CyberShake: A physics-based seismic hazard model for Southern California Pure Appl. Geophys. 168 3-4 2011 367 381
38. USC Epigenome Center. http://epigenome.usc.edu.
39. SIPHT. http://pegasus.isi.edu/applications/sipht.
40. W. Chen, E. Deelman, Workflowsim: A toolkit for simulating scientific workflows in distributed environments, in: E-Science, e-Science, 2012 IEEE 8th International Conference on, 2012, pp. 1-8.
41. R.N. Calheiros, R. Ranjan, A. Beloglazov, C.A.F. De Rose, and R. Buyya CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms Softw. - Pract. Exp. 41 1 2011 23 50
42. W. Chen, E. Deelman, Fault tolerant clustering in scientific workflows, in: Services, SERVICES, 2012 IEEE Eighth World Congress on, 2012, pp. 9-16.
43. F. Jrad, J. Tao, and A. Streit A broker-based framework for multi-cloud workflows Proceedings of the 2013 International Workshop on Multi-cloud Applications and Federated Clouds 2013 ACM 61 68
44. Amazon.com, Inc., Amazon Web Services. http://aws.amazon.com.
45. FutureGrid. http://futuregrid.org/.
46. G. Juve, E. Deelman, K. Vahi, G. Mehta, B. Berriman, Scientific workflow applications on Amazon EC2, in: Cloud Computing Workshop in Conjunction with e-Science, IEEE, 2009.
47. R. Ferreira da Silva, W. Chen, G. Juve, K. Vahi, E. Deelman, Community resources for enabling and evaluating research on scientific workflows, in: 10th IEEE International Conference on e-Science, eScience'14, 2014 (to appear).
48. R. Ferreira da Silva, T. Glatard, and F. Desprez Self-healing of workflow activity incidents on distributed computing infrastructures Future Gener. Comput. Syst. 29 8 2013 2284 2294