A characterization of workflow management systems for extreme-scale applications

Future Generation Computer Systems

Volumn 75, Issue , 2017, Pages 228-238

  Ferreira da Silva, Rafael ^a   Filgueira, Rosa ^b,c   Pietri, Ilia ^d   Jiang, Ming ^e   Sakellariou, Rizos ^f   Deelman, Ewa ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b THE LYELL CENTRE   (United Kingdom)

^c UNIVERSITY OF EDINBURGH   (United Kingdom)

^d UNIVERSITY OF ATHENS   (Greece)

^e LAWRENCE LIVERMORE NATIONAL LABORATORY   (United States)

^f UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Extreme scale computing; in situ processing; Scientific workflows; Workflow management systems

Indexed keywords

DATA HANDLING; DISTRIBUTED COMPUTER SYSTEMS; IN SITU PROCESSING; SURVEYS; WORK SIMPLIFICATION;

COMPUTATIONAL INFRASTRUCTURE; COMPUTING APPLICATIONS; EXTREME SCALE; HETEROGENEOUS COMPUTING; HIGH PERFORMANCE COMPUTING; SCIENTIFIC APPLICATIONS; SCIENTIFIC WORKFLOWS; WORKFLOW MANAGEMENT SYSTEMS;

INFORMATION MANAGEMENT;

EID: 85014024259     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2017.02.026     Document Type: Article

References (85)

1. Taylor, I.J., Deelman, E., Gannon, D.B., Shields, M., Workflows for E-Science: Scientific Workflows for Grids, 2007, Springer Publishing Company Incorporated.
2. Pegasus applications, https://pegasus.isi.edu/application-showcase/.
3. Deelman, E., Vahi, K., Juve, G., Rynge, M., Callaghan, S., Maechling, P.J., Mayani, R., Chen, W., Ferreira da Silva, R., Livny, M., Wenger, K., Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst. 46 (2015), 17–35, 10.1016/j.future.2014.10.008.
4. Filgueira, R., Krause, A., Atkinson, M., Klampanos, I., Moreno, A., dispel4py: A python framework for data-intensive scientific computing. Int. J. High Perform. Comput. Appl., 2016, 1–9, 10.1177/1094342016649766.
5. M. Albrecht, P. Donnelly, P. Bui, D. Thain, Makeflow: A portable abstraction for data intensive computing on clusters, clouds, and grids, in: Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies, ACM, 2012, p. 1. http://dx.doi.org/10.1145/2443416.2443417.
6. Jain, A., Ong, S.P., Chen, W., Medasani, B., Qu, X., Kocher, M., Brafman, M., Petretto, G., Rignanese, G.-M., Hautier, G., et al. Fireworks: a dynamic workflow system designed for high-throughput applications. Concurr. Comput. Pract. Exp. 27:17 (2015), 5037–5059, 10.1002/cpe.3505.
7. Fahringer, T., Prodan, R., Duan, R., Hofer, J., Nadeem, F., Nerieri, F., Podlipnig, S., Qin, J., Siddiqui, M., Truong, H.-L., et al. Askalon: A development and grid computing environment for scientific workflows. Workflows for E-Science, 2007, Springer, 450–471, 10.1007/978-1-84628-757-2_27.
8. K. Wolstencroft, R. Haines, D. Fellows, A. Williams, D. Withers, S. Owen, S. Soiland-Reyes, I. Dunlop, A. Nenadic, P. Fisher, et al., The taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud, Nucleic Acids Res. (2013) W557–W561. http://dx.doi.org/10.1093/nar/gkt328.
9. D. Blankenberg, G.V. Kuster, N. Coraor, G. Ananda, R. Lazarus, M. Mangan, A. Nekrutenko, J. Taylor, Galaxy: a web-based genome analysis tool for experimentalists, Curr. Protoc. Mol. Biol. (2010) 1–19. http://dx.doi.org/10.1002/0471142727.mb1910s89.
10. J. Frey, Condor Dagman: Handling Inter-Job Dependencies, University of Wisconsin, Dept. of Computer Science, Tech. Rep.
11. Liew, C.S., Atkinson, M.P., Galea, M., Ang, T.F., Martin, P., Hemert, J.I.V., Scientific workflows: Moving across paradigms. ACM Comput. Surv. (CSUR), 49(4), 2016, 66, 10.1145/3012429.
12. Adhianto, L., Banerjee, S., Fagan, M., Krentel, M., Marin, G., Mellor-Crummey, J., Tallent, N.R., HPCToolkit: Tools for performance analysis of optimized parallel programs. Concurr. Comput. Pract. Exp. 22:6 (2010), 685–701.
13. Bahsi, E.M., Ceyhan, E., Kosar, T., Conditional workflow management: A survey and analysis. Sci. Program. 15:4 (2007), 283–297.
14. M. Bux, U. Leser, Parallelization in scientific workflow management systems, arXiv preprint arXiv:1303.7195.
15. Liu, J., Pacitti, E., Valduriez, P., Mattoso, M., A survey of data-intensive scientific workflow management. J. Grid Comput. 13:4 (2015), 457–493.
16. Deelman, E., Gannon, D., Shields, M., Taylor, I., Workflows and e-science: An overview of workflow system features and capabilities. Future Gener. Comput. Syst. 25:5 (2009), 528–540.
17. Yu, J., Buyya, R., A taxonomy of workflow management systems for grid computing. J. Grid Comput. 3:3–4 (2005), 171–200.
18. Barker, A., Van Hemert, J., Scientific workflow: a survey and research directions. Parallel Processing and Applied Mathematics, 2007, Springer, 746–753.
19. The Opportunities and Challenges of Exascale Computing, ASCAC Subcommittee Report, 2010. http://science.energy.gov/~/media/ascr/ascac/pdf/reports/Exascale_subcommittee_report.pdf.
20. Dongarra, J., With extreme scale computing the rules have changed. International Congress on Mathematical Software, 2016, Springer, 3–6 http://dx.doi.org/10.1007/978-3-319-42432-3_1.
21. Report on the ASCR Workshop on Architectures I: Exascale and Beyond: Gaps in Research, Gaps in our Thinking, 2011. http://science.energy.gov/~/media/ascr/pdf/program-documents/docs/ArchitecturesIWorkshopReport.pdf.
22. Report out from the Exascale Research Planning Workshop Working Session on Data Management, Visualization, IO and Storage, 2012. http://exascaleresearch.labworks.org/apr2012planningworkshop/application/layouts/exascale-planning-workshop//public/docs/PRES_WorkingSession-DataIO_120420.pdf.
23. Scientific Discovery at the Exascale: Report from the DOE ASCR 2011 Work shop on Exascale Data Management, Analysis and Visualization, 2011. http://science.energy.gov/~/media/ascr/pdf/program-documents/docs/Exascale-ASCR-Analysis.pdf.
24. Ma, K.-L., In-situ visualization at extreme scale: Challenges and opportunities. IEEE Comput. Graph. Appl. 29:6 (2009), 14–19.
25. Reed, D.A., Dongarra, J., Exascale computing and big data. Commun. ACM 58:7 (2015), 56–68.
26. K. Shvachko, H. Kuang, S. Radia, R. Chansler, The hadoop distributed file system, in: Mass Storage Systems and Technologies MSST, 2010 IEEE 26th Symposium on, IEEE, 2010, pp. 1–10.
27. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I., Spark: Cluster computing with working sets. HotCloud, 10, 2010 10-10.
28. Apache storm, https://storm.incubator.apache.org.
29. A. Spinuso, R. Filgueira, M. Atkinson, A. Gemuend, Visualisation methods for large provenance collections in data-intensive collaborative platforms, in: EGU General Assembly 2016, Information in Earth Sciences: Visualization Techniques and Communication of Uncertainty, 2016.
30. G. Juve, B. Tovar, R. Ferreira da Silva, D. Król, D. Thain, E. Deelman, W. Allcock, M. Livny, Practical resource monitoring for robust high throughput computing, in: Cluster Computing CLUSTER, 2015 IEEE International Conference on, IEEE, 2015, pp. 650–657. http://dx.doi.org/10.1109/CLUSTER.2015.115.
31. I. Santana-Perez, M.S. Pérez-Hernández, Towards reproducibility in scientific workflows: An infrastructure-based approach, Sci. Program. (2015). http://dx.doi.org/10.1155/2015/243180.
32. Roure, D.D., Goble, C., Stevens, R., The design and realisation of the myexperiment virtual research environment for social sharing of workflows. Future Gener. Comput. Syst., 25, 2008, 7, 10.1016/j.future.2008.06.010.
33. K. Belhajjame, J. Zhao, D. Garijo, M. Gamble, K. Hettne, R. Palma, E. Mina, O. Corcho, J.M. Gómez-Pérez, S. Bechhofer, et al., Using a suite of ontologies for preserving workflow-centric research objects, Web Semant. Sci. Serv. Agents World Wide Web 32 (2015) 16–42. http://dx.doi.org/10.1016/j.websem.2015.01.003.
34. Berry, M.W., Scientific workload characterization by loop-based analyses. ACM SIGMETRICS Perform. Eval. Rev. 19:3 (1992), 17–29.
35. L. Ramakrishnan, D. Gannon, A Survey of Distributed Workflow Characteristics and Resource Requirements, Tech. Rep. TR671, Indiana University 2008.
36. Ostermann, S., Prodan, R., Fahringer, T., Iosup, A., Epema, D., On the characteristics of grid workflows. Priol, T., Vanneschi, M., (eds.) From Grids to Service and Pervasive Computing, 2008, Springer, 191–203.
37. S. Bharathi, A. Chervenak, E. Deelman, G. Mehta, M.-H. Su, K. Vahi, Characterization of scientific workflows, in: Workflows in Support of Large-Scale Science, 2008. WORKS 2008, Third Workshop on, IEEE, 2008, pp. 1–10.
38. N. Dun, K. Taura, A. Yonezawa, Paratrac: a fine-grained profiler for data-intensive workflows, in: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ACM, 2010, pp. 37–48.
39. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K., Characterizing and profiling scientific workflows. Future Gener. Comput. Syst. 29:3 (2013), 682–692.
40. Ferreira da Silva, R., Juve, G., Rynge, M., Deelman, E., Livny, M., Online task resource consumption prediction for scientific workflows. Parallel Process. Lett., 25(03), 2015, 1541003, 10.1142/S0129626415410030.
41. Iosup, A., Epema, D., Grid computing workloads. IEEE Internet Comput. 15:2 (2011), 19–26.
42. Whitlock, B., Favre, J.M., Meredith, J.S., Parallel in situ coupling of simulation with a fully featured visualization system. EGPGV, 2011, 101–109.
43. N. Fabian, K. Moreland, D. Thompson, A. Bauer, P. Marion, B. Geveci, M. Rasquin, K. Jansen, The paraview coprocessing library: A scalable, general purpose In Situ visualization library, in: IEEE Symposium on Large-Scale Data Analysis and Visualization, 2011, pp. 97–104.
44. Yu., H., Wang, C., Grout, R., Chen, J., Ma, K.-L., In situ visualization for large-scale combustion simulations. IEEE Comput.Graph. Appl. 30:3 (2010), 45–57.
45. S. Lakshminarasimhan, J. Jenkins, I. Arkatkar, Z. Gong, H. Kolla, S.-H. Ku, S. Ethier, J. Chen, C.S. Chang, S. Klasky, R. Latham, R. Ross, N.F. Samatova, Isabela-qa: Query-driven analytics with isabela-compressed extreme-scale scientific data, in: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, 2011, pp. 31:1–31:11.
46. F. Zhang, C. Docan, M. Parashar, S. Klasky, N. Podhorszki, H. Abbasi, Enabling in-situ execution of coupled scientific workflow on multi-core platform, in: Parallel Distributed Processing Symposium IPDPS, 2012 IEEE 26th International, 2012, pp. 1352–1363.
47. Ellsworth, D., Green, B., Henze, C., Moran, P., Sandstrom, T., Concurrent visualization in a production supercomputing environment. IEEE Trans. Vis. Comput. Graphics 12:5 (2006), 997–1004.
48. A. Esnard, N. Richart, O. Coulaud, A steering environment for online parallel visualization of legacy parallel simulations, in: Distributed Simulation and Real-Time Applications, 2006, DS-RT’06, Tenth IEEE International Symposium on, 2006, pp. 7 –14.
49. J. Lofstead, F. Zheng, S. Klasky, K. Schwan, Adaptable metadata rich io methods for portable high performance io, in: Parallel Distributed Processing, 2009, IPDPS 2009 IEEE International Symposium on, 2009, pp. 1 –10.
50. C. Docan, M. Parashar, S. Klasky, Dataspaces: An interaction and coordination framework for coupled simulation workflows, in: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC ‘10, 2010, pp. 25–36.
51. V. Vishwanath, M. Hereld, M. Papka, Toward simulation-time data analysis and i/o acceleration on leadership-class systems, in: Large Data Analysis and Visualization LDAV, 2011 IEEE Symposium on, 2011, pp. 9–14.
52. The Future of Scientific Workflows, Report of the DOE NGNS/CS Scientific Workflows Workshop, 2015. https://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pdf.
53. Dean, J., Ghemawat, S., Mapreduce: Simplified data processing on large clusters. Commun. ACM 51:1 (2008), 107–113.
54. W. Mitchell, J. Kallman, A. Toreja, B. Gallagher, M. Jiang, D. Laney, Developing a Learning Algorithm-Generated Empirical Relaxer, Tech. Rep. LLNL-TR-687141, Lawrence Livermore National Laboratory, 2016.
55. M. Jiang, B. Gallagher, J. Kallman, D. Laney, A Supervised Learning Framework for Arbitrary Lagrangian-Eulerian Simulations, in: IEEE International Conference on Machine Learning and Applications, 2016.
56. J.F. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, C. Jin, Flexible io and integration for scientific codes through the adaptable io system (adios), in: Proceedings of the 6th international workshop on Challenges of large applications in distributed environments, ACM, 2008, pp. 15–24.
57. S. Marru, L. Gunathilake, C. Herath, P. Tangchaisin, M. Pierce, C. Mattmann, R. Singh, T. Gunarathne, E. Chinthaka, R. Gardler, et al., Apache airavata: a framework for distributed applications and computational workflows, in: 2011 ACM workshop on Gateway computing environments, ACM, 2011, pp. 21–28. http://dx.doi.org/10.1145/2110486.2110490.
58. Z. Falt, D. Bednárek, M. Kruliš, J. Yaghob, F. Zavoral, Bobolang: A language for parallel streaming applications, in: Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, ACM, 2014, pp. 311–314. http://dx.doi.org/10.1145/2600212.2600711.
59. Barseghian, D., Altintas, I., Jones, M.B., Crawl, D., Potter, N., Gallagher, J., Cornillon, P., Schildhauer, M., Borer, E.T., Seabloom, E.W., et al. Workflows and extensions to the kepler scientific workflow system to support environmental sensor data access and analysis. Ecol. Inform. 5:1 (2010), 42–50, 10.1016/j.ecoinf.2009.08.008.
60. Glatard, T., Montagnat, J., Lingrand, D., Pennec, X., Flexible and efficient workflow deployment of data-intensive applications on grids with moteur. Int. J. High Perform. Comput. Appl. 22:3 (2008), 347–360, 10.1177/1094342008096067.
61. Nextflow. http://www.nextflow. io/index.html.
62. Wilde, M., Hategan, M., Wozniak, J.M., Clifford, B., Katz, D.S., Foster, I., Swift: A language for distributed parallel scripting. Parallel Comput. 37:9 (2011), 633–652, 10.1016/j.parco.2011.05.005.
63. Taylor, I., Shields, M., Wang, I., Harrison, A., The triana workflow environment: Architecture and applications. Workflows for E-Science, 2007, Springer, 320–339.
64. Wang, M., Du, Z., Cheng, Z., Zhu, S., A pipeline virtual service pre-scheduling pattern and its application in astronomy data processing. Simulation 83:1 (2007), 123–132.
65. Silva, C.T., Freire, J., Callahan, S.P., Provenance for visualizations: Reproducibility and beyond. Comput. Sci. Eng. 9:5 (2007), 82–89.
66. Deelman, E., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Patil, S., Su, M.-H., Vahi, K., Livny, M., Pegasus: Mapping scientific workflows onto the grid. Grid Computing, 2004, Springer, 11–20.
67. A. Lathers, M.-H. Su, A. Kulungowski, A. Lin, G. Mehta, S.T. Peltier, E. Deelman, M.H. Ellisman, Enabling parallel scientific applications with workflow tools, in: Challenges of Large Applications in Distributed Environments, 2006 IEEE, IEEE, 2006, pp. 55–60.
68. T. Andrews, F. Curbera, H. Dholakia, Y. Goland, J. Klein, F. Leymann, K. Liu, D. Roller, D. Smith, S. Thatte, et al., Business process execution language for web services, 2003.
69. Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M.R., Wipat, A., et al. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20:17 (2004), 3045–3054.
70. Hull, D., Wolstencroft, K., Stevens, R., Goble, C., Pocock, M.R., Li, P., Oinn, T., Taverna: a tool for building and running workflows of services. Nucleic Acids Res. 34:suppl. 2 (2006), W729–W732.
71. D. De Oliveira, E. Ogasawara, F. Baião, M. Mattoso, Scicumulus: A lightweight cloud middleware to explore many task computing paradigm in scientific workflows, in: Proceedings of the 3rd IEEE International Conference on Cloud Computing CLOUD, IEEE, 2010, pp. 378–385.
72. G.E. Horne, T.E. Meyer, Data farming: Discovering surprise, in: Proceedings of the 36th conference on Winter simulation, Winter Simulation Conference, 2004, pp. 807–813.
73. Mason, T., Abernathy, D., Anderson, I., Ankner, J., Egami, T., Ehlers, G., Ekkebus, A., Granroth, G., Hagen, M., Herwig, K., et al. The spallation neutron source in oak ridge: A powerful tool for materials research. Physica B 385 (2006), 955–960.
74. Deelman, E., Carothers, C., Mandal, A., Tierney, B., Vetter, J.S., Baldin, I., Castillo, C., Juve, G., Krol, D., Lynch, V., Mayer, B., Meredith, J., Proffen, T., Ruth, P., Ferreira da Silva, R., PANORAMA: An approach to performance modeling and diagnosis of extreme scale workflows. Int. J. High Perform. Comput. Appl. 31:1 (2017), 4–18, 10.1177/1094342015594515.
75. Abramson, D., Enticott, C., Altinas, I., Nimrod/k: towards massively parallel dynamic grid workflows. Proceedings of the ACM/IEEE Conference on Supercomputing, 2008, IEEE Press.
76. Abramson, D., Bethwaite, B., Enticott, C., Garic, S., Peachey, T., Parameter space exploration using scientific workflows. Computational Science, ICCS, 2009, Springer, 104–113.
77. Król, D., Kitowski, J., Self-scalable services in service oriented software for cost-effective data farming. Future Gener. Comput. Syst. 54 (2016), 1–15, 10.1016/j.future.2015.07.003.
78. D. Krol, J. Kitowski, R. Ferreira da Silva, G. Juve, K. Vahi, M. Rynge, E. Deelman, Science automation in practice: Performance data farming in workflows, in: 21st IEEE International Conference on Emerging Technologies and Factory Automation ETFA, 2016. http://dx.doi.org/10.1109/ETFA.2016.7733677.
79. Malawski, M., Juve, G., Deelman, E., Nabrzyski, J., Algorithms for cost-and deadline-constrained provisioning for scientific workflow ensembles in iaas clouds. Future Gener. Comput. Syst. 48 (2015), 1–18, 10.1016/j.future.2015.01.004.
80. Guerra, G., Rochinha, F.A., Elias, R., De Ooliveira, D., Ogasawara, E., Dias, J.F., Mattoso, M., Coutinho, A.L., Uncertainty quantification in computational predictive models for fluid dynamics using a workflow management engine. Int. J. Uncertain. Quantif. 2:1 (2012), 53–71.
81. P. Bui, D. Rajan, B. Abdul-Wahid, J. Izaguirre, D. Thain, Work queue+ python: A framework for scalable scientific ensemble applications, in: Workshop on Python for High Performance and Scientific Computing at SC11, 2011.
82. B.M. Adams, W. Bohnhoff, K. Dalbey, J. Eddy, M. Eldred, D. Gay, K. Haskell, P.D. Hough, L. Swiler, Dakota A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 5.0 User's Manual, Sandia National Laboratories, Tech. Rep. SAND2010-2183.
83. Abramson, D., Peachey, T., Lewis, A., Model optimization and parameter estimation with nimrod/o. Computational Science, ICCS, 2006, Springer, 720–727.
84. R. Filgueira, R.F. da Silva, A. Krause, E. Deelman, M. Atkinson, Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science, in: Proceedings of the 7th International Workshop on Data-Intensive Computing in the Cloud, IEEE Press, 2016, pp. 1–8. http://dx.doi.org/10.1109/DataCloud.2016.4.
85. R. Filgueira, A. Krause, M.P. Atkinson, I.A. Klampanos, A. Spinuso, S. Sanchez-Exposito, dispel4py: An agile framework for data-intensive escience, in: 11th IEEE International Conference on e-Science, e-Science 2015, Munich, Germany, August 31-September 4, 2015, pp. 454–464. http://dx.doi.org/10.1109/eScience.2015.40.