Scaling up workflow-based applications

Journal of Computer and System Sciences

Volumn 76, Issue 6, 2010, Pages 428-446

  Callaghan, Scott ^a   Deelman, Ewa ^b   Gunter, Dan ^e   Juve, Gideon ^a   Maechling, Philip ^a   Brooks, Christopher ^f   Vahi, Karan ^b   Milner, Kevin ^a   Graves, Robert ^c   Field, Edward ^d   Okaya, David ^a   Jordan, Thomas ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c URS CORPORATION   (United States)

^d U S GEOLOGICAL SURVEY   (United States)

^e LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^f UNIVERSITY OF SAN FRANCISCO   (United States)

Author keywords

Distributed applications; Scientific workflows; Workflow scalability

Indexed keywords

SCALABILITY; WORK SIMPLIFICATION;

COMPUTATIONAL WORKFLOWS; CYBER INFRASTRUCTURES; DISTRIBUTED APPLICATIONS; PERFORMANCE ANALYSIS; SCIENCE APPLICATIONS; SCIENTIFIC APPLICATIONS; SCIENTIFIC WORKFLOWS; WORKFLOW-BASED APPLICATIONS;

SEARCH ENGINES;

EID: 77955308515     PISSN: 00220000     EISSN: 10902724     Source Type: Journal    
DOI: 10.1016/j.jcss.2009.11.005     Document Type: Conference Paper

References (57)

1. E. Deelman, D. Gannon, M. Shields, I. Taylor, Workflows and e-science: An overview of workflow system features and capabilities, Future Generation Computer Systems, doi:10.1016/j.future.2008.06.012, 2008
2. I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
3. B. Berriman, A. Bergou, E. Deelman, J. Good, J. Jacob, D. Katz, C. Kesselman, A. Laity, G. Singh, M.-H. Su, R. Williams, Montage: A grid-enabled image mosaic service for the NVO, in: Astronomical Data Analysis Software & Systems (ADASS), vol. XIII, 2003
4. G.B. Berriman, E. Deelman, J. Good, J. Jacob, D.S. Katz, C. Kesselman, A. Laity, T.A. Prince, G. Singh, M.-H. Su, Montage: A grid enabled engine for delivering custom science-grade mosaics on demand, in: SPIE Conference 5487: Astronomical Telescopes, 2004
5. G.B. Berriman, Ewa Deelman, J. Good, J.C. Jacob, D.S. Katz, A.C. Laity, T.A. Prince, G. Singh, and M.-H. Su Generating Complex Astronomy Workflows I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
6. I. Taylor, M. Shields, I. Wang, R. Philp, Distributed P2P computing within Triana: A Galaxy visualization test case, in: IPDPS 2003, 2003
7. T. Oinn, P. Li, D.B. Kell, C. Goble, A. Goderis, M. Greenwood, D. Hull, R. Stevens, D. Turi, and J. Zhao Taverna/myGrid: Aligning a workflow system with the life sciences community I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
8. R.D. Stevens, A.J. Robinson, and C.A. Goble myGrid: Personalised bioinformatics on the information grid Eleventh International Conference on Intelligent Systems for Molecular Biology Bioinformatics 19 2003 302 304
9. S. Callaghan, P. Maechling, E. Deelman, K. Vahi, G. Mehta, G. Juve, K. Milner, R. Graves, E. Field, D. Okaya, D. Gunter, K. Beattie, T. Jordan, Reducing time-to-solution using distributed high-throughput mega-workflows - Experiences from SCEC CyberShake, in: Fourth IEEE International Conference on e-Science (e-Science 2008), Indianapolis, IN, USA, 2008
10. E. Deelman, S. Callaghan, E. Field, H. Francoeur, R. Graves, N. Gupta, V. Gupta, T.H. Jordan, C. Kesselman, P. Maechling, J. Mehringer, G. Mehta, D. Okaya, K. Vahi, L. Zhao, Managing large-scale workflow execution from resource provisioning to provenance tracking: The CyberShake example, in: E-SCIENCE '06: Proceedings of the Second IEEE International Conference on e-Science and Grid Computing, 2006, p. 14
11. P. Maechling, E. Deelman, L. Zhao, R. Graves, G. Mehta, N. Gupta, J. Mehringer, C. Kesselman, S. Callaghan, D. Okaya, H. Francoeur, V. Gupta, Y. Cui, K. Vahi, T. Jordan, and E. Field SCEC CyberShake workflows - Automating probabilistic seismic hazard analysis calculations I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
12. D.A. Brown, P.R. Brady, A. Dietz, J. Cao, B. Johnson, and J. McNabb A case study on the use of workflow technologies for scientific analysis: Gravitational wave data analysis I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
13. L. Piccoli, Lattice QCD workflows: A case study, in: SWBES08: Challenging Issues in Workflow Applications, Indianapolis, IN, 2008
14. M. Jones, B. Ludaescher, D. Pennington, A. Rajasekar, Data integration and workflow solutions for ecology, in: Data Integration in Life Sciences, 2005
15. Open Science Grid http://www.opensciencegrid.org
16. TeraGrid http://www.teragrid.org/
17. Enabling Grids for E-sciencE (EGEE) http://www.eu-egee.org/
18. Southern California Earthquake Center (SCEC) http://www.scec.org/ 2006
19. L. Zhao, P. Chen, and T.H. Jordan Strain Green's tensors, reciprocity, and their applications to seismic source and structure studies Bull. Seismol. Soc. Amer. 96 2006 1753 1763
20. 2007 Working Group on California Earthquake Probabilities, The Uniform California Earthquake Rupture Forecast, Version 2 (UCERF 2): U.S. Geological Survey Open-File Report 2007-1437 and California Geological Survey Special Report 203 http://pubs.usgs.gov/of/2007/1437/ 2008
21. Pegasus http://pegasus.isi.edu
22. E. Deelman, M. Livny, G. Mehta, A. Pavlo, G. Singh, M.-H. Su, K. Vahi, and R.K. Wenger Pegasus and DAGMan from concept to execution: Mapping scientific workflows onto today's cyberinfrastructure L. Grandinetti, High Performance Computing (HPC) and Grids in Action 2008 IOS Press
23. E. Deelman, G. Mehta, G. Singh, M.-H. Su, and K. Vahi Pegasus: Mapping large-scale workflows to distributed resources I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
24. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob, and D.S. Katz Pegasus: A framework for mapping complex scientific workflows onto distributed systems Scientific Programming J. 13 2005 219 237
25. DAGMan http://www.cs.wisc.edu/condor/dagman
26. R. Henderson, D. Tweten, Portable Batch System: External Reference Specification, 1996
27. S. Zhou, LSF: Load sharing in large-scale heterogeneous distributed systems, in: Workshop on Cluster Computing, 1992
28. M.J. Litzkow, M. Livny, M.W. Mutka, Condor - a hunter of idle workstations, in: Proc. 8th Int. Conf. on Distributed Computin Systems, 13-17 June 1988, pp. 104-111
29. V. Welch, F. Siebenlist, I. Foster, J. Bresnahan, K. Czajkowski, J. Gawor, C. Kesselman, S. Meder, L. Pearlman, S. Tuecke, Security for grid services, in: Twelfth International Symposium on High Performance Distributed Computing (HPDC-12), 2003
30. A. Chervenak, E. Deelman, I. Foster, L. Guy, W. Hoschek, A. Iamnitchi, C. Kesselman, P. Kunst, M. Ripenu, B. Schwartzkopf, H. Stockinger, K. Stockinger, B. Tierney, Giggle: A framework for constructing scalable replica location services, in: Proceedings of Supercomputing 2002 (SC2002), 2002
31. E. Deelman, C. Kesselman, G. Mehta, Transformation catalog design for GriPhyN, Technical report GriPhyN-2001-17, 2001
32. J. Voeckler, G. Mehta, Y. Zhao, E. Deelman, M. Wilde, Kickstarting remote applications, in: GCE06 Second International Workshop on Grid Computing Environments, Tampa, Florida, 2006
33. B. Tierney, D. Gunter, NetLogger: A toolkit for distributed system performance tuning and debugging, in: Proceedings of the 8th IFIP/IEEE International Symposium on Integrated Network Management, 2003
34. R. Ihaka, and R. Gentleman R: A language for data analysis and graphics J. Comput. Graph. Statist. 1996 299 314
35. Grid Logging: Best Practices Guide http://www.cedps.net/index.php/ LoggingBestPractices 2008
36. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger-Frank, M. Jones, E. Lee, J. Tao, Y. Zhao, Scientific workflow management and the Kepler system, Concurrency and Computation: Practice & Experience, Special Issue: Workflow in Grid Systems (2005)
37. D. Hull, K. Wolstencroft, R. Stevens, C. Goble, M.R. Pocock, P. Li, and T. Oinn Taverna: A tool for building and running workflows of services Nucleic Acids Res. 34 April 2006 729 732
38. T. Andrews, F. Curbera, H. Dholakia, Y. Goland, J. Klein, F. Leymann, K. Liu, D. Roller, D. Smith, S. Thatte, I. Trickovic, and S. Weerawarana Specification: Business process execution language for web services Version 1.1, 2003 http://www-106.ibm.com/developerworks/webservices/library/ws-bpel/
39. A. Slominski Adapting BPEL to scientific workflows I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows in e-Science 2006 Springer
40. Pegasus, Generating abstract workflows for Pegasus Planner http://pegasus.isi.edu/mapper/docs/guides/ch0/index.html 2009
41. condor-glidein http://www.cs.wisc.edu/condor/glidein
42. G. Juve, E. Deelman, Resource provisioning options for large-scale scientific workflows, in: Third International Workshop on Scientific Workflows and Business Workflow Standards in e-Science (SWBES) in conjunction with Fourth IEEE International Conference on e-Science (e-Science 2008) Indianapolis, Indiana, USA, 2008
43. G. Singh, M.H. Su, K. Vahi, E. Deelman, B. Berriman, J. Good, D.S. Katz, G. Mehta, Workflow task clustering for best effort systems with Pegasus, in: Proceedings of the 15th ACM Mardi Gras Conference: From Lightweight Mash-Ups to Lambda Grids: Understanding the Spectrum of Distributed Computing Requirements, Applications, Tools, Infrastructures, Interoperability, and the Incremental Adoption of Key Capabilities, 2008
44. R. Graves, Physics based probabilistic seismic hazard calculations for Southern California, in: 14th World Conference on Earthquake, Engineering, Beijing, China, 2008
45. J. Yu, and R. Buyya A taxonomy of workflow management systems for grid computing J. Grid Comput. 3 2005 171 200
46. S. Bharathi, A. Chervenak, E. Deelman, G. Mehta, M.-H. Su, K. Vahi, Characterization of scientific workflows, in: 3rd Workshop on Workflows in Support of Large-Scale Science (WORKS08), 2008
47. Y. Gil, E. Deelman, M. Ellisman, T. Fahringer, G. Fox, D. Gannon, C. Goble, M. Livny, L. Moreau, and J. Myers Examining the challenges of scientific workflows IEEE Comput. 40 2007 24 32
48. L. Ramakrishnan, Y. Simmhan, and B. Plale Realization of dynamically adaptive weather analysis and forecasting in LEAD: Four years down the road IPAW 4487 2007 1122
49. J. Pahwa, R. White, A. Jones, M. Burgess, W. Gray, N. Fiddian, T. Sutton, P. Brewer, C. Yesson, N. Caithness, Accessing biodiversity resources in computational environments from workflow applications, in: WORKS06, 2006
50. Active BPEL Engine http://www.active-endpoints.com/active-bpel-engine- overview.htm 2007
51. Microsoft Workflow Foundation http://msdn2.microsoft.com/en-us/ netframework/aa663328.aspx 2007
52. S. Miles, J. Papay, C. Wroe, P. Lord, C. Goble, L. Moreau, Semantic description, publication and discovery of workflows in myGrid, Electronics and Computer Science, University of Southampton, Technical report ECSTR-IAM04-001, 2004
53. D. Churches, G. Gombas, A. Harrison, J. Maassen, C. Robinson, M. Shields, I. Taylor, and I. Wang Programming scientific and distributed workflow with Triana services Concurrency and Computation: Practice and Experience 18 2006 1021 1037
54. S.P. Callahan, J. Freire, E. Santos, C.E. Scheidegger, C.T. Silva, H.T. Vo, VisTrails: Visualization meets Data Management, in: ACM SIGMOD, 2006
55. G. von Laszewski, and M. Hategan Workflow concepts of the Java CoG Kit J. Grid Comput. 3 2006 239 258
56. A. Kertész, G. Sipos, and P. Kacsuk Brokering multi-grid workflows in the P-GRADE portal Euro-Par 2006: Parallel Processing vol. 4375 2006 Springer Berlin
57. M. Wieczorek, R. Prodan, and T. Fahringer Scheduling of scientific workflows in the ASKALON grid environment SIGMOD Record 34 2005