Scientific workflow management and the Kepler system

Concurrency and Computation: Practice and Experience

Volumn 18, Issue 10, 2006, Pages 1039-1065

  Ludäscher, Bertram ^a,b   Altintas, Ilkay ^a   Berkley, Chad ^c   Higgins, Dan ^c   Jaeger, Efrat ^a   Jones, Matthew ^c   Lee, Edward A ^d   Tao, Jing ^a   Zhao, Yang ^d  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c NATIONAL CENTER FOR ECOLOGICAL ANALYSIS AND SYNTHESIS   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Dataflow networks; Grid workflows; Problem solving environments; Scientific data management; Scientific workflows

Indexed keywords

COMPUTATION THEORY; DATA FLOW ANALYSIS; PROJECT MANAGEMENT; QUERY LANGUAGES;

DATAFLOW NETWORKS; GRID WORKFLOWS; SCIENTIFIC DATA MANAGEMENT; SCIENTIFIC WORKFLOWS;

DATABASE SYSTEMS;

EID: 33646040949     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.994     Document Type: Conference Paper

References (71)

1. Web Services Description Language (WSDL) Version 1.2, June 2003. http://www.w3.org/TR/wsdl12.
2. OWL Web Ontology Language Reference, W3C Proposed Recommendation, December 2003. http://www.w3.org/TR/owl-ref/.
3. Scientific Data Management Framework Workshop, Argonne National Labs, August 2003. Available at: http://sdm.lbl.gov/~arie/sdm/SDM.Framework.wshp.htm.
4. e-Science Workflow Services Workshop, e-Science Institute, Edinburgh, U.K., December 2003. Available at: http://www.nesc.ac.uk/esi/events/303/index.html.
5. e-Science Grid Environments Workshop, e-Science Institute, Edinburgh, U.K., May 2004. Available at: http://www.nesc.ac.uk/esi/events/.
6. GRIST Workshop on Service Composition for Data Exploration in the Virtual Observatory, California Institute of Technology, July 2004. Available at: http://grist.caltech.edu/sc4devo/.
7. LINK-Up Workshop on Scientific Workflows, San Diego Supercomputer Center, October 2004. Available at: http://kbis.sdsc.edu/events/link-up-11-04/.
8. Workflow in Grid Systems Workshop, GGF10, Berlin, Germany, March 2004. Available at: http://www.extreme.indiana.edu/groc/Worflow-call.html.
9. NSF/ITR. Enabling the Science Environment for Ecological Knowledge (SEEK), http://www.seek.ecoinformatics.org.
10. Michener WK, Beach JH, Jones MB, Ludäscher B, Pennington DD, Pereira RS, Rajasekar A, Schildhauer M. A knowledge environment for the biodiversity and ecological sciences. Journal of Intelligent Information Systems 2004.
11. Altintas I et al. A modeling and execution environment for distributed scientific workflows. Proceedings of the 15th International Conference on Scientific and Statistical Database Management (SSDBM), Boston, MA, 2003.
12. Bowers S, Ludäscher B. An ontology driven framework for data transformation in scientific workflows. Proceedings of the International Workshop on Data Integration in the Life Sciences (DILS), Leipzig, Germany, March 2004 (Lecture Notes in Computer Science, vol. 2994). Springer: Berlin, 2004.
13. Bowers S, Ludäscher B. Actor-oriented design of scientific workflows. Proceedings of the 24th International Conference on Conceptual Modeling, Klagenfurt, Austria, October 2005 (Lecture Notes in Computer Science). Springer: Berlin (to appear).
14. Kepler: A system for scientific workflows. http://www.kepler-project.org.
15. Werner T. Target gene identification from expression array data by promoter analysis. Biomolecular Engineering 2001; 17:87-94.
16. Peterson L, Yin E, Nelson D, Altintas I, Ludäscher B, Critchlow T, Wyrobek AJ, Coleman MA. Mining the frequency distribution of transcription factor binding sites of ionizing radiation responsive genes. New Horizons in Genomics, DOE/SC-0071, Santa Fe, NM, 30 March-1 April 2003.
17. Liu L, Pu C, Han W. An XML-enabled data extraction tool for Web sources. International Journal of Information Systems (Special Issue on Data Extraction, Cleaning, and Reconciliation) 2001.
18. National Center for Biotechnology Information (NCBI), 2004. http://www.ncbi.nlm.nih.gov/.
19. Ptolemy II project and system. Department of EECS, UC Berkeley, 2004. http://ptolemy.eecs.berkeley.edu/ptolemyII/.
20. Brooks C, Lee EA, Liu X, Neuendorffer S, Zhao Y, Zheng H. Heterogeneous concurrent modeling and design in Java (vols. 1-3). Technical Memoranda UCB/ERL M04/27, M04/16, M04/17, Department of EECS, University of California, Berkeley, 2004.
21. Schmidt M et al. The general atomic and molecular electronic structure system. Journal of Computational Chemistry 1993; 14:1347-1363. Available at: http://www.msg.ameslab.gov/GAMESS/GAMESS.html.
22. Abramson D, Giddy J, Kotler L. High performance parametric modeling with Nimrod/G: Killer application for the global Grid. Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS), Cancun, Mexico, May 2000. Available at: http://www.csse.monash.edu.au/~davida/nimrod/.
23. Gao J, Thompson MA. (eds.). Combined Quantum Mechanical and Molecular Mechanical Methods. American Chemical Society, 1998.
24. Sudholt W, Baldridge K, Abramson D, Enticott C, Garic S. Parameter scan of an effective group difference pseudopotential using Grid computing. New Generation Computing 2004; 22:137-146.
25. NSF/ITR. GEON: A research project to create cyberinfrastructure for the geosciences. http://www.geongrid.org.
26. Scientific Data Management Center (SDM). http://sdm.lbl.gov/sdmcenter/; http://www.npaci.edu/online/v5.17/scidac.html.
27. Biomedical Informatics Research Network Coordinating Center (BIRN-CC), University of California, San Diego, CA. http://nbirn.net/.
28. ROADNet: Real-time observatories, applications and data management network, http://roadnet.ucsd.edu.
29. Sheth A. Changing focus on interoperability in information systems: From system, syntax, structure to semantics. Interoperating Geographic Information Systems, Goodchild M, Egenhofer M, Fegeas R, Kottman C (eds.). Kluwer: Dordrecht, 1998; 5-30.
30. Halevy A. Answering queries using views: A survey. VLDB Journal 2001; 10(4):270-294.
31. Nash A, Ludäscher B. Processing unions of conjunctive queries with negation under limited access patterns. Proceedings of the 9th International Conference on Extending Database Technology (EDBT), Heraklion, Crete, Greece, 2004 (Lecture Notes in Computer Science, vol. 2992). Springer: Berlin, 2004; 422-440.
32. Ludäscher B, Gupta A, Martone ME. A model-based mediator system for scientific data management. Bioinformatics: Managing Scientific Data, Lacroix Z, Critchlow T (eds.). Morgan Kaufmann: San Francisco, CA, 2003.
33. Bowers S, Lin K, Ludäscher B. On integrating scientific resources through semantic registration. Proceedings of the 16th International Conference on Scientific and Statistical Database Management (SSDBM), Santorini Island, Greece, 2004.
34. Proceedings of the ICAPS Workshop on Planning for Web Services, Trento, Italy, June 2003.
35. Blythe J, Deelman E, Gil Y. Planning for workflow construction and maintenance on the Grid. Proceedings of the ICAPS Workshop on Planning for Web Services, Trento, Italy, June 2003.
36. Ludäscher B, Altintas I, Gupta A. Compiling abstract scientific workflows into Web service workflows. Proceedings of the 15th International Conference on Scientific and Statistical Database Management (SSDBM), Boston, MA, 2003. Available at: http://kbis.sdsc.edu/SciDAC-SDM/ludaescher-compiling.pdf.
37. Ludäscher B, Nash A. Web service composition through declarative queries: The case of conjunctive queries with union and negation. Proceedings of the 20th International Conference on Data Engineering (ICDE), 2004.
38. Alonso G, Mohan C. Workflow management systems: The next generation of distributed processing tools. Advanced Transaction Models and Architectures, Jajodia S, Kerschberg L (eds.), 1997.
39. zur Muehlen M. Workflow-based Process Controlling. Logos Verlag: Berlin, 2004.
40. van der Aalst W, van Hee K. Workflow Management: Models, Methods, and Systems (Cooperative Information Systems). MIT Press: Cambridge, MA, 2002.
41. van der Aalst W. Don't go with the flow: Web services composition standards exposed. IEEE Intelligent Systems. Web Services - Been There Done That? Trends and Controversies, January/February 2003. Available at: http://tmitwww.tm.tue.nl/research/patterns/download/ieeewebflow.pdf.
42. Curbera F, Goland Y, Klein J, Leyman F, Roller D, Thatte S, Weerawarana S. Business Process Execution Language for Web Services (BPEL4WS), Version 1.0, 2002. http://www.ibm.com/developerworks/library/ws-bpel/.
43. van der Aalst W, ter Hofstede A, Kiepuszewski B, Barros A. Workflow patterns. Distributed and Parallel Databases 2003; 14(3):5-51.
44. The Taverna Project. http://taverna.sf.net/.
45. The Triana Project. http://www.trianacode.org/.
46. Chen I, Markowitz V. The Object-Protocol Model: Design, implementation, and scientific applications. ACM Transactions on Information Systems 1995; 20(5).
47. Meidanis J, Vossen G, Weske M. Using workflow management in DNA sequencing. Proceedings of the International Conference on Cooperative Information Systems (CoopIS), 1996.
48. Ailamaki A, Ioannidis YE, Livny M. Scientific workflow management by database management. Proceedings of the 10th International Conference on Scientific and Statistical Database Management (SSDBM), Capri, Italy, 1998.
49. Kiepuszewski B. Expressiveness and suitability of languages for control flow modelling in workflows. PhD Thesis. Queensland University of Technology, 2002.
50. Kahn G, MacQueen DB. Coroutines and networks of parallel processes. Proceedings of the IFIP Congress 77, Gilchrist B (ed.), 1977; 993-998.
51. Lee EA, Parks T. Dataflow process networks. Proceedings of the IEEE 1995; 83(5):773-799. Available at: http://citeseer.nj.nec.com/455847.html.
52. Morrison JP. Flow-Based Programming - A New Approach to Application Development. Van Nostrand Reinhold: New York, 1994.
53. Wright H, Brodlie K, Brown M. The dataflow visualization pipeline as a problem solving environment. Virtual Environments and Scientific Visualization, Gobel M, David J, Slavik P, van Wijk JJ (eds.). Springer: Berlin, 1996; 267-276.
54. McPhillips TM. Pipelined scientific workflows for inferring evolutionary relationships. Unpublished Paper, Natural Diversity Discovery Project, 2005.
55. Project G. GridFTP - Universal Data Transfer for the Grid, 2000. http://www.globus.org/datagrid/gridftp.html.
56. The Globus Alliance. http://www.globus.org.
57. SDSC Storage Resource Broker. http://www.sdsc.edu/srb/.
58. R - Statistical data analysis. http://www.r-project.org.
59. Eker J, Janneck JW, Lee EA, Liu J, Liu X, Ludvig J, Neuendorffer S, Sachs S, Xiong Y. Taming heterogeneity - the Ptolemy approach. Proceedings of the IEEE 2003; 91(1).
60. Ludäscher B, Altintas I. On providing declarative design and programming constructs for scientific workflows based on process networks. Technical Report SciDAC-SPA-TN-2003-01, San Diego Supercomputer Center, 2003. Available at: http://kbi.sdsc.edu/SciDAC-SDM/scidac-tn-map-constructs.
61. Reekie HJ. Realtime signal processing: Dataflow, visual, and functional programming. PhD Thesis, School of Electrical Engineering, University of Technology, Sydney, 1995.
62. Nikhil RS, Arvind. Implicit Parallel Programming in pH. Morgan Kaufmann: San Francisco, CA, 2001.
63. Ludäscher B. Towards actor-oriented Web service-based scientific workflows (or: How to handle handles). Technical Report, San Diego Supercomputer Center, September 2004.
64. Weinstein D, Parker S, Simpson J, Zimmerman K, Jones G. Visualization in the SCIRun problem-solving environment. Visualization Handbook, Hansen C, Johnson C (eds.). Elsevier: Amsterdam, 2005; 615-632.
65. Oinn T et al. Taverna: A tool for the composition and enactment of bioinformatics workflows. Bioinformatics Journal 2004; 20(17):3045-3054.
66. Churches D, Gombas G, Harrison A, Maassen J, Robinson C, Shields M, Taylor I, Wang I. Programming scientific and distributed workflow with Triana services. Concurrency and Computation: Practice and Experience 2006; [this issue].
67. Deelman E et al. Mapping abstract complex workflows onto Grid environments. Journal of Grid Computing 2003; 1(1):25-39.
68. Thain D, Tannenbaum T, Livny M. Distributed computing in practice: The Condor experience. Concurrency and Computation: Practice and Experience 2005; 17(2-4):323-356.
69. Yu J, Buyya R. A taxonomy of workflow management systems for Grid computing. Technical Report GRIDS-TR-2005-1, Grid Computing and Distributed Systems Laboratory, University of Melbourne, 2005. Available at: http://www.gridbus.org/reports/GridWorkflowTaxonomy.pdf.
70. Takatuska M, Gahegan M. GeoVISTA Studio: A codeless visual programming environment for geoscientific data analysis and visualization. Computers and Geosciences 2002; 28(2):1131-1144.
71. Armstrong R, Gannon D, Geist A, Keahey K, Kohn S, McInnes L, Parker S, Smolinski B. Toward a common component architecture for high-performance scientific computing. Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computation, August 1999.