Scientific process automation and workflow management

Scientific Data Management: Challenges, Technology, and Deployment

Volumn , Issue , 2009, Pages 467-

  Ludäscher, Bertram ^a   Altintas, Ilkay ^b   Bowers, Shawn ^a   Cummings, Julian ^c   Critchlow, Terence ^d   Deelman, Ewa ^e   De Roure, David ^f   Freire, Juliana ^g   Goble, Carole ^h   Jones, Matthew ⁱ   Klasky, Scott ^a   McPhillips, Timothy ^a   Podhorszki, Norbert ^j   Silva, Claudio ^g   Taylor, Ian ^k   Vouk, Mladen ^l  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^d PACIFIC NORTHWEST NATIONAL LABORATORY   (United States)

^e UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^f UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^g UNIVERSITY OF UTAH   (United States)

^h UNIVERSITY OF MANCHESTER   (United Kingdom)

ⁱ UNIVERSITY OF CALIFORNIA   (United States)

^j OAK RIDGE NATIONAL LABORATORY   (United States)

^k CARDIFF UNIVERSITY   (United Kingdom)

^l NORTH CAROLINA STATE UNIVERSITY   (United States)

more..

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85015240841     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1201/9781420069815     Document Type: Chapter

References (90)

1. Scientific Discovery through Advanced Computing (SciDAC), Department of Energy. http://www.scidac.gov/. Accessed July 20, 2009.
2. National Science Foundation, Office of Cyberinfrastructure. http://www.nsf.gov/dir/index.jsp?org=OCI. Accessed July 20, 2009.
3. Defining e-Science. http://www.nesc.ac.uk/nesc/define.html. Accessed July 20, 2009.
4. C. Anderson. The End of Theory: The Data Deluge Makes the Scientific Method Obsolete. WIRED Magazine, June 2008.
5. B. Ludäscher, S. Bowers, and T. McPhillips. Scientific workflows. In T. Özsu and L. Liu, editors, Encyclopedia of Database Systems. Heidelberg, Germany: Springer, 2009.
6. I. Taylor, E. Deelman, D. Gannon, and M. Shields, editors. Workflows for e-Science: Scientific Workflows for Grids. Springer, 2007.
7. E. Deelman, D. Gannon, M. Shields, and I. Taylor. Workflows and e-Science: An overview of workflow system features and capabilities. Future Generation Computer Systems, 25(5): 528-540, 2009.
8. Y. Gil, E. Deelman, M. Ellisman, T. Fahringer, G. Fox, D. Gannon, C. Goble, M. Livny, L. Moreau, and J. Myers. Examining the challenges of scientific workflows. Computer, 40(12): 24-32, 2007.
9. M. Weske. Business Process Management: Concepts, Languages, Architectures. Springer, 2007.
10. J. R. Rice and R. F. Boisvert. From Scientific Software Libraries to Problem-Solving Environments. IEEE Computational Science and Engineering, 3: 44-53, 1996.
11. J.Wainer, M.Weske, G. Vossen, and C. B. Medeiros. ScientificWorkflow Systems. In Proceedings of the NSF Workshop on Workflow and Process Automation in Information Systems: State of the Art and Future Directions, 1996.
12. C. Rueda and M. Gertz. Real-Time Integration of Geospatial Raster and Point Data Streams. In B. Ludäscher and N. Mamoulis, editors, 20th Intl. Conf. on Scientific and Statistical Database Management (SSDBM), volume 5069 of LNCS, pp. 605-611, Hong Kong, China, July 2008. Springer.
13. D. Jordan and J. Evdemo. Web Services Business Process Execution Language, Version 2.0 (WS-BPEL 2.0), April 11, 2007.
14. The Taverna Project, http://www.taverna.org.uk. Accessed July 20, 2009.
15. Y. Wang, C. Hu, and J. Huai. A New Grid Workflow Description Language. In IEEE International Conference on Services Computing, IEEE SCC, pages 257-260. IEEE Computer Society, 2005.
16. C. Brooks, E. A. Lee, X. Liu, S. Neuendorffer, Y. Zhao, and H. Zheng. Heterogeneous Concurrent Modeling and Design in Java (Volume 3: Ptolemy II Domains). Technical Report No. UCB/EECS-2008-37, April 2008. http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-37.pdf. Accessed on July 20, 2009.
17. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, K. Blackburn, A. Lazzarini, A. Arbree, R. Cavanaugh, and S. Koranda. Mapping Abstract Complex Workflows onto Grid Environments. Journal of Grid Computing, 1(1): 25-39, 2003.
18. The Kepler Project, http://www.kepler-project.org. Accessed on July 20, 2009.
19. G. Kahn. The Semantics of Simple Language for Parallel Programming. In IFIP Congress, pp. 471-475, 1974.
20. E. A. Lee and T. M. Parks. Dataflow Process Networks. In Proceedings of the IEEE, pp. 773-799, 1995.
21. E. A. Lee and E. Matsikoudis. The semantics of dataflow with firing. In G. Huet, G. Plotkin, J.-J. Lévy, and Y. Bertot, editors, From Semantics to Computer Science: Essays in memory of Gilles Kahn. Cambridge University Press, March 2008. preprint, http://ptolemy.eecs.berkeley.edu/publications/papers/08/DataflowWithFiring/. Accessed on July 20, 2009.
22. T. McPhillips, S. Bowers, D. Zinn, and B. Ludäscher. Scientific workflows for mere mortals. Future Generation Computer Systems, 25(5): 541-551, 2009. http://dx.doi.org/10.1016/j.future.2008.06.013. Accessed on July 20, 2009.
23. U.S. Department of Energy, The Center for Plasma Edge Simulation Project (CPES). http://www.cims.nyu.edu/cpes. Accessed October 10th, 2009.
24. S.-H. Ku, C. Chang, M. Adams, J. Cummings, F. Hinton, D. Keyes, S. Klasky, W. Lee, Z. Lin, and S. Parker. Gyrokinetic particle simulation of neoclassical transport in the pedestal/scrape-off region of a Tokamak plasma. Journal of Physics: Conference Series, 46: 87-91, 2006. Institute of Physics Publishing.
25. J. Cummings, A. Pankin, N. Podhorszki, G. Park, S. Ku, R. Barreto, S. Klasky, C. Chang, H. Strauss, L. Sugiyama, P. Snyder, D. Pearlstein, B. Ludäscher, G. Bateman, and A. Kritz. Plasma Edge Kinetic-MHD Modeling in Tokamaks Using Kepler Workflow for Code Coupling, Data Management and Visualization. Communications in Computational Physics, 4(3): 675-702, 2008.
26. N. Podhorszki, B. Ludäscher, and S. A. Klasky. Workflow Automation for Processing Plasma Fusion Simulation Data. In 2nd Workshop on Workflows in Support of Large-Scale Science (WORKS), pp. 35-44, 2007.
27. D. Zinn, S. Bowers, T. McPhillips, and B. Ludäscher. X-CSR: Dataflow Optimization for Distributed XML Process Pipelines. In ICDE, 2009.
28. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice & Experience, 18(10): 1039-1065, 2006.
29. The Pegasus Project, http://pegasus.isi.edu. Accessed July 20, 2009.
30. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, S. Patil, M.-H. Su, K. Vahi, and M. Livny. Pegasus: Mapping Scientific Workflows onto the Grid. In European Across Grids Conference, pp. 11-20, 2004.
31. T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. Pocock, A. Wipat, and P. Li. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20(17), 2004.
32. T. Oinn, M. Greenwood, M. Addis, M. N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M. R. Pocock, M. Senger, R. Stevens, A. Wipat, and C.Wroe. Taverna: lessons in creating a workflow environment for the life sciences. In Fox and Gannon [32].
33. The Triana Project, http://www.trianacode.org. Accessed July 20, 2009.
34. I. J. Taylor, M. S. Shields, I. Wang, and O. F. Rana. Triana applications within grid computing and peer to peer environments. Journal of Grid Computing, 1(2): 199-217, 2003.
35. Y. Gil, V. Ratnakar, E. Deelman, G. Mehta, and J. Kim. Wings for Pegasus: Creating Large-Scale Scientific Applications Using Semantic Representations of Computational Workflows. In AAAI, pp. 1767-1774, 2007.
36. J. Yu and R. Buyya. A taxonomy of scientific workflow systems for grid computing. In Ludäscher and Goble [55].
37. The Condor Team. Condor Version 7.0.4 Manual. University of Wisconsin-Madison, 2002. http://www.cs.wisc.edu/condor/manual/v7.0/. Accessed July 21, 2009.
38. G. B. Berriman, J. C. Good, A. C. Laity, A. Bergou, J. Jacob, D. S. Katz, E. Deelman, C. Kesselman, G. Singh, M. Su, and R. Williams. Montage: A Grid-Enabled Image Mosaic Service for NVO. Astronomical Data Analysis Software and Systems, ADASS, 13, 2003.
39. D. S. Katz, A. Bergou, G. B. Berriman, G. L. Block, J. Collier, D. W. Curkendall, J. Good, L. Husman, J. C. Jacob, A. C. Laity, P. Li, C. Miller, T. Prince, H. Siegel, and R. Williams. Accessing and Visualizing Scientific Spatiotemporal Data. In 16th Intl. Conf. on Scientific and Statistical Database Management (SSDBM), pp. 107-110. IEEE Computer Society, 2004.
40. A. Lathers, M.-H. Su, A. Kulungowski, A. Lin, G. Mehta, S. Peltier, E. Deelman, and M. Ellisman. Enabling parallel scientific applications with workflow tools. In Challenges of Large Applications in Distributed Environments, 2006 IEEE, pp. 55-60, 2006.
41. J. Muench, H. P. Maechling, Francoeur, D. Okaya, and Y. Cui. SCEC Earthworks Science Gateway: Widening SCEC Community Access to the TeraGrid. In The First Annual TeraGrid Conference, 2006.
42. C. Berkley, S. Bowers, M. Jones, B. Ludäscher, M. Schildhauer, and J. Tao. Incorporating Semantics in Scientific Workflow Authoring. In J. Frew, editor, 17th Intl. Conf. on Scientific and Statistical Database Management (SSDBM), pp. 75-78, 2005.
43. S. Bowers and B. Ludäscher. Actor-Oriented Design of Scientific Workflows. In 24th Intl. Conference on Conceptual Modeling (ER), pp. 369-384, Klagenfurt, Austria, October 2005. Springer.
44. K. Belhajjame, S. M. Embury, N. W. Paton, R. Stevens, and C. A. Goble. Automatic annotation of Web services based on workflow definitions. ACM Trans. Web, 2(2): 1-34, 2008.
45. C. Goble, K. Wolstencroft, A. Goderis, D. Hull, J. Zhao, P. Alper, P. Lord, C. Wroe, K. Belhajjame, D. Turi, R. Stevens, and D. D. Roure. Knowledge discovery for in silico experiments with Taverna: Producing and consuming semantics on the Web of science. In C. Baker and K.-H. Cheung, editors, Semantic Web: Revolutionising Knowledge Discovery in Life Sciences. New York: Springer Science & Business Media. 2007.
46. P. Lord, P. Alper, C. Wroe, and C. Goble. Feta: A Light-Weight Architecture for User Oriented Semantic Service Discovery. In 2nd European Semantic Web Conference, volume 3532 of LNCS, pp. 17-31, Crete, 2005. Springer.
47. C. Goble, R. Stevens, D. Hull, K. Wolstencroft, and R. Lopez. Data curation + process curation = data integration + science. Briefings in Bioinformatics, December 2008.
48. I. Taylor. Triana Generations. In 2nd Intl. Conf. on e-Science and Grid Technologies (e-Science), page 143. IEEE Computer Society, 2006.
49. D. Hull, R. Stevens, P. Lord, C. Wroe, and C. Goble. Treating shimantic Web syndrome with ontologies. In First Advanced Knowledge Technologies Workshop on Semantic Web Services (AKT-SWS04), CEUR-WS.org, Volume 122, 2004.
50. J. Kommineni and D. Abramson. GriddLeS Enhancements and Building Virtual Applications for the GRID with Legacy Components. In Advances in Grid Computing (European Grid Conference), pp. 961-971, 2005.
51. D. Abramson, R. Sosic, J. Giddy, and B. Hall. Nimrod: A tool for performing parametised simulations using distributed workstations. In 4th IEEE Symposium on High Performance Distributed Computing, pp. 112-121, 1995.
52. GridLab Resource Management System (GRMS), 2005. http://www.gridlab.org/WorkPackages/wp-9/. Accessed July 20, 2009.
53. G. Singh, M.-H. Su, K. Vahi, E. Deelman, G. B. Berriman, J. Good, D. S. Katz, and G. Mehta. Workflow Task Clustering for Best Effort Systems with Pegasus. In Mardi Gras Conference, p. 9, 2008.
54. R. Wolski, D. Nurmi, and J. Brevik. An Analysis of Availability Distributions in Condor. In IPDPS, pp. 1-6, 2007.
55. Y. Simmhan, B. Plale, and D. Gannon. A Survey of Data Provenance in e-Science. In Ludäscher and Goble, see Reference 82.
56. Grid Resource Allocation and Management (GRAM), 2008. http://www.globus.org/toolkit/docs/4.2/4.2.0/developer/globusrun-ws.html. Accessed July 20, 2009.
57. A. Goderis, C. Brooks, I. Altintas, E. A. Lee, and C. A. Goble. Composing Different Models of Computation in Kepler and Ptolemy II. In Y. Shi, G. D. van Albada, J. Dongarra, and P. M. A. Sloot, editors, International Conference on Computational Science (3), volume 4489 of Lecture Notes in Computer Science, pp. 182-190. Springer, 2007.
58. N. Mandal, E. Deelman, G. Mehta, M.-H. Su, and K. Vahi. Integrating existing scientific workflow systems: the Kepler/Pegasus example. In 2nd Workshop on Workflows in Support of Large-Scale Science (WORKS’07), pages 21-28, New York, NY, USA, 2007. ACM.
59. S. Bowers, T. McPhillips, M. Wu, and B. Ludäscher. Project Histories: Managing Data Provenance Across Collection-Oriented Scientific Workflow Runs. In 4th Intl. Workshop on Data Integration in the Life Sciences (DILS), University of Pennsylvania, June 2007.
60. S. B. Davidson and J. Freire. Provenance and ScientificWorkflows: Challenges and Opportunities. In SIGMOD Conference, pp. 1345-1350, 2008.
61. B. Ludäscher, N. Podhorszki, I. Altintas, S. Bowers, and T. McPhillips. From computation models to models of provenance: The RWS approach. Concurrency and Computation: Practice & Experience, 20(5): 507-518, 2008.
62. S. Bowers, T. M. McPhillips, B. Ludäscher, S. Cohen, and S. B. Davidson. A Model for User-Oriented Data Provenance in Pipelined Scientific Workflows. In L. Moreau and I. T. Foster, editors, Intl. Provenance and Annotation Workshop (IPAW), volume 4145 of Lecture Notes in Computer Science, pp. 133-147. Springer, 2006.
63. P. Missier, K. Belhajjame, J. Zhao, and C. Goble. Data lineage model for Taverna workflows with lightweight annotation requirements. In Intl. Provenance and Annotation Workshop (IPAW), 2008.
64. S. Bowers, T. McPhillips, S. Riddle, M. Anand, and B. Ludäscher. Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life. In Intl. Provenance and Annotation Workshop (IPAW), 2008.
65. The VisTrails Project, http://www.vistrails.org. Accessed July 20, 2009.
66. A. Shoshani, I. Altintas, A. Choudhary, T. Critchlow, C. Kamath, B. Ludäscher, J. Nieplocha, S. Parker, R. Ross, N. Samatova, and M. Vouk. SDM Center technologies for accelerating scientific discoveries. Journal of Physics: Conference Series, 78, 2007.
67. C. Scheidegger, D. Koop, E. Santos, H. Vo, S. Callahan, J. Freire, and C. Silva. Tackling the provenance challenge one layer at a time. Concurrency and Computation: Practice & Experience, 20(5): 473-483, 2008.
68. J. Freire, C. T. Silva, S. P. Callahan, E. Santos, C. E. Scheidegger, and H. T. Vo. Managing Rapidly-Evolving Scientific Workflows. In Intl. Provenance and Annotation Workshop (IPAW), volume 4145 of Lecture Notes in Computer Science, pages 10-18. Springer, 2006.
69. C. E. Scheidegger, H. T. Vo, D. Koop, J. Freire, and C. T. Silva. Querying and creating visualizations by analogy. IEEE Trans. Vis. Comput. Graph., 13(6): 1560-1567, 2007.
70. L. Moreau, et al. The first provenance challenge. Concurrency and Computation: Practice and Experience-Special Issue on the First Provenance Challenge, 20(5), 2008.
71. R. Barreto, T. Critchlow, A. Khan, S. Klasky, L. Kora, J. Ligon, P. Mouallem, M. Nagappan, N. Podhorszki, and M. Vouk. Managing and Monitoring Scientific Workflows through Dashboards. In Microsoft eScience Workshop, p. 108, Chapell Hill, NC, 2007.
72. I. Altintas, G. Chin, D. Crawl, T. Critchlow, D. Koop, J. Ligon, B. Ludäscher, P. Mouallem, M. Nagappan, N. Podhorszki, C. Silva, and M. Vouk. Provenance in Kepler-based Scientific Workflow Systems. In Microsoft eScience Workshop, page 82, 2007. Poster.
73. I. Altintas, O. Barney, and E. Jaeger-Frank. Provenance Collection Support in the Kepler Scientific Workflow System. In L. Moreau and I. T. Foster, editors, Intl. Provenance and Annotation Workshop (IPAW), volume 4145 of Lecture Notes in Computer Science, pp. 118-132. Springer, 2006.
74. D. Crawl and I. Altintas. A Provenance-Based Fault Tolerance Mechanism for Scientific Workflows. In Intl. Provenance and Annotation Workshop (IPAW), 2008.
75. S. A. Klasky, M. Beck, V. Bhat, E. Feibush, B. Ludäscher, M. Parashar, A. Shoshani, D. Silver, and M. Vouk. Data management on the fusion computational pipeline. Journal of Physics: Conference Series, 16: 510-520, 2005.
76. J. Kim, E. Deelman, Y. Gil, G. Mehta, and V. Ratnakar. Provenance trails in the Wings-Pegasus system. Concurrency and Computation: Practice & Experience, 20(5): 587-597, 2008.
77. P. Fisher, C. Hedeler, K. Wolstencroft, H. Hulme, H. Noyes, S. Kemp, R. Stevens, and A. Brass. A systematic strategy for large-scale analysis of genotype phenotype correlations: identification of candidate genes involved in African trypanosomiasis. Nucleic Acids Res, 35(16): 5625-5633, 2007.
78. P. Maechling, H. Chalupsky, M. Dougherty, E. Deelman, Y. Gil, S. Gullapalli, V. Gupta, C. Kesselman, J. Kim, G. Mehta, B. Mendenhall, T. A. Russ, G. Singh, M. Spraragen, G. Staples, and K. Vahi. Simplifying Construction of Complex Workflows for Non-Expert Users of the Southern California Earthquake Center Community Modeling Environment. In Ludäscher and Goble [55].
79. D. D. Roure, C. Goble, and R. Stevens. The design and realisation of the myExperiment virtual research environment for social sharing of workflows. Future Generation Computer Systems, 25: 561-567, 2009.
80. R. D. Stevens, A. J. Robinson, and C. A. Goble. myGrid: Personalised Bioinformatics on the Information Grid. In 11th Intl. Conf. on Intelligent Systems for Molecular Biology (ISMB), pp. 302-304, 2003.
81. D. D. Roure and C. Goble. Software design for empowering scientists. IEEE Software, 26(1): 88-95, January/February 2009.
82. B. Ludäscher and C. Goble, editors. ACM SIGMOD Record: Special Issue on Scientific Workflows, volume 34(3), September 2005.
83. G. C. Fox and D. Gannon, editors. Concurrency and Computation: Practice & Experience. Special Issue: Workflow in Grid Systems, volume 18(10). Hoboken, New Jersey: John Wiley & Sons, 2006.
84. E. Deelman and A. Chervenak. Data Management Challenges of Data-Intensive Scientific Workflows. In 8th IEEE Intl. Symposium on Cluster Computing and the Grid (CCGRID), pp. 687-692. IEEE Computer Society, 2008.
85. S. B. Davidson, S. C. Boulakia, A. Eyal, B. Ludäscher, T. M. McPhillips, S. Bowers, M. K. Anand, and J. Freire. Provenance in Scientific Workflow Systems. IEEE Data Eng. Bull., 30(4): 44-50, 2007.
86. B. Ludäscher, M. Weske, T. McPhillips, and S. Bowers. Scientific-Workflows: Business as Usual? In U. Dayal, J. Eder, J. Koehler, and H. Reijers, editors, 7th Intl. Conf. on Business Process Management (BPM), LNCS 5701, Ulm, Germany, 2009.
87. C. Beeri, A. Eyal, S. Kamenkovich, and T. Milo. Querying Business Processes. In Very Large Data Bases (VLDB) conference pp. 343-354, 2006.
88. J. Freire, D Koop, E. Santos, and C. T. Silva. Provenance for Computational Tasks: A Survey. Computing in Science and Engineering, 10(3): 11-21, 2008.
89. T. McPhillips, S. Bowers, and B. B. Ludäscher. Collection-Oriented Scientific Workflows for Integrating and Analyzing Biological Data. In 3rd Intl. Workshop on Data Integration in the Life Sciences (DILS), LNCS, European Bioinformatics Institute, Hinxton, UK, July 2006. Springer.
90. C. E. Scheidegger, D. Koop, E. Santos, H. T. Vo, S. P. Callahan, J. Freire, and C. T. Silva. Tackling the Provenance Challenge one layer at a time. Concurrency and Computation: Practice and Experience, 20(5): 473-483, 2008.