Scripting distributed scientific workflows using Weaver

Concurrency and Computation: Practice and Experience

Volumn 24, Issue 15, 2012, Pages 1685-1707

  Bui, Peter ^a   Yu, Li ^a   Thrasher, Andrew ^a   Carmichael, Rory ^a   Lanc, Irena ^a   Donnelly, Patrick ^a   Thain, Douglas ^a  

^a University of Notre Dame   (United States)

Author keywords

distributed systems; makeflow; python; scripting; weaver; workflow

Indexed keywords

COMPUTER SOFTWARE; DATA HANDLING; DIRECTED GRAPHS; DISTRIBUTED DATABASE SYSTEMS; PROBLEM ORIENTED LANGUAGES;

DISTRIBUTED SYSTEMS; MAKEFLOW; PYTHON; SCRIPTING; WEAVER; WORKFLOW;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 84866508528     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.1871     Document Type: Article

References (34)

1. Dean J, Ghemawat S, Google Inc. MapReduce: simplified data processing on large clusters. In OSDI'04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation. USENIX Association, 2004.
2. Moretti C, Bulosan J, et al,. All-Pairs: an abstraction for data intensive cloud computing. IEEE International Parallel and Distributed Processing Symposium (IPDPS), Miami, FL, 2008; 1-11.
3. Isard M, Budiu M, et al,. Dryad: distributed data parallel programs from sequential building blocks, Proceedings of EuroSys. Lisbon, Portugal, 2007.
4. The directed acyclic graph manager, 2002. Available from: http://www.cs.wisc.edu/condor/dagman.
5. Deelman E, Singh G, et al,. Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Scientific Programming Journal 2005; 13 (3): 219-237.
6. Thain D, Tannenbaum T, et al,. Condor and the grid. In Grid Computing: Making the Global Infrastructure a Reality, Berman F, Fox G, Hey T, (eds). John Wiley: Chichester, 2003
7. Chambers C, Raniwala A, et al,. FlumeJava: easy, efficient data-parallel pipelines. In Proceedings of the 2010 ACM SIGPLAN conference on Programming Language Design and Implementation, PLDI '10. ACM: New York, NY, USA, 2010; 363-375.
8. Isard M, Yu Y,. Distributed data-parallel computing using a high-level programming language. In SIGMOD '09: Proceedings of the 35th SIGMOD International Conference on Management of Data. ACM: New York, NY, USA, 2009; 987-994.
9. Bui P, Yu L, et al,. Weaver: integrating distributed computing abstractions into scientific workflows using Python. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10. ACM: New York, NY, USA, 2010; 636-643.
10. Zhao Y, Dobson J, Foster I, Wilde LMM,. A Notation and System for Expressing and Executing Cleanly Typed Workflows on Messy Scientific Data. SIGMOD Record. 2005; 34: 37-43.
11. Barker A, van Hemert J,. Scientific workflow: a survey and research directions. In Seventh International Conference on Parallel Processing and Applied Mathematics, Revised Selected Papers, Vol. 4967, Wyrzykowski R, et al. (eds), LNCS. Springer: Berlin, Germany, 2008; 746-753.
12. Python Programming Language, 2010. Available from: http://www.python.org/ .
13. Yu L, Moretti C, et al,. Harnessing parallelism in multi-core clusters with the all-pairs, wavefront, and makeflow abstractions. Cluster Computing 2010; 13: 243-256.
14. Thrasher A, Carmichael R, et al,. Taming complex bioinformatics workflows with Weaver, Makeflow, and Starch. Workflows in Support of Large-Scale Science (WORKS), 2010 5th Workshop on, New Orleans, LA, 2010; 1-6.
15. SQLAlchemy, 2010. Available from: http://sqlalchemy.org/.
16. Sirvent R, Pérez JM, et al,. Automatic grid workflow based on imperative programming languages: research articles. Concurrency Computing: Practice and Experience August 2006; 18: 1169-1186. DOI: 10.1002/cpe.v18:10.
17. Singh G, Kesselman C, et al,. Optimizing grid-based workflow execution. Journal of Grid Computing 2005; 3: 201-219.
18. von Laszewski G, Hategan M,. Workflow concepts of the Java CoG kit. Journal of Grid Computing 2005; 3: 239-258. DOI: 10.1007/s10723-005-9013-5.
19. Bui H, Kelly M, et al,. Experience with BXGrid: a data repository and computing grid for biometrics research. Journal of Cluster Computing 2009; 12 (4): 373.
20. Bui H, Bui P, et al,. ROARS: a scalable repository for data intensive scientific computing. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10. ACM: New York, NY, USA, 2010; 766-775.
21. Bui H, Wright D, et al,. Towards long term data quality in a large scale biometrics experiment. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10. ACM: New York, NY, USA, 2010; 565-572.
22. Altschul S, Gish W, et al,. Basic Local Alignment Search Tool. Journal of Molecular Biology 1990; 3 (215): 403-410.
23. Korbel J, Abyzov A, et al,. PEMer: a computational framework with simulation-based error models for inferring genomic structural variants from massive paired-end sequencing data. Genome Biology 2009; 10 (2): R23. DOI: 10.1186/gb-2009-10-2-r23.
24. Zhang Z, Schwartz S, et al,. A greedy algorithm for aligning DNA sequences. Journal of Computational Biology 2000; 7: 203-214.
25. Olston C, Reed B, et al,. Pig latin: a not-so-foreign language for data processing. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. ACM: New York, NY, USA, 2008; 1099-1110.
26. Pike R, Dorward S, et al,. Interpreting the data: parallel analysis with Sawzall. Scientific Programming Journal; 13 (4): 227-298.
27. Hadoop, 2007. Available from: http://hadoop.apache.org/.
28. Cascading, 2010. Available from: http://www.cascading.org/.
29. Ludäscher B, Altintas I, et al,. Scientific workflow management and the kepler system. Concurrency and Computation: Practice and Experience 2006; 18 (10): 1039-1065.
30. Oozie, 2010. Available from: http://yahoo.github.com/oozie/.
31. Meijer E, Beckman B, et al,. LINQ: reconciling object, relations and XML in the.NET framework. In Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, SIGMOD '06. ACM: New York, NY, USA, 2006; 706-706.
32. Chaiken R, Jenkins B, et al,. Scope: easy and efficient parallel processing of massive data sets. Proceedings of the VLDB Endowment 2008; 1: 1265-1276.
33. Yu J, Buyya R,. A taxonomy of scientific workflow systems for grid computing. SIGMOD Record 2005; 34: 44-49.
34. Thain D, Moretti C, et al,. Chirp: a practical global filesystem for cluster and grid computing. Journal of Grid Computing 2009; 7 (1): 51-72.