Weaver: Integrating distributed computing abstractions into scientific workflows using Python

HPDC 2010 - Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Volumn , Issue , 2010, Pages 636-643

  Bui, Peter ^a   Yu, Li ^a   Thain, Douglas ^a  

^a UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BARRIERS TO ADOPTION; DATA SETS; DIRECTED ACYCLIC GRAPHS; DISTRIBUTED COMPUTING; PROGRAMMING MODELS; PYTHON PROGRAMMING LANGUAGE; SCIENTIFIC RESEARCHES; SCIENTIFIC WORKFLOWS; WORK-FLOWS; WORKFLOW LANGUAGE;

ABSTRACTING; COMPUTER PROGRAMMING; COMPUTER SOFTWARE; HIGH LEVEL LANGUAGES; INTEGRATION;

MANAGEMENT;

EID: 78650006401     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1851476.1851570     Document Type: Conference Paper

References (13)

1. The directed acyclic graph manager. http://www.cs.wisc.edu/condor/dagman, 2002.
2. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In Operating Systems Design and Implementation, 2004.
3. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, B. Berriman, J. Good, A. Laity, J. Jacob, and D. Katz. Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming Journal, 13(3), 2005.
4. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data parallel programs from sequential building blocks. In Proceedings of EuroSys, March 2007.
5. M. Isard and Y. Yu. Distributed data-parallel computing using a high-level programming language. In SIGMOD '09: Proceedings of the 35th SIGMOD international conference on Management of data, pages 987-994, New York, NY, USA, 2009. ACM.
6. C. Moretti, J. Bulosan, D. Thain, and P. Flynn. All-Pairs: An Abstraction for Data Intensive Cloud Computing. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1-11, 2008.
7. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1099-1110, New York, NY, USA, 2008. ACM.
8. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with sawzall. Scientific Programming Journal, 13(4):227-298.
9. Python Programming Language. http://www.python.org/, 2010.
10. SQLAlchemy. http://sqlalchemy.org/, 2010.
11. D. Thain, T. Tannenbaum, and M. Livny. Condor and the grid. In F. Berman, G. Fox, and T. Hey, editors, Grid Computing: Making the Global Infrastructure a Reality. John Wiley, 2003.
12. L. Yu, C. Moretti, A. Thrasher, S. Emrich, K. Judd, and D. Thain. Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions. to appear in Journal of Cluster Computing, 2010.
13. Y. Zhao, J. Dobson, L. Moreau, I. Foster, and M. Wilde. A notation and system for expressing and executing cleanly typed workflows on messy scientific data. In SIGMOD, 2005.