Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-Intensive Science

Proceedings of DataCloud 2016: 7th International Workshop on Data-Intensive Computing in the Clouds - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis

Volumn , Issue , 2017, Pages 1-8

  Filgueira, Rosa ^a   Silva, Rafael Ferreira Da ^b   Krause, Amrey ^c   Deelman, Ewa ^b   Atkinson, Malcolm ^c  

^a THE LYELL CENTRE   (United Kingdom)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Data Intensive science; Deployment and reusability of execution environments; scientific workows; stream based system

Indexed keywords

APPLICATION PROGRAMS; COST ENGINEERING; DIGITAL STORAGE; DISTRIBUTED COMPUTER SYSTEMS; MINERALOGY; OPEN SOURCE SOFTWARE; OPEN SYSTEMS; REUSABILITY; WORK SIMPLIFICATION;

COMPUTATIONAL PLATFORMS; DATA INTENSIVE SCIENCE; DATA-INTENSIVE APPLICATION; EXECUTION ENVIRONMENTS; HETEROGENEOUS RESOURCES; SCIENTIFIC WORKOWS; STREAM-BASED; WORKFLOW MANAGEMENT SYSTEMS;

INFORMATION MANAGEMENT;

EID: 85015852401     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/DataCloud.2016.004     Document Type: Conference Paper

References (35)

1. M. Albrecht, P. Donnelly, et al. Makeflow: A portable abstraction for data intensive computing on clusters, clouds, grids. In 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies, page 1. ACM, 2012.
2. M. D. Assuncao, R. N. Calheiros, et al. Big data computing and clouds: Trends and future directions. Journal of Parallel and Distributed Computing, 79-80:3-15, 2015.
3. Asterism: Research Object. https://scitech. isi. edu/ro/asterism.
4. S. Azarnoosh, M. Rynge, et al. Introducing precip: an api for managing repeatable experiments in the cloud. In 2013 IEEE 5th International Conference on Cloud Computing Technology and Science, volume 2 of CloudCom, pages 19-26, 2013.
5. M. Beyreuther, R. Barsch, et al. Obspy: A python toolbox for seismology. Seismological Research Letters, 81(3):530-533, 2010.
6. C. Boettiger. An introduction to docker for reproducible research. SIGOPS Oper. Syst. Rev., 49(1):71-79, Jan. 2015.
7. R. Chard, K. Chard, et al. Cost-aware cloud provisioning. In 2015 IEEE 11th International Conference on e-Science (e-Science), pages 136-144. IEEE, 2015.
8. E. Deelman, K. Vahi, et al. Pegasus, a workflow management system for science automation. Future Generation Computer Systems, 46:17-35, 2015.
9. E. Deelman, K. Vahi, et al. Pegasus in the cloud: Science automation through workflow technologies. IEEE Internet Computing, 20(1):70-76, 2016.
10. Docker. https://www.docker. com.
11. Docker hub. https://hub. docker.com/.
12. R. Evans. Apache storm, a hands on tutorial. In 2015 IEEE International Conference on Cloud Engineering (IC2E), pages 2-2, 2015.
13. T. Fahringer, R. Prodan, et al. Askalon: A development and grid computing environment for scientific workflows. In Workflows for e-Science, pages 450-471. 2007.
14. R. Ferreira da Silva, W. Chen, et al. Community resources for enabling and evaluating research on scientific workflows. In 2014 IEEE 10th International Conference on e-Science, eScience'14, pages 177-184, 2014.
15. R. Ferreira da Silva, E. Deelman, et al. Automating environmental computing applications with scientific workflows. In Environmental Computing Workshop (ECW'16), 2016.
16. R. Filgueira, A. Krause, et al. dispel4py: An agile framework for data-intensive escience. In Proc. 11th IEEE eScience Conf., 2015.
17. R. Filgueira, A. Krause, et al. dispel4py: A python framework for data-intensive scientific computing. International Journal of High Performance Computing Applications (IJHPCA), pages 9-16, 2016.
18. J. Frey. Condor dagman: Handling inter-job dependencies. University of Wisconsin, Dept. of Computer Science, Tech. Rep, 2002.
19. W. Gerlach, W. Tang, A. Wilke. Container orchestration for scientific workflows. In 2015 IEEE International Conference on Cloud Engineering, 06/2015 2015.
20. P. Hunt, M. Konar, et al. Zookeeper: Wait-free coordination for internet-scale systems. In USENIX Annual Technical Conference, volume 8, page 9, 2010.
21. A. Jain, S. P. Ong, et al. Fireworks: a dynamic workflow system designed for high-throughput applications. Concurrency and Computation: Practice and Experience, 27(17):5037-5059, 2015.
22. F. Lordan, E. Tejedor, J. Ejarque, R. Rafanell, J. Aalvarez, F. Marozzo, D. Lezzi, R. Sirvent, D. Talia, R. Badia. Servicess: An interoperable programming framework for the cloud. Journal of Grid Computing, 12(1):67-91, 2014.
23. J. Mambretti, J. Chen, et al. Next generation clouds, the chameleon cloud testbed, software defined networking (sdn). In 2015 International Conference on Cloud Computing Research and Innovation (ICCCRI), pages 73-79, 2015.
24. M. Nardelli. A framework for data stream applications in a distributed cloud. In Proceedings of the 8th ZEUS Workshop, Vienna, Austria, January 27-28, 2016., pages 56-63, 2016.
25. Nextflow. http://www.nextflow. io/index. html.
26. Open mpi: Open source high performance computing. https://www.open-mpi.org.
27. Pegasus and LIGO. http://pegasus. isi. edu/2016/02/11/pegasus-powers-ligo-gravitational-waves-detection-analysis.
28. I. J. Taylor, E. Deelman, et al. Workflows for e-Science: scientific workflows for grids. 2014.
29. D. Thain, T. Tannenbaum, et al. Distributed computing in practice: the condor experience. Concurrency and computation: practice and experience, 17(2-4):323-356, 2005.
30. USArray TA. http://ds. iris. edu/ds/nodes/dmc/earthscope/usarray/US-TA-operational.
31. VERCE. http://www.verce. eu.
32. K. Vukojevic-Haupt, F. Haupt, et al. Bootstrapping complex workflow middleware systems into the cloud. In 2015 IEEE 11th International Conference on e-Science (e-Science), pages 126-135. IEEE, 2015.
33. C. Walter. Kryder's Law: The doubling of processor speed every 18 months is a snail's pace compared with rising hard-disk capacity, Mark Kryder plans to squeeze in even more bits. Scientific American, pages 32-33, August 2005.
34. K. Wolstencroft, R. Haines, et al. The taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud. Nucleic acids research, 41(Web Server issue):W557-W561, 2013.
35. C. Zheng and D. Thain. Integrating containers into workflows: A case study using makeflow, work queue, docker. In 8th International Workshop on Virtualization Technologies in Distributed Computing, pages 31-38. ACM, 2015.