A system architecture for running big data workflows in the cloud

Proceedings - 2014 IEEE International Conference on Services Computing, SCC 2014

Volumn , Issue , 2014, Pages 51-58

  Kashlev, Andrey ^a   Lu, Shiyong ^a  

^a Wayne State University   (United States)

Author keywords

Big data; Cloud; Component; Scientific workflow

Indexed keywords

AUTOMOTIVE INDUSTRY; BIG DATA; CLOUDS; COMPUTER ARCHITECTURE; DIGITAL STORAGE; NETWORK ARCHITECTURE; PLATFORM AS A SERVICE (PAAS); WORK SIMPLIFICATION;

CLOUD INFRASTRUCTURES; COMPONENT; GENERIC IMPLEMENTATION; PROPOSED ARCHITECTURES; RESEARCH AND DEVELOPMENT; SCIENTIFIC WORKFLOW MANAGEMENTS; SCIENTIFIC WORKFLOWS; SYSTEM ARCHITECTURES;

INFORMATION MANAGEMENT;

EID: 84919703357     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SCC.2014.16     Document Type: Conference Paper

References (23)

1. L.J. Zhang, "Editorial: Quality-driven service and workflow management," IEEE Transactions on Services Computing (TSC), vol. 4, no. 2, pp. 84, 2011.
2. A. Tsalgatidou, et al., "Developing scientific workflows from heterogeneous services," SIGMOD Record, vol. 35, no. 2, pp. 22-28, 2006.
3. C. Hoffa, et al., "On the use of cloud computing for scientific workflows," eScience, pp. 640-645, 2008.
4. M. Abouelhoda, S. Issa, M. Ghanem, "Tavaxy: Integrating Taverna and Galaxy workflows with cloud computing support," BMC Bioinformatics, vol. 13, no. 77, pp. 1-19, 2012.
5. V. C. Emeakaroha et al., "Managing and optimizing bioinformatics workflows for data analysis in clouds," J. Grid Comput., vol. 11, no. 3, pp. 407-428, 2013.
6. J.S. Vöckler, et al., "Experiences using Cloud computing for a scientific workflow application," in Proc. of the 2nd International. Workshop on Scientific Cloud Computing, pp. 15-24, 2011.
7. D. de Oliveira et al., "SciCumulus: A lightweight cloud middleware to explore many task computing paradigm in scientific workflows," IEEE International Conference on Cloud Comput. (CLOUD), pp. 378-385, 2010.
8. Z. Wu et al., "A market-oriented hierarchical scheduling strategy in cloud workflow systems," Journal of Supercomputing, vol. 63, no. 1, pp. 256-293, 2013.
9. Y. Zhao et al., "Opportunities and challenges for running scientific workflows on the Cloud," International Conference on Cyber-Enabled Distr. Computing and Knowledge Discovery (CyberC), pp.455-462, 2011.
10. FutureGrid portal, http://www.futuregrid.org.
11. Amazon EC2 Cloud, http://aws.amazon.com/ec2/.
12. OpenStack Cloud Software, http://www.openstack.org.
13. The OPM Provenance Model(OPM),http://www.openprovenance.org.
14. The OpenXC Platform, http://openxcplatform.com.
15. H. Topcuoglu, S. Hariri, and M. Wu, "Performance-effective and low-complexity task scheduling for heterogeneous computing," IEEE Trans. on Parallel Distrib. Systems, vol. 13, no. 3, pp. 260-274, 2002.
16. E. Deelman et al., The cost of doing science on the Cloud: the Montage example. In Proc. of the 2008 ACM/IEEE conf. on Supercomputing (SC), pp. 50:1-50:12, 2008.
17. A. Iosup et al., "Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing," IEEE Trans. on Parallel and Distributed Syst., vol 22, no. 6, pp. 931-945, 2011.
18. E. Deelman, "Grids and Clouds: Making workflow applications work in heterogeneous environments," International J. on High Perf. Comput. Applications (IJHPCA), vol. 24, no. 3, pp. 284-298, 2010.
19. J. Wang, I. Altintas, "Early cloud experiences with the Kepler scientific workflow system," In Proc. of the Int'l. Conf. on Computational Science (ICCS), pp.1630-1634, 2012.
20. S. Pandey, D. Karunamoorthy, R. Buyya, "Workflow Engine for Clouds," in Cloud Computing: Principles and Paradigms, John Wiley &Sons Inc., pp.321-344, 2011.
21. D. Yuan, et al., "A cost-effective strategy for intermediate data storage in scientific cloud workflow systems," IEEE Intern. Sympos. on Parallel and Distr. Processing (IPDPS), pp. 1-12, 2010.
22. D. Williams, "The Arbitron national in-car study," Arbitron Inc., 2009.
23. A. Kashlev, S. Lu, A. Chebotko, "A System Architecture for Running Big Data Workflows in the Cloud," Technical Report TR-BIGDATA-02-2014-KL, Wayne State University, February 2014, http://www.cs.wayne.edu/andrey/papers/TR-BIGDATA-02-2014-KL.pdf.