Opportunities and challenges in running scientific workflows on the cloud

Proceedings - 2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2011

Volumn , Issue , 2011, Pages 455-462

  Zhao, Yong ^a   Fei, Xubo ^b   Raicu, Ioan ^c   Lu, Shiyong ^b  

^a UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b Wayne State University   (United States)

^c Illinois Institute of Technology   (United States)

Author keywords

Cloud computing; Cloud workflow; Data Intensive Computing; Scientific Workflow

Indexed keywords

BUSINESS APPLICATIONS; COMPUTING RESOURCE; DATA-INTENSIVE COMPUTING; RESEARCH OPPORTUNITIES; SCIENTIFIC WORKFLOWS; WEB APPLICATION; WORK-FLOWS;

INFORMATION MANAGEMENT;

CLOUD COMPUTING;

EID: 83455236527     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CyberC.2011.80     Document Type: Conference Paper

References (31)

1. http://www-03.ibm.com/press/us/en/pressrelease/22613.wss
2. http://www.microsoft.com/azure/default.mspx
3. "Introduction to Amazon elastic MapReduce," available from http://awsmedia.s3.amazonaws.com/pdf/introduction-to-amazon-elastic-mapreduce. pdf.
4. I. Foster, Y. Zhao, I. Raicu, S. Lu. "Cloud Computing and Grid Computing 360-Degree Compared", IEEE Grid Computing Environments (GCE08) 2008, co-located with IEEE/ACM Supercomputing 2008.
5. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica and M. Zaharia, Above the Clouds: A Berkeley View of Cloud Computing, EECS Department, University of California, Berkeley, Technical Report No. UCB/EECS-2009-28, February 10, 2009.
6. http://www.psc.edu/general/software/packages/genbank/,2010
7. R. Chaiken, B. Jenkins, P.-Å. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. in Proc. of the 2008 VDLB Conference (VLDB'08).
8. A. Chebotko, S. Lu, S. Chang, F. Fotouhi, and P. Yang, "Secure Abstraction Views for Scientific Workflow Provenance Querying", IEEE Transactions on Services Computing, 3(4), pp.322-337, 2010.
9. Jeffrey Dean, Sanjay Ghemawat: MapReduce: simplified data processing on large clusters. OSDI 2004: 137-149.
10. Xubo Fei, Shiyong Lu, and Cui Lin: A MapReduce-Enabled Scientific Workflow Composition Framework, ICWS 2009: 663-670.
11. Foster, I., Voeckler, J., Wilde, M. and Zhao, Y., Chimera: A Virtual Data System for Representing, Querying, and Automating Data Derivation, In 14th Conference on Scientific and Statistical Database Management, 2002.
12. The Fourth Paradigm: Data-Intensive Scientific Discovery, Edited by Tony Hey, Stewart Tansley, and Kristin Tolle. Microsoft Research.
13. C. Hoffa, G. Mehta, T. Freeman, E. Deelman, K. Keahey, B. Berriman, J. Good, "On the Use of Cloud Computing for Scientific Workflows", 3rd International Workshop on Scientific Workflows and Business Workflow Standards in e-Science (SWBES), 10 December 2008 in Indianapolis, Indiana, USA
14. D. Hull, K. Wolstencroft, R. Stevens, C. Goble, M. Pocock, P. Li, and T. Oinn, "Taverna: a tool for building and running workflows of services.," Nucleic Acids Research, vol. 34, iss. Web Server issue, pp. 729-732, 2006.
15. Keahey K., T. Freeman. Science Clouds: Early Experiences in Cloud Computing for Scientific Applications, Cloud Computing and Its Applications 2008 (CCA-08), Chicago, IL. October 2008
16. C. Lin, S. Lu, X. Fei, A. Chebotko, D. Pai, Z. Lai, F. Fotouhi, and J. Hua, "A Reference Architecture for Scientific Workflow Management Systems and the VIEW SOA Solution", IEEE Transactions on Services Computing (TSC), 2(1), pp.79-92, 2009.
17. Cui Lin, Shiyong Lu, Zhaoqiang Lai, Artem Chebotko, Xubo Fei, Jing Hua, and Farshad Fotouhi, "Service-Oriented Architecture for VIEW: a Visual Scientific Workflow Management System", In Proc. of the IEEE 2008 International Conference on Services Computing (SCC), Honolulu, Hawaii, USA, July 2008, pp.335-342.
18. Shiyong Lu and Jia Zhang. "Collaborative Scientific Workflows", IEEE International Conference on Web Services (ICWS), pp.527-534, Los Angeles, CA, 2009.
19. D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, D. Zagorodnov, The Eucalyptus Open-source Cloud-computing System, in Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid, Shanghai, China.
20. Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan: Interpreting the data: Parallel analysis with Sawzall. Scientific Programming 13(4): 277-298 (2005).
21. Plale, B., D. Gannon, J. Brotzge, K. Droegemeier, J. Kurose, D. McLaughlin, R. Wilhelmson, S. Graves, M. Ramamurthy, R.D. Clark, S. Yalda, D.A. Reed, E. Joseph, V. Chandrasekar, CASA and LEAD: Adaptive Cyberinfrastructure for Real-Time Multiscale Weather Forecasting, Computer special issue on System-Level Science, IEEE Computer Science Press, Vol. 39, No. 11, pp. 56-63. Nov 2006.
22. Ioan Raicu, Ian Foster, Yong Zhao. "Many-Task Computing for Grids and Supercomputers", IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08), 2008, co-located with IEEE/ACM Supercomputing 2008.
23. Ioan Raicu, Yong Zhao, Catalin Dumitrescu, Ian Foster, Mike Wilde. "Falkon: a Fast and Light-weight tasK executiON framework, IEEE/ACM SuperComputing 2007.
24. Ioan Raicu, Yong Zhao, Ian Foster, Alex Szalay. "Accelerating Large-scale Data Exploration through Data Diffusion", International Workshop on Data-Aware Distributed Computing 2008, co-locate with ACM/IEEE International Symposium High Performance Distributed Computing (HPDC) 2008.
25. W. Tan, K. Chard, D. Sulakhe, R. K. Madduri, I. T. Foster, S. S.-Reyes, C. A. Goble: Scientific Workflows as Services in caGrid: A Taverna and gRAVI Approach. ICWS 2009: 413-420.
26. Edward Walker, Weijia Xu, Vinoth Chandar, Composing and executing parallel data-flow graphs with shell pipes, Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science 2009, Portland, Oregon, November 16-16, 2009.
27. M. Wilde, I. Foster, K. Iskra, P. Beckman, Z. Zhang, Allan Espinosa, Mihael Hategan, Ben Clifford, Ioan Raicu. "Parallel Scripting for Applications at the Petascale and Beyond", IEEE Computer Nov. 2009 Special Issue on Extreme Scale Computing, 2009.
28. Zijiang Yang, Shiyong Lu, Ping Yang: Itinerary-Based Access Control for Mobile Tasks in Scientific Workflows. Ubisafe: 506-511
29. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey, DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language, Symposium on Operating System Design and Implementation (OSDI), San Diego, CA, 2008.
30. Y. Zhao, J. Dobson, I. Foster, L. Moreau, M. Wilde, A Notation and System for Expressing and Executing Cleanly Typed Workflows on Messy Scientific Data, SIGMOD Record, Volume 34, Number 3, September 2005
31. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. v. Laszewski, I. Raicu, T. Stef-Praun, M. Wilde. "Swift: Fast, Reliable, Loosely Coupled Parallel Computation", IEEE Workshop on Scientific Workflows 2007.