Kepler + Hadoop: A general architecture facilitating data-intensive applications in scientific workflow systems

Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009

Volumn , Issue , 2009, Pages

  Wang, Jianwu ^a   Crawl, Daniel ^a   Altintas, Ilkay ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Distributed computing; Easy to use; Hadoop; Kepler; MapReduce; Parallel computing; Scientific workflow

Indexed keywords

COMPONENT REUSE; COMPUTING RESOURCE; DATA PARTITIONING; DATA-INTENSIVE APPLICATION; DE FACTO; DISTRIBUTED COMPUTING; DOMAIN SPECIFIC; LARGE DATASETS; LOAD-BALANCING; OPEN SOURCE PROJECTS; PARALLEL COMPUTING; PARALLEL PROCESSING; PEGASUS; PROGRAMMING MODELS; SCIENTIFIC APPLICATIONS; SCIENTIFIC WORKFLOWS;

DISTRIBUTED COMPUTER SYSTEMS; FAULT TOLERANCE; FAULT TOLERANT COMPUTER SYSTEMS; GRAPHICAL USER INTERFACES; PARALLEL ARCHITECTURES; PARALLEL PROGRAMMING; QUALITY ASSURANCE; WORK SIMPLIFICATION;

MANAGEMENT;

EID: 74049123821     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1645164.1645176     Document Type: Conference Paper

References (20)

1. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI 2004), pages 137-150. USENIX Association, 2004.
2. Apache Hadoop Project. http://hadoop.apache.org/core/.
3. I. Gorton, P. Greenfield, A. Szalay, and R. Williams. Data-Intensive Computing in the 21st Century. In Computer, 41(4):30-32, Apr. 2008, doi:10.1109/MC.2008.122.
4. R. E. Bryant. Data-intensive Supercomputing: The Case for DISC. Technical Report CMU-CS-07-128, Carnegie Mellon University, 2007.
5. J. Ekanayake, S. Pallickara, and G. Fox. MapReduce for Data Intensive Scientific Analyses. In Proceedings of the 4th IEEE International Conference on eScience (e Science 2008), pages 277-284, 2008.
6. M. C. Schatz. Cloudburst: Highly Sensitive Read Mapping with MapReduce. In Bioinformatics 2009 25(11):1363-1369. Apr. 2009.
7. Q. Chen, L. Wang, and Z. Shang. MRGIS: A MapReduce-Enabled High Performance Workflow System for GIS. In Proceedings of Workshop SWBES08: Challenging Issues in Workflow Applications, the 4th IEEE International Conference on eScience (e Science 2008), pages 646-651, 2008.
8. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. B. Jones, E. A. Lee, J. Tao, and Y. Zhao. Scientific Workflow Management and the Kepler System. In Concurrency and Computation: Practice and Experience, 18(10):1039-1065, 2006.
9. P. R. Hosseini. Pattern Formation and Individual-Based Models: The Importance of Understanding Individual-Based Movement. In Ecological Modeling. 194(4): 357-371. doi:10.1016/j.ecolmodel.2005.10.041. 2006.
10. J. Wang, I. Altintas, P. R. Hosseini, D. Barseghian, D. Crawl, C. Berkley, and M. B. Jones. Accelerating Parameter Sweep Workflows by Utilizing Ad-hoc Network Computing Resources: an Ecological Example. In Proceedings of IEEE 2009 Third International Workshop on Scientific Workflows (SWF 2009), 2009 Congress on Services (Services 2009), pages 267-274, 2009.
11. X. Fei, S. Lu, and C. Lin. A MapReduce-Enabled Scientific Workflow Composition Framework. In Proceedings of 2009 IEEE International Conference on Web Services (ICWS 2009), pages 663-670, 2009.
12. D. J. Goodman. Introduction and Evaluation of Martlet: a Scientific Workflow Language for Abstracted Parallelisation. In Proceedings of the 16th International Conference on World Wide Web (WWW 2007), pages 983-992, 2007.
13. T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. R. Pocock, A. Wipat, and P. Li. Taverna: a tool for the composition and enactment of bioinformatics workflows. In Bioinformatics, 20(17), pages 3045-3054, Oxford University Press, London, UK, 2004.
14. I. Taylor, M. Shields, I. Wang, and O. Rana. Triana Applications within Grid Computing and Peer to Peer Environments. In Journal of Grid Computing, 1(2):199-217. Kluwer Academic Press, 2003.
15. E. Deelman, G. Mehta, G. Singh, M. Su, and K. Vahi. Pegasus: Mapping Large-Scale Workflows to Distributed Resources. In I. Taylor, E. Deelman, D. Gannon, and M. Shields, editors, Workflows for e-Science, pages 376-394. Springer, New York, Secaucus, NJ, USA, 2007.
16. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, and M. Wilde. Swift: Fast, Reliable, Loosely Coupled Parallel Computation. In Proceedings of 2007 IEEE Congress on Services (Services 2007), pages 199-206, 2007.
17. J. Qin and T. Fahringer. Advanced data flow support for scientific grid workflow applications. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC2007), Article No. 42. 2007.
18. D. Zinn, S. Bowers, S. Köhler, and B. Ludäscher. Parallelizing XML Processing Pipelines via MapReduce. In Special issue on Scientific Workflows Journal of Computer and System Sciences, 2010. Accepted for publication.
19. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A Distributed Storage System for Structured Data. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI 2006), pages 205-218. USENIX Association, 2006.
20. Apache HBase Project: http://hadoop.apache.org/hbase/.