Provenance for MapReduce-based data-intensive workflows

WORKS'11 - Proceedings of the 6th Workshop on Workflows in Support of Large-Scale Science, Co-located with SC'11

Volumn , Issue , 2011, Pages 21-29

  Crawl, Daniel ^a   Wang, Jianwu ^a   Altintas, Ilkay ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

MapReduce; Provenance; Scientific workflows

Indexed keywords

LARGE DATASETS; MAP-REDUCE; MYSQL CLUSTERS; PROGRAMMING INTERFACE; PROGRAMMING MODELS; PROVENANCE; SCIENTIFIC APPLICATIONS; SCIENTIFIC WORKFLOWS; WORK-FLOWS; WORKFLOW EXECUTION;

CLUSTER COMPUTING; COMPUTER PROGRAMMING; MODELS;

DATA HANDLING;

EID: 84863275579     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2110497.2110501     Document Type: Conference Paper

References (21)

1. I. Altintas, O. Barney, and E. Jaeger-Frank. Provenance Collection Support in the Kepler Scientific Workflow System. In L. Moreau and I. Foster, editors, Provenance and Annotation of Data (IPAW 2006, Revised Selected Papers), volume 4145 of Lecture Notes in Computer Science, pages 118-132. Springer Berlin / Heidelberg, 2006. (Pubitemid 44577314)
2. S. F. Altschul, W. Gish, W. Miller, E. W. Myers, and D. J. Lipman. Basic Local Alignment Search Tool. Journal of Molecular Biology, 215(3):403 - 410, 1990.
3. B. Chapman, G. Jost, R. van der Pas, and D. Kuck. Using OpenMP: Portable Shared Memory Parallel Programming. The MIT Press, Cambridge, MA, USA, 2007.
4. D. Crawl and I. Altintas. A Provenance-Based Fault Tolerance Mechanism for Scientific Workflows. In J. Freire, D. Koop, and L. Moreau, editors, Provenance and Annotation of Data and Processes (IPAW 2008, Revised Selected Papers), volume 5272 of Lecture Notes in Computer Science, pages 152-159. Springer, 2008.
5. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51:107-113, January 2008.
6. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox. Twister: A Runtime for Iterative MapReduce. In The First International Workshop on MapReduce and its Applications (MAPREDUCE'10) - HPDC2010. 2010, 2010.
7. X. Fei, S. Lu, and C. Lin. A MapReduce-Enabled Scientific Workflow Composition Framework. In Web Services, 2009. ICWS 2009. IEEE International Conference on, pages 663-670, July 2009.
8. C. Franke, S. Morin, A. Chebotko, J. Abraham, and P. Brazier. Distributed Semantic Web Data Management in HBase and MySQL Cluster. In Proc. of the 4th IEEE International Conference on Cloud Computing (CLOUD'11), pages 105-112, Washington DC, USA, July 2011.
9. A. Goderis, C. Brooks, I. Altintas, E. A. Lee, and C. A. Goble. Heterogeneous Composition of Models of Computation. Future Generation Computer Systems, 25(5):552-560, 2009.
10. W. Gropp, E. Lusk, and A. Skjellum. Using MPI: Portable Parallel Programming with the Message Passing Interface. Scientific And Engineering Computation Series. MIT Press, Cambridge, MA, USA, 2nd edition edition, 1999.
11. Y. Gu and R. Grossman. Sector and Sphere: The Design and Implementation of a High Performance Data Cloud. Philosophical Transactions of the Royal Society A, 367(1897):2429-2445, June 2009.
12. R. Ikeda, H. Park, and J. Widom. Provenance for Generalized Map and Reduce Workflows. In Proceedings of CIDR'2011, pages 273-283, 2011.
13. M. Isard and Y. Yu. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In SIGMOD '09: Proceedings of the 35th SIGMOD international conference on Management of data, pages 987-994, New York, NY, USA, 2009. ACM.
14. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger-Frank, M. Jones, E. Lee, J. Tao, and Y. Zhao. Scientific Workflow Management and the Kepler System. Concurrency and Computation: Practice and Experience, Special Issue on Scientific Workflows, 18(10):1039-1065, 2006. (Pubitemid 44258497)
15. T. G. Mattson, B. A. Sanders, and B. Massingill. Patterns for Parallel Programming. Addison-Wesley, 2005.
16. L. Moreau, B. Clifford, J. Freire, J. Futrelle, Y. Gil, P. Groth, N. Kwasnikowska, S. Miles, P. Missier, J. Myers, B. Plale, Y. Simmhan, E. Stephan, and J. Van den Bussche. The Open Provenance Model core specification (v1.1). Future Generation Computer Systems (FGCS), 27(doi:10.1016/j.future.2010.07.005): 734-756, 2011.
17. C. Moretti, H. Bui, K. Hollingsworth, B. Rich, P. Flynn, and D. Thain. All-Pairs: An Abstraction for Data-Intensive Computing on Campus Grids. IEEE Transactions on Parallel and Distributed Systems, 21:33-46, 2010.
18. J. Wang, I. Altintas, C. Berkley, L. Gilbert, and M. B. Jones. A high-level distributed execution framework for scientific workflows. In IEEE International Conference on eScience, pages 634-639, Los Alamitos, CA, USA, 2008. IEEE Computer Society.
19. J. Wang, D. Crawl, and I. Altintas. Kepler + Hadoop: A General Architecture Facilitating Data-Intensive Applications in Scientific Workflow Systems. In WORKS '09: Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, pages 1-8, Portland, Oregon, 2009. ACM New York, NY, USA.
20. J. Wang, P. Korambath, and I. Altintas. A Physical and Virtual Compute Cluster Resource Load Balancing Approach to Data-Parallel Scientific Workflow Scheduling. In IEEE 2011 Fifth International Workshop on Scientific Workflows (SWF 2011), at 2011 Congress on Services (Services 2011), Washington, DC, USA, July 2011.
21. C. Zhang and H. De Sterck. CloudWF: A Computational Workflow System for Clouds Based on Hadoop. In M. Jaatun, G. Zhao, and C. Rong, editors, Cloud Computing, volume 5931 of Lecture Notes in Computer Science, pages 393-404. Springer Berlin / Heidelberg, 2009.