Provenance for generalized map and reduce workflows

CIDR 2011 - 5th Biennial Conference on Innovative Data Systems Research, Conference Proceedings

Volumn , Issue , 2011, Pages 273-283

  Ikeda, Robert ^a   Park, Hyunjung ^a   Widom, Jennifer ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACYCLIC GRAPHS; DATA PROVENANCE; GENERALIZED MAP; INPUT DATAS; PARALLEL EXECUTIONS; PROTOTYPE SYSTEM; SYSTEM USE; USER INTERVENTION; WORK-FLOWS; WRAPPER-BASED APPROACH;

FAULT TOLERANCE;

EID: 80053549450     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. The Open Provenance Model-Core Specification (v1.1). Dec. 2009. http://eprints.ecs.soton.ac.uk/18332/.
2. M. K. Anand, S. Bowers, and B. Ludäscher. Techniques for efficiently querying scientific workflow provenance graphs. In EDBT, 2010.
3. Apache. Hadoop. http://hadoop.apache.org/.
4. Apache. Hadoop cluster setup. http://hadoop.apache.org/common/docs/r0.21. 0/cluster setup.html.
5. Apache. Mapreduce tutorial. http://hadoop.apache.org/mapreduce/docs/r0. 21.0/mapred tutorial.html.
6. Apache. Pigmix benchmarks. http://wiki.apache.org/pig/PigMix.
7. O. Biton, S. Cohen-Boulakia, S. B. Davidson, and C. S. Hara. Querying and managing provenance through user views in scientific workflows. In ICDE, 2008.
8. R. Bose and J. Frew. Lineage retrieval for scientific data processing: a survey. ACM Comput. Surv., 37(1), 2005.
9. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: Easy, efficient data-parallel pipelines. In PLDI, 2010.
10. A. P. Chapman, H. V. Jagadish, and P. Ramanan. Efficient provenance storage. In SIGMOD, 2008.
11. J. Cheney, L. Chiticariu, and W.-C. Tan. Provenance in databases: Why, how, and where. Foundations and Trends in Databases, 1(4), 2009.
12. Y. Cui and J. Widom. Lineage tracing for general data warehouse transformations. The VLDB Journal, 12(1), 2003.
13. Y. Cui, J. Widom, and J. L. Wiener. Tracing the lineage of view data in a warehousing environment. ACM TODS, 25(2), 2000.
14. S. B. Davidson and J. Freire. Provenance and scientific workflows: challenges and opportunities. In SIGMOD, 2008.
15. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
16. T. J. Green, G. Karvounarakis, Z. G. Ives, and V. Tannen. Update exchange with mappings and provenance. In VLDB, 2007.
17. T. Heinis and G. Alonso. Efficient lineage tracking for scientific workflows. In SIGMOD, 2008.
18. R. Ikeda, S. Salihoglu, and J. Widom. Provenance-based refresh in data-oriented workflows. Technical report, Stanford University InfoLab, March 2010.
19. P. Missier, K. Belhajjame, J. Zhao, M. Roos, and C. Goble. Data lineage model for Taverna workflows with lightweight annotation requirements. In IPAW, 2008.
20. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In SIGMOD, 2008.
21. O. O'Malley and A. C. Murthy. Winning a 60 second dash with a yellow elephant. http://l.yimg.com/a/i/ydn/blogs/hadoop/yahoo2009.pdf, 2009.
22. S. Sarawagi. Information extraction. Found. Trends databases, 1:261-377, March 2008.
23. Y. L. Simmhan, B. Plale, and D. Gannon. A survey of data provenance in e-science. SIGMOD Rec., 34(3), 2005.
24. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: A warehousing solution over a map-reduce framework. In VLDB, 2009.
25. K. Weil. hadoop-lzo. http://www.github.com/kevinweil/hadoop-lzo/.
26. A. Woodruff and M. Stonebraker. Supporting fine-grained data lineage in a database visualization environment. In ICDE, 1997.
27. H.-C. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker. Map-Reduce-Merge: Simplified relational data processing on large clusters. In SIGMOD, 2007.