Stubby: A transformation-based optimizer for MapReduce Workflows

Proceedings of the VLDB Endowment

Volumn 5, Issue 11, 2012, Pages 1196-1207

  Lim, Harold ^a   Herodotou, Herodotos ^a   Babu, Shivnath ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COST-BASED OPTIMIZATION; DESIRABLE FEATURES; EXECUTION PLANS; LARGE DATASETS; OPTIMIZERS; SEARCH ALGORITHMS; SPACE-BASED; TRANSFORMATION BASED;

EID: 84873155673     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/2350229.2350239     Document Type: Article

References (27)

1. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: Easy, Efficient Data-Parallel Pipelines. In PLDI, pages 363-375, 2010.
2. D. Chatziantoniou and K. A. Ross. Querying Multiple Features of Groups in Relational Databases. In VLDB, pages 295-306, 1996.
3. Cloudera: 7 Tips for Improving MapReduce Performance. cloudera. com/blog/2009/12/7-tips-for-improving-mapreduce-performance.
4. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, pages 137-150, 2004.
5. A. Gates, O. Natkovich, S. Chopra, P. Kamath, S. Narayanam, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava. Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience. PVLDB, 2(2):1414-1425, 2009.
6. G. Graefe and D. J. DeWitt. The EXODUS Optimizer Generator. In SIGMOD, pages 160-172, 1987.
7. Apache Hadoop. http://hadoop.apache.org/.
8. H. Herodotou and S. Babu. Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs. PVLDB, 4(11):1111-1122, 2011.
9. Apache Hive. http://hive.apache.org/.
10. E. Jahani, M. J. Cafarella, and C. Ŕe. Automatic Optimization for MapReduce Programs. PVLDB, 4(6):385-396, 2011.
11. R. Lee, T. Luo, Y. Huai, F. Wang, Y. He, and X. Zhang. YSmart: Yet Another SQL-to-MapReduce Translator. In ICDCS, pages 25-36, 2011.
12. H. Lim, H. Herodotou, and S. Babu. Stubby: A Transformation-based Optimizer for MapReduce Workflows. Technical report, Duke Computer Science, 2012. http://www.cs.duke.edu/starfish/files/stubby-technical-report.pdf.
13. T. Nykiel, M. Potamias, C. Mishra, G. Kollios, and N. Koudas. MRShare: Sharing Across Multiple Queries in MapReduce. PVLDB, 3(1):494-505, 2010.
14. C. Olston, B. Reed, A. Silberstein, and U. Srivastava. Automatic Optimization of Parallel Dataflow Programs. In USENIX Annual Technical Conference, pages 267-273, 2008.
15. Oozie: Workflow Engine for Hadoop. http://yahoo.github.com/oozie/.
16. L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank Citation Ranking: Bringing Order to the Web. Technical Report 1999-66, Stanford Info Lab, November 1999.
17. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. A Comparison of Approaches to Large-Scale Data Analysis. In SIGMOD, pages 165-178, 2009.
18. Apache Pig. http://pig.apache.org/.
19. A. Simitsis, P. Vassiliadis, and T. K. Sellis. Optimizing ETL Processes in Data Warehouses. In ICDE, pages 564-575, 2005.
20. TPC-H Benchmark Specification. http://www.tpc.org/tpch/.
21. P. Upadhyaya, Y. Kwon, and M. Balazinska. A Latency and Fault-Tolerance Optimizer for Online Parallel Query Plans. In SIGMOD ACM, pages 241-252. , 2011.
22. X. Wang, A. D. Sarma, C. Olston, and R. Burns. CoScan: Cooperative Scan Sharing in the Cloud. In SOCC, 2011.
23. S. Wu, F. Li, S. Mehrotra, and B. C. Ooi. Query Optimization for Massively Parallel Data Processing. In SOCC, 2011.
24. T. Ye and S. Kalyanaraman. A Recursive Random Search Algorithm for Large-Scale Network Parameter Configuration. SIGMETRICS, pages 196-205, 2003.
25. Y. Yu, P. K. Gunda, and M. Isard. Distributed Aggregation for Data-Parallel Computing: Interfaces and Implementations. In SOSP, pages 247-260, 2009.
26. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In OSDI, pages 1-14, 2008.
27. J. Zhou, P.-A. Larson, and R. Chaiken. Incorporating Partitioning and Parallel Plans into the SCOPE Optimizer. In ICDE, pages 1060-1071, 2010..