Profiling, what-if analysis, and costbased optimization of mapreduce programs

Proceedings of the VLDB Endowment

Volumn 4, Issue 11, 2011, Pages 1111-1122

  Herodotou, Herodotos ^a   Babu, Shivnath ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPLICATION PROGRAMS; C++ (PROGRAMMING LANGUAGE); COST ESTIMATING; DATA HANDLING; DATABASE SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS;

APPLICATIONS DOMAINS; COST-BASED OPTIMIZATION; DATA ANALYTICS; LANGUAGE PROCESSING; MAP-REDUCE; NATURAL LANGUAGES; SOCIAL NETWORK ANALYSIS; TEXT ANALYSIS; WEB GRAPHS; WHAT-IF ANALYSIS;

MAPREDUCE;

EID: 82155174846     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/3402707.3402746     Document Type: Conference Paper

References (28)

1. A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi, A. Rasin, and A. Silberschatz. HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads. PVLDB, 2:922-933, 2009.
2. F. Afrati and J. D. Ullman. Optimizing Joins in a MapReduce Environment. In EDBT, pages 99-110, 2010.
3. S. Babu. Towards Automatic Optimization of MapReduce Programs. In SoCC, pages 137-142, 2010.
4. S. Blanas, J. M. Patel, V. Ercegovac, J. Rao, E. J. Shekita, and Y. Tian. A Comparison of Join Algorithms for Log Processing in MapReduce. In SIGMOD, pages 975-986, 2010.
5. A Dynamic Instrumentation Tool for Java. kenai.com/projects/btrace.
6. Y. Bu, B. Howe, M. Balazinska, and M. Ernst. HaLoop: Efficient Iterative Data Processing on Large Clusters. PVLDB, 3:285-296, 2010.
7. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1):107-113, 2008.
8. J. Dittrich, J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing). PVLDB, 3:515-529, 2010.
9. A. Gates. Comparing Pig Latin and SQL for Constructing Data Processing Pipelines. http://tinyurl.com/4ek25of.
10. How to dynamically assign reducers to a Hadoop Job at runtime. http://tinyurl.com/6eqqadl.
11. Hadoop Performance Monitoring UI. http://code.google.com/p/hadoop-toolkit/wiki/HadoopPerformanceMonitoring.
12. Vaidya. hadoop.apache.org/mapreduce/docs/r0.21.0/vaidya.html.
13. H. Herodotou. Hadoop Performance Models. Technical Report CS-2011-05, Duke Computer Science, 2011. http://www.cs.duke.edu/starfish/files/hadoop-models.pdf.
14. H. Herodotou, F. Dong, and S. Babu. No One (Cluster) Size Fits All: Automatic Cluster Sizing for Data-intensive Analytics. Technical Report CS-2011-06, Duke Computer Science, 2011. http://www.cs.duke.edu/starfish/files/cluster-sizing.pdf.
15. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A Self-tuning System for Big Data Analytics. In CIDR, pages 261-272, 2011.
16. E. Jahani, M. J. Cafarella, and C. Ré. Automatic Optimization of MapReduce Programs. PVLDB, 4:386-396, 2011.
17. D. Jiang, B. C. Ooi, L. Shi, and S. Wu. The Performance of MapReduce: An In-depth Study. PVLDB, 3:472-483, 2010.
18. Y. Kwon et al. Skew-Resistant Parallel Processing of Feature Extracting Scientific User-Defined Functions. In SoCC, pages 75-86, 2010.
19. J. Lin and C. Dyer. Data-Intensive Text Processing with MapReduce. Morgan and Claypool, 2010.
20. Hadoop MapReduce Tutorial. http://hadoop.apache.org/common/docs/current/mapredtutorial.html.
21. Mumak: Map-Reduce Simulator. https://issues.apache.org/jira/browse/MAPREDUCE-728.
22. T. Nykiel, M. Potamias, C. Mishra, G. Kollios, and N. Koudas. MRShare: Sharing Across Multiple Queries in MapReduce. PVLDB, 3(1-2):494-505, 2010.
23. C. Olston, B. Reed, A. Silberstein, and U. Srivastava. Automatic Optimization of Parallel Dataflow Programs. In USENIX, pages 267-273, 2008.
24. OpenCore Vs. BTrace. http://opencore.jinspired.com/?pageid=588.
25. G. Wang, A. Butt, P. Pandey, and K. Gupta. A Simulation Approach to Evaluating Design Decisions in MapReduce Setups. In MASCOTS, pages 1-11, 2009.
26. T. Weise. Global Optimization Algorithms: Theory and Application. Abrufdatum, 2008.
27. T. White. Hadoop: The Definitive Guide. Yahoo Press, 2010.
28. T. Ye and S. Kalyanaraman. A Recursive Random Search Algorithm for Large-scale Network Parameter Configuration. In SIGMETRICS, pages 196-205, 2003.