Mronline: MapReduce online performance tuning

HPDC 2014 - Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing

Volumn , Issue , 2014, Pages 165-176

  Li, Min ^a   Zeng, Liangzhao ^b   Meng, Shicong ^b   Tan, Jian ^b   Zhang, Li ^b   Butt, Ali R ^a   Fuller, Nicholas ^b  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Cloud computing; Dynamic parameter configuration; Hill climbing; MapReduce; Online parameter tuning; Performance tuning; YARN

Indexed keywords

CLOUD COMPUTING; WOOL; YARN;

DYNAMIC PARAMETERS; HILL CLIMBING; MAP-REDUCE; ON-LINE PARAMETER; PERFORMANCE TUNING;

OPTIMIZATION;

EID: 84904434408     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2600212.2600229     Document Type: Conference Paper

References (37)

1. Cloudera. 7 tips for improving MapReduce performance, 2009. http://blog.cloudera.com/blog/2009/12/7-tips-forimproving-mapreduce-performance/ .
2. Cloudera. Optimizing MapReduce job performance, 2012. http://www.slideshare.net/cloudera/mr-perf.
3. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In Proc. USENIX OSDI, 2004.
4. J. Dittrich, J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). Proceedings of the VLDB Endowment, 3(1-2):515-529, 2010.
5. S. Duan, V. Thummala, and S. Babu. Tuning database configuration parameters with ituned. Proceedings of the VLDB Endowment, 2(1):1246-1257, 2009.
6. A. S. Foundation. Mumak: MapReduce simulator, 2009. https://issues. apache.org/jira/browse/MAPREDUCE-728.
7. A. S. Foundation. Apache Giraph, 2013. http://giraph.apache.org/.
8. A. S. Foundation. Hadoop-2.1.0-Beta, 2013. http://www.trieuvan.com/ apache/hadoop/common/hadoop-2.1.0-beta/.
9. A. S. Foundation. Grep example, 2014. http://wiki.apache.org/hadoop/Grep.
10. A. S. Foundation. Terasort example, 2014. https://hadoop. apache.org/docs/current/api/org/apache/hadoop/examples/terasort/package-summary. html.
11. S. Ghemawat, H. Gobioff, and S. Leung. The Google file system. ACM SIGOPS Operating Systems Review, 37(5):29-43, 2003.
12. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica. Dominant resource fairness: fair allocation of multiple resource types. In Proc. USENIX NSDI, 2011.
13. H. Herodotou and S. Babu. Profiling, what-if analysis, and cost-based optimization of mapreduce programs. Proceedings of the VLDB Endowment, 4(11):1111-1122, 2011.
14. H. Herodotou, F. Dong, and S. Babu. Mapreduce programming and cost-based optimization? Crossing this chasm with starfish. Proceedings of the VLDB Endowment, 4(12):1446-1449, 2011.
15. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A self-tuning system for big data analytics. In Proc. Conference on Innovative Data System Research, 2011.
16. Impetus. Advanced Hadoop tuning and optimizations, 2009. http://www.slideshare.net/ImpetusInfo/ppt-on-advancedhadoop-tuning-n- optimisation.
17. Impetus. Hadoop performance tuning, 2012. https://hadoop-toolkit. googlecode.com/files/White paper-HadoopPerformanceTuning.pdf.
18. G. Inc. Freebase data dumps, 2013. https://developers.google.com/ freebase/data.
19. A. Incubator. Spark: Lightning-fast cluster computing, 2013. http://spark.incubator.apache.org/.
20. E. Jahani, M. J. Cafarella, and C. Ré. Automatic optimization for mapreduce programs. Proceedings of the VLDB Endowment, 4(6):385-396, 2011.
21. D. Jiang, B. C. Ooi, L. Shi, and S. Wu. The performance of mapreduce: An in-depth study. Proceedings of the VLDB Endowment, 3(1-2):472-483, 2010.
22. N. Khoussainova, M. Balazinska, and D. Suciu. Perfxplain: debugging mapreduce job performance. Proceedings of the VLDB Endowment, 5(7):598-609, 2012.
23. Y. Kwon, M. Balazinska, B. Howe, and J. Rolia. Skewtune: Mitigating skew in mapreduce applications. In Proc. ACM SIGMOD International Conference on Management of Data, 2012.
24. P. Lama and X. Zhou. Aroma: Automated resource allocation and configuration of mapreduce environment in the cloud. In Proc. ACM International Conference on Autonomic Computing, 2012.
25. G. Liao, K. Datta, and T. L. Willke. Gunther: Search-based auto-tuning of mapreduce. In Proc. Springer Euro-Par, 2013.
26. J. Lin and C. Dyer. Cloud9: A hadoop toolkit for working with big data, 2010. http://lintool.github.io/Cloud9/index.html.
27. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In Proc. ACM SIGMOD International Conference on Management of Data, 2010.
28. Twitter. Storm: Distributed and fault-tolerant realtime computation, 2013. http://storm-project.net/.
29. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, et al. Apache Hadoop Yarn: Yet another resource negotiator. In Proc. ACM Symposium on Cloud Computing, 2013.
30. G. Wang, A. R. Butt, P. Pandey, and K. Gupta. A simulation approach to evaluating design decisions in mapreduce setups. In Proc. IEEE MASCOTS, 2009.
31. T. White. Hadoop: The Definitive Guide. O'Reilly, 2012.
32. Wikipedia. Wikipedia data dumps, 2014. http://dumps.wikimedia.org/enwiki/ latest/.
33. B. Xi, Z. Liu, M. Raghavachari, C. H. Xia, and L. Zhang. A smart hill-climbing algorithm for application server configuration. In Proc. ACM International Conference on World Wide Web, 2004.
34. T. Ye and S. Kalyanaraman. A recursive random search algorithm for large-scale network parameter configuration. ACM SIGMETRICS Performance Evaluation Review, 31(1):196-205, 2003.
35. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster computing with working sets. In Proc. USENIX Conference on Hot Topics in Cloud Computing, 2010.
36. J. Zhang, H. Zhou, R. Chen, X. Fan, Z. Guo, H. Lin, J. Y. Li, W. Lin, J. Zhou, and L. Zhou. Optimizing data shuffling in data-parallel computation by understanding user-defined functions. In Proc. USENIX NSDI, 2012.
37. W. Zheng, R. Bianchini, G. J. Janakiraman, J. R. Santos, and Y. Turner. JustRunIt: Experiment-based management of virtualized data centers. In Proc. USENIX ATC, 2009.