Hitune: Dataflow-based performance analysis for big data cloud

Proceedings of the 2011 USENIX Annual Technical Conference, USENIX ATC 2011

Volumn , Issue , 2019, Pages 87-100

  Dai, Jinquan ^a   Huang, Jie ^a   Huang, Shengsheng ^a   Huang, Bo ^a   Liu, Yan ^a  

^a INTEL CHINA RESEARCH CENTER   (China)

Author keywords

[No Author keywords available]

Indexed keywords

BIG DATA;

DATA CLOUDS; DATAFLOW; DISTRIBUTED SYSTEMS; FINE TUNING; MAP-REDUCE; PERFORMANCE ANALYSIS; SYSTEM STATISTICS;

DATA FLOW ANALYSIS;

EID: 85077087102     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, H. Liu. “Data warehousing and analytics infrastructure at facebook”. The 36th ACM SIGMOD International Conference on Management of Data, 2010.
2. J. Dean, S. Ghemawat. “MapReduce: Simplified Data Processing on Large Clusters”. The 6th Symposium on Operating Systems Design and Implementation, 2004.
3. Hadoop. http://hadoop.apache.org/
4. M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly. “Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks”. The 2nd European Conference on Computer Systems, 2007.
5. C. Olston, B. Reed, U. Srivastava, R. Kumar, A. Tomkins. “Pig latin: a not-so-foreign language for data processing”. The 34th ACM SIGMOD international conference on Management of data, 2008.
6. A. Thusoo, R. Murthy, J. S. Sarma, Z. Shao, N. Jain, P. Chakka, S. Anthony, H. Liu, N. Zhang. “Hive - A Petabyte Scale Data Warehousing Using Hadoop”. The 26th IEEE International Conference on Data Engineering, 2010.
7. Dryad. http://research.microsoft.com/en-us/projects/Dryad/
8. “Hadoop and HBase at RIPE NCC”. http://www.cloudera.com/blog/2010/11/hadoop-and-hbase-atripe-ncc/
9. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, M. Stonebraker. “A comparison of approaches to large-scale data analysis”. The 35th SIGMOD international conference on Management of data, 2009.
10. S. L. Graham, P. B. Kessler, M. K. Mckusick. “Gprof: A call graph execution profiler”. The 1982 ACM SIGPLAN Symposium on Compiler Construction, 1982.
11. Intel VTune Performance Analyzer. http://software.intel.com/en-us/intel-vtune/
12. A. Rabkin, R. H. Katz. “Chukwa: A system for reliable large-scale log collection”, Technical Report UCB/EECS-2010-25, UC Berkeley, 2010.
13. Scribe. http://github.com/facebook/scribe
14. Flume. https://github.com/cloudera/flume
15. Java Instrumentation. http://download.oracle.com/javase/6/docs/api/java/lang/instrument/packagesummary.html
16. Sort benchmark. http://sortbenchmark.org/
17. S. Huang, J. Huang, J. Dai, T. Xie, B. Huang. “The HiBench Benchmark suite: Characterization of the MapReduce-Based Data Analysis”. IEEE 26th International Conference on Data Engineering Workshops, 2010.
18. th edition, 2006.
19. Jeff Dean. “Designs, Lessons and Advice from Building Large Distributed Systems”. The 3rd ACM SIGOPS International Workshop on Large Scale Distributed Systems and Middleware, 2009.
20. Jeffrey Dean. “Experiences with MapReduce, an abstraction for large-scale computation”. The 15th International Conference on Parallel Architectures and Compilation Techniques, 2006.
21. Ganglia. http://ganglia.sourceforge.net/
22. A fair sharing job scheduler. https://issues.apache.org/jira/browse/HADOOP-3746
23. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, I. Stoica. “Job Scheduling for Multi-User MapReduce Clusters”. Technical Report UCB/EECS-2009-55, UC Berkeley, 2009.
24. Hadoop Fair Scheduler Design Document. https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/fairscheduler/designdoc/fair_sche duler_design_doc.pdf
25. Hadoop LZO patch. http://github.com/kevinweil/ hadoop-lzo
26. Hive performance benchmark. https://issues.apache.org/jira/browse/HIVE-396
27. P. Barham, R. Isaacs, R. Mortier, D. Narayanan. “Magpie: online modelling and performance-aware systems”. The 9th conference on Hot Topics in Operating Systems, 2003.
28. R. Fonseca, G. Porter, R. H. Katz, S. Shenker, I. Stoica. “X-trace: A pervasive network tracing framework”. The 4th USENIX Symposium on Networked Systems Design & Implementation, 2007.
29. B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, C. Shanbhag. “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure”. Google Research, 2010.
30. B. Chun, K. Chen, G. Lee, R. Katz, S. Shenker. “D3: Declarative Distributed Debugging”. Technical Report UCB/EECS-2008-27, UC Berkeley, 208.
31. G. Ren, E. Tune, T. Moseley, Y. Shi, S. Rus, R. Hundt. “Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers”. IEEE Micro (2010), pp. 65-79.
32. “HPROF: a Heap/CPU Profiling Tool in J2SE 5.0”. http://java.sun.com/developer/technicalArticles/Programming/HPROF.html
33. Nagios. http://www.nagios.org/
34. Cacti. http://www.cacti.net/
35. V. Bhat, S. Gogate, M. Bhandarkar. “Hadoop Vaidya”. Hadoop World 2009.
36. J. Tan, X. Pan, S. Kavulya, R. Gandhi, P. Narasimhan. “Kahuna: Problem Diagnosis for MapReduce-Based Cloud Computing Environments”. IEEE/IFIP Network Operations and Management Symposium (NOMS), 2010.
37. G. Cretu, M. Budiu, M. Goldszmidt . “Hunting for problems with Artemis”. First USENIX conference on Analysis of system logs, 2008.