Hadoop's adolescence: An analysis of hadoop usage in scientific workloads

Proceedings of the VLDB Endowment

Volumn 6, Issue 10, 2013, Pages 853-864

  Ren, Kai ^a   Kwon, Yong Chul ^b   Balazinska, Magdalena ^c   Howe, Bill ^c  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b MICROSOFT   (United States)

^c UNIVERSITY OF WASHINGTON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC TUNING; MEASUREMENT STUDY; RESOURCE USAGE; SCIENTIFIC WORKLOADS; USER-CENTERED; USER-CENTRIC;

EID: 84891051380     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/2536206.2536213     Document Type: Article

References (27)

1. Yahoo! reaches for the stars with M45 supercomputing project. http://research.yahoo.com/node/1884.
2. G. Ananthanarayanan et al. Reining in the outliers in Map-Reduce clusters using Mantri. In OSDI, 2010.
3. G. Ananthanarayanan et al. PACMan: Coordinated memory caching for parallel jobs. In NSDI, 2012.
4. Apache Foundation. Hadoop. http://hadoop.apache.org/.
5. Apache Foundation. Mahout: Scalable machine learning and data mining. http://mahout.apache.org/.
6. Ashish Thusoo et. al. Hive: a petabyte scale data warehouse using Hadoop. In ICDE, pages 996-1005, 2010.
7. S. Babu. Towards automatic optimization of mapreduce programs. In SoCC, pages 137-142, 2010.
8. D. Borthakur. The Hadoop distributed file system: Architecture and design. http://lucene.apache.org/hadoop/hdfs_design.pdf, 2007.
9. Y. Chen et al. The case for evaluating MapReduce performance using workload suites. In MASCOTS, pages 390-399.
10. Y. Chen et al. Interactive query processing in big data systems: A cross-industry study of MapReduce workloads. PVLDB, 5(12):1802-1813, 2012.
11. Concurrent, Inc. Cascading. http://www.cascading.org/, 2012.
12. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
13. B. Fan et al. DiskReduce: RAID for data-intensive scalable computing. Technical Report CMU-PDL-11-112, PDL, Carnegie Mellon University, 2011.
14. H. Herodotou and S. Babu. Profiling, what-if analysis, and cost-based optimization of MapReduce programs. PVLDB, 4(11):1111-1122, 2011.
15. U. Kang et al. PEGASUS: A peta-scale graph mining system implementation and observations. In ICDM, pages 229-238, 2009.
16. S. Kavulya et al. An analysis of traces from a production MapReduce cluster. In CCGRID, pages 94-103, 2010.
17. Q. Ke et al. Optimizing data partitioning for data-parallel computing. In HotOS, 2011.
18. N. Khoussainova et al. Perfxplain: Debugging mapreduce job performance. PVLDB, 5(7):598-609, 2012.
19. Y. Kwon et al. SkewTune: mitigating skew in mapreduce applications. In SIGMOD, pages 25-36, 2012.
20. S. P. Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129-137, 1982.
21. G. Malewicz et al. Pregel: a system for large-scale graph processing. In SIGMOD, pages 135-146, 2010.
22. C. Olston et al. Pig Latin: a not-so-foreign language for data processing. In SIGMOD, pages 1099-1110, 2008.
23. C. Olston et al. Generating example data for dataflow programs. In SIGMOD, pages 245-256, 2009.
24. A. Pavlo et al. A comparison of approaches to large-scale data analysis. In SIGMOD, pages 165-178, 2009.
25. Scoobi Team. A Scalar productivity framework for Hadoop. https://github.com/NICTA/scoobi, 2012.
26. B. Sharma et al. Modeling and synthesizing task placement constraints in Google compute clusters. In SoCC, pages 3:1-3:14, 2011.
27. R. Vernica et al. Adaptive MapReduce using situation-aware mappers. In EDBT, pages 420-431, 2012.