Clash of the titans: Mapreduce vs. spark for large scale data analytics

Proceedings of the VLDB Endowment

Volumn 8, Issue 13 13, 2015, Pages 2110-2121

  Shi, Juwei ^c   Qiu, Yunjie ^a   Minhas, Umar Farooq ^b   Jiao, Limei ^a   Wang, Chen ^d   Reinwald, Berthold ^b   Özcan, Fatma ^b  

^a IBM RESEARCH   (United States)

^b IBM ALMADEN RESEARCH CENTER   (United States)

^c RENMIN UNIVERSITY OF CHINA   (China)

^d TSINGHUA UNIVERSITY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

APPLICATION PROGRAMMING INTERFACES (API); BENCHMARKING; CLUSTER COMPUTING; COMPUTER ARCHITECTURE; FAULT TOLERANCE; OPEN SOURCE SOFTWARE; OPEN SYSTEMS;

COMPUTING FRAMEWORKS; DATA ANALYTICS; EXECUTION MODELING; FRAME-WORK; LARGE SCALE DATA; MAP-REDUCE; OPEN-SOURCE; PERFORMANCE; TASK EXECUTIONS; TASK PARALLELISM;

MAPREDUCE;

EID: 84952777770     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/2831360.2831365     Document Type: Chapter

References (19)

1. Apache Hadoop. http://hadoop.apache.org/.
2. Apache Mahout. https://mahout.apache.org/.
3. HDFS caching. http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html.
4. HPROF: A heap/cpu profiling tool. http://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html.
5. RRDtool. http://oss.oetiker.ch/rrdtool/.
6. Spark wins 2014 graysort competition. http://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html.
7. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. CACM, 51(1):107-113, 2008.
8. P. Henderson. Functional Programming: Application and Implementation. Prentice-Hall International London, 1980.
9. H. Herodotou and S. Babu. Profiling, what-if analysis, and cost-based optimization of mapreduce programs. VLDB, 4(11):1111-1122, 2011.
10. S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang. The hibench benchmark suite: Characterization of the mapreduce-based data analysis. In ICDEW, pages 41-51, 2010.
11. H. Li, A. Ghodsi, M. Zaharia, S. Shenker, and I. Stoica. Tachyon:Reliable, memory speed storage for cluster computing frameworks. In SOCC, pages 1-15, 2014.
12. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In SIGMOD, pages 135-146, 2010.
13. M. L. Massie, B. N. Chun, and D. E. Culler. The ganglia distributed monitoring system: design, implementation, and experience. Parallel Computing, 30(7):817-840, 2004.
14. A. Mislove, M. Marcon, K. Gummadi, P. Druschel, and B. Bhattacharjee. Measurement and analysis of online social networks. In IMC, pages 29-42, 2007.
15. O. OMalley and A. C. Murthy. Winning a 60 second dash with a yellow elephant. Sort Benchmark, 2009.
16. J. Shi, J. Zou, J. Lu, Z. Cao, S. Li, and C. Wang. MRTuner: A toolkit to enable holistic optimization for mapreduce jobs. VLDB, 7(13):1319-1330, 2014.
17. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache Hadoop YARN: Yet another resource negotiator. In SOCC, pages 5:1-5:16, 2013.
18. W. Xue, J. Shi, and B. Yang. X-RIME: Cloud-based large scale social network analysis. In SCC, pages 506-513, 2010.
19. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI, 2012.