Performance and energy efficiency of big data applications in cloud environments: A Hadoop case study

Journal of Parallel and Distributed Computing

Volumn 79-80, Issue , 2015, Pages 80-89

  Feller, Eugen ^a,b   Ramakrishnan, Lavanya ^a   Morin, Christine ^a,b  

^a LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^b INRIA   (France)

Author keywords

Cloud computing; Energy efficiency; Hadoop MapReduce; Performance; Virtualization

Indexed keywords

CLOUD COMPUTING; DISTRIBUTED COMPUTER SYSTEMS; ENERGY EFFICIENCY; OPEN SOURCE SOFTWARE; PARALLEL PROGRAMMING; SEPARATION; VIRTUAL REALITY;

CLOUD COMPUTING ENVIRONMENTS; ENERGY EFFICIENCY EVALUATION; HADOOP MAPREDUCE; LARGE-SCALE DATA ANALYSIS; OPEN SOURCE IMPLEMENTATION; PARALLEL PROGRAMMING MODEL; PERFORMANCE; VIRTUALIZATIONS;

BIG DATA;

EID: 84930383808     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2015.01.001     Document Type: Article

References (26)

1. Amazon Elastic MapReduce, http://aws.amazon.com/elasticmapreduce/ (2013).
2. The Apache Hadoop Framework, http://hadoop.apache.org (2013).
3. R. Bolze, F. Cappello, E. Caron, M. Daydé, F. Desprez, E. Jeannot, Y. Jégou, S. Lanteri, J. Leduc, N. Melab, G. Mornet, R. Namyst, P. Primet, B. Quetier, O. Richard, E.-G. Talbi, and I. Touche Grid'5000: a large scale and highly reconfigurable experimental grid testbed Int. J. High Perform. Comput. Appl. 20 4 2006 481 494
4. Y. Chen, S. Alspaugh, D. Borthakur, R. Katz, Energy efficiency for large-scale mapreduce workloads with significant interactive analysis, in: Proceedings of the 7th ACM European conference on Computer Systems, EuroSys '12, 2012, pp. 43-56.
5. J. Dean, S. Ghemawat, Mapreduce: simplified data processing on large clusters, in: Proceedings of the 6th conference on Symposium on Operating Systems Design & Implementation, Volume 6, OSDI'04, 2004, pp. 10-10.
6. J. Ekanayake, S. Pallickara, G. Fox, Mapreduce for data intensive scientific analyses, in: Proceedings of the 2008 Fourth IEEE International Conference on eScience, ESCIENCE '08, 2008, pp. 277-284.
7. Z. Fadika, M. Govindaraju, R. Canon, L. Ramakrishnan, Evaluating hadoop for data-intensive scientific operations, in: Proceedings of the 2012 IEEE Fifth International Conference on Cloud Computing, CLOUD '12, 2012, pp. 67-74.
8. E. Feller, L. Rilling, C. Morin, Snooze: a scalable and autonomic virtual machine management framework for private clouds, in: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID '12, 2012, pp. 482-489.
9. D. Ghoshal, R.S. Canon, L. Ramakrishnan, I/o performance of virtualized cloud environments, in: Proceedings of the second international workshop on Data intensive computing in the clouds, DataCloud-SC '11, 2011, pp. 71-80.
10. The Gluster Filesystem, http://www.gluster.org (2013).
11. I.n. Goiri, K. Le, T.D. Nguyen, J. Guitart, J. Torres, R. Bianchini, Greenhadoop: leveraging green energy in data-processing frameworks, in: Proceedings of the 7th ACM European conference on Computer Systems, EuroSys '12, 2012, pp. 57-70.
12. S. Ibrahim, H. Jin, L. Lu, L. Qi, S. Wu, X. Shi, Evaluating mapreduce on virtual machines: The hadoop case, in: Proceedings of the 1st International Conference on Cloud Computing, CloudCom '09, 2009, pp. 519-528.
13. A. Iordache, C. Morin, N. Parlavantzas, E. Feller, and P. Riteau Resilin: Elastic mapreduce over multiple clouds IEEE International Symposium on Cluster Computing and the Grid, 0 2013 261 268 http://doi.ieeecomputersociety.org/10.1109/CCGrid.2013.48
14. IOR HPC Benchmark, http://sourceforge.net/projects/ior-sio/ (2013).
15. D. Jiang, B.C. Ooi, L. Shi, and S. Wu The performance of mapreduce: an in-depth study Proc. VLDB Endow. 3 1-2 2010 472 483
16. W. Lang, and J.M. Patel Energy management for mapreduce clusters Proc. VLDB Endow. 3 1-2 2010 129 139
17. E. Le Sueur, G. Heiser, Dynamic voltage and frequency scaling: the laws of diminishing returns, in: Proceedings of the 2010 international conference on Power aware computing and systems, HotPower'10, 2010, pp. 1-8.
18. J. Leverich, and C. Kozyrakis On the energy (in)efficiency of hadoop clusters SIGOPS Oper. Syst. Rev. 44 1 2010 61 65
19. J. Liu, W. Huang, B. Abali, and D.K. Panda High performance VMM-bypass I/O in virtual machines USENIX Annual Technical Conference (ATC) 2006 USENIX Association Berkeley, CA, USA 29 42 URL http://portal.acm.org/citation.cfm?id=1267359.1267362
20. R. McDougall, Towards an elastic elephant: enabling hadoop for the cloud, http://cto.vmware.com/towards-an-elastic-elephant-enabling-hadoop-for-the-cloud/ (2012).
21. A. Menon, Big data @ Facebook, in: Proceedings of the 2012 workshop on Management of big data systems, MBDS '12, 2012, pp. 31-32.
22. D. Moise, and A. Carpen-Amarie Mapreduce applications in the cloud: A cost evaluation of computation and storage A. Hameurlain, F. Hussain, F. Morvan, A. Tjoa, Data Management in Cloud, Grid and P2P Systems Lecture Notes in Computer Science vol. 7450 2012 Springer Berlin Heidelberg 37 48
23. J. Shafer, S. Rixner, and A. Cox The hadoop distributed filesystem: Balancing portability and performance 2010 IEEE International Symposium on Performance Analysis of Systems Software (ISPASS) 2010 122 133 http://dx.doi.org/10.1109/ISPASS.2010.5452045
24. S. Weil, S.A. Brandt, E.L. Miller, D.D.E. Long, C. Maltzahn, Ceph: a scalable, high-performance distributed file system, in: Proceedings of the 7th symposium on Operating systems design and implementation, OSDI '06, 2006, pp. 307-320.
25. T. Wirtz, R. Ge, Improving mapreduce energy efficiency for computation intensive workloads, in: Proceedings of the 2011 International Green Computing Conference and Workshops, IGCC '11, 2011.
26. K. Yelick, S. Coghlan, B. Draney, and R.S. Canon The Magellan Report on Cloud Computing for Science, Tech. Rep. 2011 U.S. Department of Energy Office of Science Office of Advanced Scientific Computing Research (ASCR) Dec.