Improving MapReduce energy efficiency for computation intensive workloads

2011 International Green Computing Conference and Workshops, IGCC 2011

Volumn , Issue , 2011, Pages

  Wirtz, Thomas ^a   Ge, Rong ^a  

^a MARQUETTE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL GRIDS; DATA CENTERS; DATA-INTENSIVE COMPUTING; DYNAMIC VOLTAGE AND FREQUENCY SCALING; LARGE-SCALE DISTRIBUTED SYSTEM; MAP-REDUCE; PROGRAMMING MODELS; RUN-TIME CONFIGURATION; WORKLOAD CHARACTERISTICS;

COMPUTER PROGRAMMING; DISTRIBUTED COMPUTER SYSTEMS; RESOURCE ALLOCATION; SCHEDULING;

ENERGY EFFICIENCY;

EID: 80053190893     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IGCC.2011.6008564     Document Type: Conference Paper

References (30)

1. J. Dean, and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," in Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation (OSDI 2004), San Francisco, CA, 2004, pp. 10-10.
2. P. Dubey, "Recognition, Mining and Synthesis Moves Computers to the Era of Tera," Technology@Intel, 2005.
3. EPA, Report to Congress on Server and Data Center Energy Efficiency, Public Law 109-431, U.S., 2007.
4. Intel, "Increasing Data Center Density While Driving Down Power and Cooling Costs," ftp://download.intel.com/design/servers/technologies/ thermal.pdf,2006].
5. J. Pan, Y. L. Biannic, and F. Magoulès, "Parallelizing Multiple Group-by Query in Share-Nothing Environment: a MapReduce Study Case," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, Chicago, Illinois, 2010, pp. 856-863.
6. S. Chen, and S. Schlosser, Map-Reduce Meets Wider Varieties of Applications, Intel, 2005.
7. S. Leo, and G. Zanetti, "Pydoop: a Python MapReduce and HDFS API for Hadoop," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, Chicago, Illinois, 2010, pp. 819-825.
8. D. Huang, X. Shi, S. Ibrahim et al., "MR-Scope: a Real-Time Tracing Tool for MapReduce," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, Chicago, Illinois, 2010, pp. 849-855.
9. J. Ekanayake, H. Li, B. Zhang et al., "Twister: a Runtime for Iterative MapReduce," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, Chicago, Illinois, 2010, pp. 810-818.
10. R. T. Kaushik, and M. Bhandarkar, "GreenHDFS: towards an Energy-Conserving, Storage-Efficient, Hybrid Hadoop Compute Cluster," in Proceedings of the 2010 international conference on Power Aware Computing and Systems, Vancouver, BC, Canada, 2010, pp. 1-9.
11. Y. Chen, L. Keys, and R. H. Katz, Towards Energy Efficient MapReduce, UCB/EECS-2009-109, EECS Department, University of California, Berkeley, 2009.
12. Y. Chen, A. Ganapathi, A. Fox et al., Statistical Workloads for Energy Efficiency MapReduce, UCB/EECS-2010-6, University of California, Berkeley, 2010.
13. J. Leverich, and C. Kozyrakis, "On the Energy (in)Efficiency of Hadoop Clusters," SIGOPS Oper. Syst. Rev., vol. 44, no. 1, pp. 61-65, 2010.
14. T. Hoefler, A. Lumsdaine, and J. Dongarra, "Towards Efficient MapReduce Using MPI," in Proceedings of the 16th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, Espoo, Finland, 2009, pp. 240-249.
15. C. Ranger, R. Raghuraman, A. Penmetsa et al., "Evaluating MapReduce for Multi-Core and Multiprocessor Systems," in Proceedings of the 13th IEEE International Symposium on High Performance Computer Architecture, USA, 2007, pp. 12.
16. L. A. Barroso, and U. Hölzle, "The Case for Energy-Proportional Computing," Computer, vol. 40, no. 12, pp. 33-37, 2007.
17. "Hadoop webpage," http://hadoop.apache.org/.
18. R. Ge, X. Feng, S. Song et al., "PowerPack: Energy Profiling and Analysis of High-Performance Systems and Applications," IEEE Transactions on Parallel and Distributed Systems, vol. 21, no. 5, pp. 658-671, 2010.
19. D. Economou, S. Rivoire, C. Kozyrakis et al., "Full-System Power Analysis and Modeling for Server Environments," Workshop on Modeling, Benchmarking, and Simulation (MoBS), 2006.
20. P. Bohrer, E. N. Elnozahy, T. Keller et al., "The Case for Power Management in Web Servers," Power aware computing, pp. 261-289: Kluwer Academic Publishers, 2002.
21. D. Meisner, B. T. Gold, and T. F. Wenisch, "PowerNap: Eliminating Server Idle Power," in Proceeding of the 14th international conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2009), Washington, DC, USA, 2009, pp. 205-216.
22. B. M. Oppenheim, "Reducing Cluster Power Consumption by Dynamically Suspending Idle Nodes," DigitalCommons@CalPoly, 2010.
23. W. Lang, and J. M. Patel, "Energy Management for MapReduce Clusters," Proc. VLDB Endow., vol. 3, no. 1-2, pp. 129-139, 2010.
24. J. Xie, S. Yin, X. Ruan et al., "Improving MapReduce Performance through Data Placement in Heterogeneous Hadoop Clusters," in Proceedings of the 19th International Heterogeneity in Computing Workshop, Atlanta, Georgia, 2010.
25. M. Cardosa, A. Singh, H. Pucha et al., "Exploiting Spatio-Temporal Tradeoffs for Energy Efficient MapReduce in the Cloud," Department of Computer Science and Engineering, University of Minnesota, 2010.
26. Y. Chen, A. Ganapathi, and R. H. Katz, "To Compress or Not to Compress - Compute vs. IO Tradeoffs for Mapreduce Energy Efficiency," in Proceedings of the first ACM SIGCOMM workshop on Green networking, New Delhi, India, 2010, pp. 23-28.
27. J. Norstad. "A MapReduce Algorithm for Matrix Multiplication," 2010; http://homepage.mac.com/j.norstad/matrix-multiply/index.html.
28. M. C. Schatz, "CloudBurst: Highly Sensitive Read Mapping with MapReduce," Bioinformatics, vol. 25, no. 11, pp. 1363-9, 2009.
29. R. Ge, X. Feng, W.-c. Feng et al., "CPU MISER: A Performance-Directed, Run-Time System for Power-Aware Clusters," in Proceedings of International Conference on Parallel Processing (ICPP 2007), 2007, pp. 18-18.
30. R. Ge, X. Feng, and K. W. Cameron, "Performance-constrained Distributed DVS Scheduling for Scientific Applications on Poweraware Clusters," in Proceedings of the 2005 ACM/IEEE conference on Supercomputing, 2005, pp. 34.