Towards energy awareness in Hadoop

Proceedings of NDM 2014: 4th International Workshop on Network-Aware Data Management - Held in Conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis

Volumn , Issue , 2014, Pages 16-22

  Krish, R K ^a   Iqbal, M Safdar ^a   Rafique, M Mustafa ^b   Butt, Ali R ^b  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

^b IBM RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; DIGITAL STORAGE; HARDWARE; INFORMATION MANAGEMENT; SCHEDULING;

APPLICATION PERFORMANCE; DATA-INTENSIVE APPLICATION; EFFICIENCY AND PERFORMANCE; ENERGY CHARACTERISTICS; HARDWARE CONFIGURATIONS; HETEROGENEOUS CLUSTERS; SCHEDULING CONSTRAINTS; SERVICE LEVEL AGREEMENTS;

ENERGY EFFICIENCY;

EID: 84988268526     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/NDM.2014.6     Document Type: Conference Paper

References (35)

1. H. Amur, J. Cipar, V. Gupta, G. R. Ganger, M. A. Kozuch, and K. Schwan. Robust and flexible power-proportional storage. In ACM SOCC, 2010.
2. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. Pacman: Coordinated memory caching for parallel jobs. In Proc. USENIX NSDI, 2012.
3. Apache Software Foundation. Hadoop, 2011. http://hadoop.apache.org/core/.
4. J. L. Berral, I. Goiri, R. Nou, F. Julia, J. Guitart, R. Gavalda, and J. Torres. Towards energy-aware scheduling in data centers using machine learning. In Proceedings of the 1st International Conference on energy-Efficient Computing and Networking, pages 215-224. ACM, 2010.
5. D. e. Borthakur. Apache hadoop goes realtime at Facebook. Proc. ACM SIGMOD, 2011.
6. Y. Chen, S. Alspaugh, and R. Katz. Interactive analytical processing in big data systems: A cross-industry study of mapreduce workloads. Proc. VLDB Endowment, 5(12):1802-1813, 2012.
7. Y. Chen, A. Ganapathi, R. Griffith, and R. Katz. The case for evaluating mapreduce performance using workload suites. In Proc. IEEE MASCOTS, 2011.
8. Y. Chen, L. Keys, and R. H. Katz. Towards energy efficient mapreduce. Technical Report UCB/EECS-2009-109, EECS Department, University of California, Berkeley, Aug 2009.
9. E. S. Chung, J. D. Davis, and J. Lee. Linqits: Big data on little clients. In Proc. ACM ISCA, 2013.
10. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107-113, 2008.
11. E. E. Devices. Watts up pro, 2009.
12. F. Dong. Extending Starfish to Support the Growing Hadoop Ecosystem. PhD thesis, Duke University, 2012.
13. G. Fettweis and E. Zimmermann. let energy consumption-trends and challenges. In Proceedings of the 11th International Symposium on Wireless Personal Multimedia Communications, volume 2, page 6, 2008.
14. I. Goiri, K. Le, T. D. Nguyen, J. Guitart, J. Torres, and R. Bianchini. Greenhadoop: leveraging green energy in data-processing frameworks. In Eurosys, 2012.
15. Y Hotta, M. Sato, H. Kimura, S. Matsuoka, T. Boku, and D. Takahashi. Profile-based optimization of power performance by using dynamic voltage scaling on a pc cluster. In IEEE IPDPS, 2006.
16. J. Hu and R. Marculescu. Energy-aware communication and task scheduling for network-on-chip architectures under real-time constraints. In Design, Automation and Test in Europe Conference and Exhibition, 2004. Proceedings, volume 1, pages 234-239. IEEE, 2004.
17. S. Huang et al. Hibench: A representative and comprehensive hadoop benchmark suite.
18. S. Huang, J. Huang, Y. Liu, L. Yi, and J. Dai. Hibench: A representative and comprehensive hadoop benchmark suite. In Proc. ICDE Workshops, 2010.
19. M. Islam, A. K. Huang, M. Battisha, M. Chiang, S. Srinivasan, C. Peters, A. Neumann, and A. Abdelnur. Oozie: towards a scalable workflow management system for hadoop. In Proc. ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies, 2012.
20. R. Kaushik, M. Bhandarkar, and K. Nahrstedt. Evaluation and analysis of greenhdfs: A self-adaptive, energy-conserving variant of the hadoop distributed file system. In Proc. IEEE International Conference on Cloud Computing Technology and Science, 2010.
21. K. Krish, A. Anwar, and A. R. Butt. ?sched: A heterogeneity-aware hadoop workflow scheduler. In Proc. IEEE MASCOTS, 2014.
22. J. Leverich and C. Kozyrakis. On the energy (in)efficiency of hadoop clusters. In ACM SIGOPS Operating Systems Review, 2010.
23. R. Mantri, R. Ingle, and P. Patil. Scdp: Scalable, cost-effective, distributed and parallel computing model for academics. In Proc. ICECT, 2011.
24. M. Mihailescu, G. Soundararajan, and C. Amza. Mixapart: Decoupled analytics for shared storage systems. In Proc. USENIX HotStorage, 2012.
25. R. Mishra, N. Rastogi, D. Zhu, D. Mossé, and R. Melhem. Energy aware scheduling for distributed real-time systems. In Parallel and Distributed Processing Symposium, 2003. Proceedings. International, pages 9-pp. IEEE, 2003.
26. C. Olston, G. Chiou, L. Chitnis, F. Liu, Y. Han, M. Larsson, A. Neumann, V. B. Rao, V. Sankarasubramanian, S. Seth, et al. Nova: continuous pig/hadoop workflows. In Proc. ACM SIGMOD, 2011.
27. J. Peltonen. Energy-aware scheduling. In Proceedings of Seminar on Energy-Aware Software, page 7, 2007.
28. S. Rivoire, M. A. Shah, P. Ranganathan, and C. Kozyrakis. Joulesort: a balanced energy-efficiency benchmark. In ACM SIGMOD, 2007.
29. C. Wang, X. Li, X. Zhou, Y. Chen, and R. C. Cheung. Big data genome sequencing on zynq based clusters. In Proc. ACM SIGDA, 2014.
30. L. Wang, G. Von Laszewski, J. Dayal, and F. Wang. Towards energy aware scheduling for precedence constrained parallel tasks in a cluster with dvfs. In Cluster, Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference on, pages 368-377. IEEE, 2010.
31. M. Weiser, B. Welch, A. Demers, and S. Shenker. Scheduling for reduced cpu energy. In Mobile Computing, pages 449-471. Springer, 1996.
32. F. Yao, A. Demers, and S. Shenker. A scheduling model for reduced cpu energy. In Foundations of Computer Science, 1995. Proceedings., 36th Annual Symposium on, pages 374-382. IEEE, 1995.
33. N. Yigitbasi, K. Datta, N. Jain, and T. Willke. Energy efficient scheduling of mapreduce workloads on heterogeneous clusters. In GCM, 2011.
34. M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In Proc. USENIX OSDI, 2008.
35. N. Zhu, L. Rao, X. Liu, J. Liu, and H. Guan. Taming power peaks in mapreduce clusters. In ACM SIGCOMM Computer Communication Review, 2011.