Energy-Aware Scheduling of MapReduce Jobs for Big Data Applications

IEEE Transactions on Parallel and Distributed Systems

Volumn 26, Issue 10, 2015, Pages 2720-2733

  Mashayekhy, Lena ^a   Nejad, Mahyar Movahed ^a   Grosu, Daniel ^a   Zhang, Quan ^a   Shi, Weisong ^a  

^a Wayne State University   (United States)

Author keywords

Algorithm design and analysis; Clustering algorithms; Energy consumption; Schedules; Scheduling; Scheduling algorithms

Indexed keywords

ALGORITHMS; BENCHMARKING; BIG DATA; ENERGY EFFICIENCY; ENERGY UTILIZATION; HEURISTIC ALGORITHMS; INTEGER PROGRAMMING; JOB SHOP SCHEDULING; MULTIPROCESSING SYSTEMS; POWER MANAGEMENT; SCHEDULING; SCHEDULING ALGORITHMS;

ALGORITHM DESIGN AND ANALYSIS; BENCHMARK SUITES; BIG DATA APPLICATIONS; ENERGY-AWARE SCHEDULING; EXTENSIVE SIMULATIONS; K-MEANS CLUSTERING; OPEN SOURCE IMPLEMENTATION; SERVICE LEVEL AGREEMENTS;

CLUSTERING ALGORITHMS;

EID: 84961828601     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2014.2358556     Document Type: Article

References (46)

1. J. Koomey, "Growth in data center electricity use 2005 to 2010, " vol. 1. Oakland, CA, USA: Analytics Press, Aug. 2011.
2. J. Hamilton, "Cooperative expendable micro-slice servers (CEMS): Low cost, low power servers for internet-scale services, " in Proc. Conf. Innovative Data Syst. Res., 2009.
3. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters, " in Proc. 6th USENIX Symp. Oper. Syst. Des. Implementation, 2004, pp. 137-150.
4. Hadoop. (2014) [Online]. Available: http://hadoop.apache.org
5. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Job scheduling for multi-user MapReduce clusters, " Electrical Eng. Comput. Sci. Dept., Univ. California, Berkeley, CA, USA, Tech. Rep. UCB/EECS-2009-55, Apr. 2009.
6. Apc. (2014) [Online]. Available: http://www.apc.com/
7. J. Polo, C. Castillo, D. Carrera, Y. Becerra, I. Whalley, M. Steinder, J. Torres, and E. Ayguade, "Resource-Aware adaptive scheduling for mapreduce clusters, " in Proc. 12th ACM/IFIP/USENIX Int. Middleware Conf., 2011, pp. 187-207.
8. A. Verma, L. Cherkasova, and R. H. Campbell, "ARIA: Automatic resource inference and allocation for MapReduce environments, " in Proc. 8th ACM Int. Conf. Autonomic Comput., 2011, pp. 235-244.
9. A. Verma, L. Cherkasova, and R. H. Campbell, "Two sides of a coin: Optimizing the schedule of MapReduce jobs to minimize their makespan and improve cluster performance, " in Proc. IEEE 20th Int. Symp. Model., Anal. Simul. Comput. Telecommun. Syst., 2012, pp. 11-18.
10. B. Palanisamy, A. Singh, L. Liu, and B. Jain, "Purlieus: Localityaware resource allocation for MapReduce in a cloud, " in Proc. Conf. High Perform. Comput., Netw., Storage Anal., 2011.
11. B. Moseley, A. Dasgupta, R. Kumar, and T. Sarlos, "On scheduling in map-reduce and flow-shops, " in Proc. 23rd Annu. ACM Symp. Parallelism Algorithms Archit., 2011, pp. 289-298.
12. H. Chang, M. S. Kodialam, R. R. Kompella, T. V. Lakshman, M. Lee, and S. Mukherjee, "Scheduling in MapReduce-like systems for fast completion time, " in Proc. IEEE 30th Int. Conf. Comput. Commun., 2011, pp. 3074-3082.
13. F. Chen, M. S. Kodialam, and T. V. Lakshman, "Joint scheduling of processing and shuffle phases in MapReduce systems, " in Proc. IEEE Conf. Comput. Commun., 2012, pp. 1143-1151.
14. Y. Zheng, N. B. Shroff, and P. Sinha, "A new analytical technique for designing provably efficient MapReduce schedulers, " in Proc. IEEE Conf. Comput. Commun., 2013, pp. 1600-1608.
15. T. J. Hacker and K. Mahadik, "Flexible resource allocation for reliable virtual cluster computing systems, " in Proc. Int. Conf. High Perform. Comp., Netw., Storage Anal., 2011, pp. 1-12.
16. B. Palanisamy, A. Singh, and L. Liu, "Cost-effective resource provisioning for MapReduce in a cloud, " IEEE Trans. Parallel Distrib. Syst., no. 1, pp. 1, PrePrints, 2014.
17. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox, "Twister: A runtime for iterative MapReduce, " in Proc. 19th ACM Int. Symp. High Perform. Distrib. Comput., 2010, pp. 810-818.
18. F. Tian and K. Chen, "Towards optimal resource provisioning for running MapReduce programs in public clouds, " in Proc. IEEE Int. Conf. Cloud Comput., 2011, pp. 155-162.
19. J. Zhan, L. Wang, X. Li, W. Shi, C. Weng, W. Zhang, and X. Zang, "Cost-Aware cooperative resource provisioning for heterogeneous workloads in data centers, " IEEE Trans. Comput., vol. 62, no. 11, pp. 2155-2168, Nov. 2013.
20. Y. Song, Y. Sun, and W. Shi, "A two-Tiered on-demand resource allocation mechanism for VM-based data centers, " IEEE Trans. Services Comput., vol. 6, no. 1, pp. 116-129, Jan. 2013.
21. M. Nejad, L. Mashayekhy, and D. Grosu, "Truthful greedy mechanisms for dynamic virtual machine provisioning and allocation in clouds, " IEEE Trans. Parallel Distrib. Syst., vol. PP, no. 1, pp. 1, 2014.
22. L. Mashayekhy, M. Nejad, and D. Grosu, "Cloud federations in the sky: Formation game and mechanism, " IEEE Trans. Cloud Comput., vol. PP, no. 1, pp. 1, 2014.
23. L. Mashayekhy, M. Nejad, D. Grosu, and A. V. Vasilakos, "Incentive-compatible online mechanisms for resource provisioning and allocation in clouds, " in Proc. 7th IEEE Int. Conf. Cloud Comput., 2014.
24. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving MapReduce performance in heterogeneous environments, " in Proc. 8th USENIX Conf. Oper. Syst. Des. Implementation, 2008, pp. 29-42.
25. T. Sandholm and K. Lai, "MapReduce optimization using regulated dynamic prioritization, " in Proc. 11th ACM Int. Conf. Meas. Model. Comput. Syst., 2009, pp. 299-310.
26. X. Wang, D. Shen, G. Yu, T. Nie, and Y. Kou, "A throughput driven task scheduler for improving MapReduce performance in job-intensive environments, " in Proc. IEEE 2nd Int. Congr. Big Data, 2013, pp. 211-218.
27. Z. Ren, X. Xu, M. Zhou, J. Wan, and W. Shi, "Workload analysis, implications, and optimization on a production Hadoop cluster: A case study on taobao, " IEEE Trans. Services Comput., vol. 7, no. 2, pp. 307-321, Apr.-Jun. 2014.
28. M. Pastorelli, A. Barbuzzi, D. Carra, M. Dell'Amico, and P. Michiardi, "HFSP: Size-based scheduling for Hadoop, " in Proc. IEEE Int. Conf. Big Data, 2013, pp. 51-59.
29. J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh, K.-L. Wu, and A. Balmin, "Flex: A slot allocation scheduling optimizer for MapReduce workloads, " in Proc. ACM/IFIP/USENIX 11th Int. Conf. Middleware, 2010, pp. 1-20.
30. T. Sandholm and K. Lai, "Dynamic proportional share scheduling in Hadoop, " in Job Scheduling Strategies for Parallel Processing. New York, NY, USA: Springer, 2010, pp. 110-131.
31. R. Nanduri, N. Maheshwari, A. Reddyraja, and V. Varma, "Job aware scheduling algorithm for MapReduce framework, " in Proc. IEEE 3rd Int. Conf. Cloud Comput. Technol. Sci., 2011, pp. 724-729.
32. S. Ibrahim, H. Jin, L. Lu, B. He, G. Antoniu, and S. Wu, "Maestro: Replica-Aware map scheduling for MapReduce, " in Proc. 12th IEEE/ACM Int. Symp. Cluster, Cloud Grid Comput., 2012, pp. 435-442.
33. S. Kurazumi, T. Tsumura, S. Saito, and H. Matsuo, "Dynamic processing slots scheduling for i/o intensive jobs of hadoop MapReduce, " in Proc. IEEE 3rd Int. Conf. Netw. Comput., 2012, pp. 288-292.
34. R. T. Kaushik, M. Bhandarkar, and K. Nahrstedt, "Evaluation and analysis of greenHDFS: A self-Adaptive, energy-conserving variant of the Hadoop distributed file system, " in Proc. IEEE 2nd Int. Conf. Cloud Comput. Technol. Sci., 2010, pp. 274-287.
35. M. Cardosa, A. Singh, H. Pucha, and A. Chandra, "Exploiting spatio-Temporal tradeoffs for energy-Aware MapReduce in the cloud, " IEEE Trans. Comput., vol. 61, no. 12, pp. 1737-1751, Dec. 2012.
36. J. Leverich and C. Kozyrakis, "On the energy (in) efficiency of Hadoop clusters, " ACM SIGOPS Oper. Syst. Rev., vol. 44, no. 1, pp. 61-65, 2010.
37. Y. Chen, S. Alspaugh, D. Borthakur, and R. Katz, "Energy efficiency for large-scale MapReduce workloads with significant interactive analysis, " in Proc. 7th ACM Eur. Conf. Comput. Syst., 2012, pp. 43-56.
38. N. Maheshwari, R. Nanduri, and V. Varma, "Dynamic energy efficient data placement and cluster reconfiguration algorithm for mapreduce framework, " Future Generation Comput. Syst., vol. 28, no. 1, pp. 119-127, 2012.
39. W. Lang and J. M. Patel, "Energy management for MapReduce clusters, " Proc. VLDB Endowment, vol. 3, nos. 1/2, pp. 129-139, 2010.
40. T. Wirtz and R. Ge, "Improving MapReduce energy efficiency for computation intensive workloads, " in Proc. IEEE Int. Green Comput. Conf. Workshops, 2011, pp. 1-8.
41. I. Goiri, K. Le, T. D. Nguyen, J. Guitart, J. Torres, and R. Bianchini, "GreenHadoop: Leveraging green energy in data-processing frameworks, " in Proc. 7th ACM Eur. Conf. Comput. Syst., 2012, pp. 57-70.
42. M. A. Salehi, P. Radha Krishna, K. S. Deepak, and R. Buyya, "Preemption-Aware energy management in virtualized data centers, " in Proc. IEEE 5th Int. Conf. Cloud Comput., 2012, pp. 844-851.
43. S. Shen and J. Wang, "Stochastic modeling and approaches for managing energy footprints in cloud computing service, " Service Sci., vol. 6, no. 1, pp. 15-33, 2014.
44. IBM ILOG CPLEX V12.1 user's manual. (2014). [Online]. Available: ftp://public.dhe.ibm.com/software/websphere/ilog/docs/op-Timization/cplex/ps-usrmancplex.pdf
45. S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang, "The HiBench benchmark suite: Characterization of the MapReduce-based data analysis, " in Proc. IEEE 26th Conf. Data Eng. Workshops, 2010, pp. 41-51.
46. L. Mashayekhy, M. Nejad, D. Grosu, D. Lu, and W. Shi, "Energyaware scheduling of MapReduce jobs, " in Proc. IEEE 3rd Int. Congr. Big Data, 2014.