Modulo Based Data Placement Algorithm for Energy Consumption Optimization of MapReduce System

Journal of Grid Computing

Volumn 16, Issue 3, 2018, Pages 409-424

  Song, Jie ^a   He, HongYan ^a   Wang, Zhi ^a   Yu, Ge ^b   Pierson, Jean Marc ^c  

^a NORTHEASTERN UNIVERSITY   (China)

^b NORTHEASTERN UNIVERSITY   (China)

^c IRIT   (France)

Author keywords

Big data; Data placement; Energy consumption optimization; MapReduce

Indexed keywords

ENERGY UTILIZATION; HTTP; INFORMATION MANAGEMENT;

ADDITIONAL COSTS; DATA PLACEMENT; DATA PLACEMENT ALGORITHMS; DISTRIBUTED SYSTEMS; ENERGY CONSUMPTION OPTIMIZATION; IT INFRASTRUCTURES; MAP-REDUCE; TIME CONSUMPTION;

BIG DATA;

EID: 84976330191     PISSN: 15707873     EISSN: 15729184     Source Type: Journal    
DOI: 10.1007/s10723-016-9370-2     Document Type: Article

References (31)

1. Labrinidis, A., Jagadish, H.V.: Challenges and opportunities with big data. PVLDB 5(12), 2032–2033 (2012)
2. Dean, J., Ghemawat, S.: Mapreduce: a flexible data processing tool. Commun. ACM (CACM) 53(1), 72–77 (2010)
3. Song, J., Liu, X., Zhu, Z., Zhao, D., Yu, G.: A Novel Task Scheduling Approach for Reducing Energy Consumption of MapReduce Cluster. IETE Tech. Rev. 31(1), 65–74 (2014)
4. Elnozahy, E.N., Kistler, M., Rajamony, R.: Energy-efficient server clusters. PACS 2002, 179–196
5. Lee, K.G., Bharadwaj, V., Sivakumar, V.: Design of fast and efficient Energy-Aware Gradient-Based scheduling algorithms heterogeneous embedded multiprocessor systems. IEEE Trans. Parallel Distrib. Syst. (TPDS) 20(1), 1–12 (2009)
6. Da Costa, G., Dias de Assunção, M., Gelas, J-P, Georgiou, Y., Lefèvre, L., Orgerie, A., Pierson, J-M, Olivier, R., Sayah, A.: Multi-facet approach to reduce energy consumption in clouds and grids: the GREEN-NET framework. e-Energy 2010, 95–104
7. Lang, W., Patel, J.M.: Energy management for MapReduce clusters. PVLDB 3(1), 129–139 (2010)
8. Maheshwari, N., Nanduri, R., Varma, V.: Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework. Future Generation Comp. Syst. (FGCS) 28(1), 119–127 (2012)
9. Xiong, W., Kansal, A.: Energy efficient data intensive distributed computing. IEEE Data Eng. Bull. (DEBU) 34(1), 24–33 (2011)
10. Palanisamy, B., Singh, A., Liu, L., Jain, B.: Purlieus: locality-aware resource allocation for MapReduce in a cloud. In: SC, pp. 58:1–58:11 (2011)
11. Pinheiro, E., Bianchini, R., Enrique, V.C., Heath, T.: Load balancing and unbalancing for power and performance in cluster-based systems. In: Workshop on compilers and operating systems for low power, pp. 182– 195 (2001)
12. Chen, Y., Keys, L., Katz, R.H.: Towards Energy Efficient MapReduce. Technical Report of EECS Department University of California, Berkeley. Available via http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-109.pdf (2008)
13. Pinheiro, E., Bianchini, R., Carrera, E.V., Heath, T.: Dynamic cluster reconfiguration for power and performance: Compilers and operating systems for low power (book), Kluwer Academic Publishers Norwell, ISBN:1-4020-7573-1, 75–93 (2003)
14. Chen, Y., Alspaugh, S., Borthakur, D., Katz, R.: Energy Efficiency for Large-Scale MapReduce Workloads with Significant Interactive Analysis. In: 7th ACM Europan Conference on Computer System, pp. 43– 56 (2012)
15. Yigitbasi, N., Datta, K., Jain, N., Willke, T.: Energy efficient scheduling of MapReduce workloads on heterogeneous clusters. In: Green Computing Middleware (ACM), pp. 1–6 (2011)
16. Pinheiro, E., Bianchini, R.: Energy conservation techniques for disk array-based servers. ICS 2004, 68–78
17. Kaushik, R.T., Abdelzaher, T.F., Egashira, R., Nahrstedt, K.: Predictive data and energy management in GreenHDFS. IGCC 2011, 1–9
18. Colarelli, D., Grunwald, D.: Massive arrays of idle disks for storage archives. SC 2002, 1–11
19. Karger, D.R., Lehman, E., Leighton, F.T., Panigrahy, R., Levine, M.S., Lewin, D.: Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web. STOC 1997, 654–663
20. Brinkmann, A., Salzwedel, K., Scheideler, C.: Efficient, distributed data placement strategies for storage area networks (extended abstract). SPAA 2000, 119–128
21. Tao, C., Nong, X., Fang, L., et al.: Clustering-Based And consistent Hashing-Aware data placement algorithm. J. Softw. 21(12), 3175–3185 (2010). (in Chinese)
22. Yuan, D., Yang, Y., Liu, X., Chen, J.: A data placement strategy in scientific cloud workflows. Future Generation Comp. Syst. (FGCS) 26(8), 1200–1214 (2010)
23. Profiling Energy Usage for Efficient Consumption. https://msdn.microsoft.com/en-us/library/dd393312.aspx. 2016/6/1
24. Unixbench https://code.google.com/p/byte-unixbench/
25. Liu, Z.: Efficient, balanced data placement algorithm in scalable storage clusters. Journal of Communication and Computer, 2007, (7):8-17
26. Ronald, L.: Graham: Bounds on Multiprocessing Timing anoMalies. SIAM J. Appl. Math. (SIAMAM) 17(2), 416–429 (1969)
27. WordCount program: Available in Hadoop source distribution: https://src/examples/org/apache/hadoop/examples/ WordCount
28. Sort program: Available in Hadoop source distribution: https://src/examples/org/apache/hadoop/examples/sort
29. MRBench program: Available in Hadoop source distribution: https://src/examples/org/apache/hadoop/mapred/ MRBench
30. Jie, S., Li, T., Zhi, W., Zhiliang, Z.: Study on energy-consumption regularities of cloud computing systems by a novel evaluation model. Computing, 1–19 (2013)
31. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM (CACM) 51(1), 107–113 (2008)