A MapReduce task scheduling algorithm for deadline constraints

Cluster Computing

Volumn 16, Issue 4, 2013, Pages 651-662

  Tang, Zhuo ^a   Zhou, Junqing ^a   Li, Kenli ^a   Li, Ruixuan ^b  

^a HUNAN UNIVERSITY   (China)

^b HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

Data locality; Deadline constraints; Hadoop; MapReduce; Scheduling algorithm

Indexed keywords

MULTITASKING; SCHEDULING; SCHEDULING ALGORITHMS;

CLASSIFICATION ALGORITHM; COMPUTING CAPACITY; DATA LOCALITY; DEADLINE CONSTRAINT; HADOOP; HETEROGENEOUS CLUSTERS; MAP-REDUCE; TASK-SCHEDULING ALGORITHMS;

CLUSTERING ALGORITHMS;

EID: 84888844489     PISSN: 13867857     EISSN: 15737543     Source Type: Journal    
DOI: 10.1007/s10586-012-0236-5     Document Type: Article

References (29)

1. Chu, C. T., Kim, S. K., Lin, Y. A., Yu, Y. Y., Bradski, G., Ng, A. Y., Olukotun, K.: Map-reduce for machine learning on multicore. http://www. cs. standford. edu/peop;e/ang//papers/nips06-mapreducemulticoure. pdf (2006). Accessed 1 March 2012.
2. Ekanayake, J., Pallickara, S., Fox, G.: MapReduce for data intensive scientific analyses. In: Proceedings of the 2008 IEEE Fourth International Conference on eScience, pp. 277-284 (2008). doi: 10. 1109/eScience. 2008. 59.
3. Mackey, G., Sehrish, S., Bent, J., Lopez, J., Habib, S., Wang, J.: Introducing map-reduce to high end computing. In: Proceedings of the 2008 3rd Patascale Data Storage Workshop, pp. 1-6 (2008). doi: 10. 1109/PDSW. 2008. 4811889.
4. Matsunaga, A., Tsugawa, M., Fortes, J.: CloudBLAST: combining MapReduce and virtualization on distributed resources for bioinformatics applications. In: Proceedings of 4th IEEE International Conference on eScience, pp. 222-229 (2008). doi: 10. 1109/eScience. 2008. 62.
5. Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems, vol. 41, pp. 59-72 (2007). doi: 10. 1145/1272996. 1273005.
6. Han, H., Jung, H., Eom, H., Yeom, H. Y.: Scatter-Gather-Merge: an efficient star-join query processing algorithm for data-parallel frameworks. Clust. Comput. 14(2), 183-197 (2010). doi: 10. 1007/s10586-010-01445.
7. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107-113 (2008). doi: 10. 1145/1327452. 1327492.
8. He, B., Luo, Q., Govindaraju, N. K.: Mars: accelerating MapReduce with graphics processors. IEEE Trans. Parallel Distrib. Syst. 22(4), 608-620 (2011). doi: 10. 1109/TPDS. 2010. 158.
9. Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating mapreduce for multi-core and multi-processor systems. In: IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13-24 (2007). doi: 10. 1109/HPCA. 2007. 346181.
10. The Apache Software Foundation: Hadoop (2012). http://hadoop. apache. org. Accessed 1 March 2012.
11. Palit, I., Reddy, C. K.: Scalable and parallel boosting with MapReduce. IEEE Trans. Knowl. Data Eng. 24, 1904-1916 (2012). doi: 10. 1109/TKDE. 2011. 208.
12. Verma, A., Cho, B.: Breaking the MapReduce stage barrier. In: 2010 IEEE International Conference on Cluster Computing, pp. 235-244 (2010). doi: 10. 1109/CLUSTER. 2010. 29.
13. Seo, S., Jang, I., Woo, K., Kim, I., Kim, J.-S., Maeng, S.: HPMR: prefetching and pre-shuffling in shared MapReduce computation environment. In: IEEE International Conference on Cluster Computing and Workshops, pp. 1-8 (2009). doi: 10. 1109/CLUSTR. 2009. 5289171.
14. Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R., Stoica, I.: Improving MapReduce performance in heterogeneous environments. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, pp. 29-42 (2008).
15. Zaharia, M., Borthakur, D., Sarma, J. S., Elmeleegy, K., Shenker, S., Stoica, I.: Job scheduling for multi-user MapReduce clusters. (2009). http://www. eecs. berkeley. edu/Pubs/TechRpts/2009/EECS-2009-55. pdf. Accessed 1 March 2012.
16. http://hadoop. apache. org/mapreduce/docs/r0. 21. 0/capacity_scheduler. html (2011). Accessed 1 March 2012.
17. Alexandraki, A., Paterakis, M.: Performance evaluation of the deadline credit scheduling algorithm for soft-real-time applications in distributed video-on-demand systems. Clust. Comput. 8(1), 61-75 (2005). doi: 10. 1007/s10586-004-4437-4.
18. Sandholm, T., Lai, K.: Dynamic proportional share scheduling in hadoop. In: Proceedings of the 15th Workshop on Job Scheduling Strategies for Parallel Processing, pp. 110-131 (2010).
19. Zaharia, M., Borthakur, D., Sarma, J. S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems, pp. 265-278 (2010).
20. Xie, J., Yin, S., Ruan, X. J., Ding, Z. Y., Tian, Y.: Improving MapReduce performance through data placement in heterogeneous hadoop clusters. In: IEEE International Symposium on Parallel & Distributed Processing, Workshops and PhdForum, pp. 1-9 (2010). doi: 10. 1109/IPDPSW. 2010. 5470880.
21. Zhang, X. H., Zhong, Z. Y., Feng, S. Z., Tu, B. B., Fan, J. P.: Improving data locality of MapReduce by scheduling in homogeneous computing environments. In: IEEE 9th International Symposium on Parallel and Distributed Processing with Applications, pp. 120-126 (2011). doi: 10. 1109/ISPA. 2011. 14.
22. Aboulnaga, A., Wang, Z., Zhang, Z. Y.: Packing the most onto your cloud. In: Proceedings of the First International Workshop on Cloud Data Management, pp. 25-28 (2009). doi: 10. 1145/1651263. 1651268.
23. Morton, K., Balazinska, M., Grossman, D.: Paratimer: a progress indicator for mapreduce DAGs. In: Proceedings of the 2010 International Conference on Management of Data, pp. 507-518 (2010). doi: 10. 1145/1807167. 1807223.
24. Kc, K., Anyanwu, K.: Scheduling hadoop jobs to meet deadlines. In: IEEE Second International Conference on Cloud Computing Technology and Science, pp. 388-392 (2010). doi: 10. 1109/CloudCom. 2010. 97.
25. Polo, J., Carrera, D., Becerra, Y., Torres, J., Ayguade, E., Steinder, M., Whalley, I.: Performance-driven task co-scheduling for MapReduce environments. In: IEEE Proceedings of Network Operations and Management Symposium, pp. 373-380 (2010). doi: 10. 1109/NOMS. 2010. 5488494.
26. Phan, L. T. X., Zhang, Z., Loo, B. T., Lee, I.: Real-time MapReduce scheduling. http://repository. upenn. edu/cgi/viewcontent. cgi?article=1988&context=cis_reports (2010). Accessed 1 March 2012.
27. Qin, X., Jiang, H., Manzanares, A., Ruan, X., Yin, S.: Dynamic load balancing for IO-intensive applications on clusters. ACM Trans. Storage 5(3), 1-38 (2009). doi: 10. 1145/1629075. 1629078.
28. Herodotou, H.: Hadoop Performance Models (2011). http://www. cs. duke. edu/starfish/files/hadoop-models. pdf. Accessed 1 March 2012.
29. Dong, X., Wang, Y., Liao, H.: Scheduling mixed real-time and non-real-time applications in MapReduce Environment. In: IEEE 17th International Conference on Parallel and Distributed Systems, pp. 9-16 (2011). doi: 10. 1109/ICPADS. 2011. 115.