Joint optimization of overlapping phases in MapReduce

Performance Evaluation

Volumn 70, Issue 10, 2013, Pages 720-735

  Lin, Minghong ^a   Zhang, Li ^b   Wierman, Adam ^a   Tan, Jian ^b  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Job scheduling; MapReduce; Overlapping tandem queues

Indexed keywords

JOB SCHEDULING; MAP-REDUCE; MULTIPLE PROCESSING; OVERLAPPING PHASIS; RESOURCE AUGMENTATION; RESOURCE UTILIZATIONS; SCALABLE PARALLEL COMPUTING; TANDEM QUEUE;

DATA PROCESSING; IMAGE MATCHING; OPTIMIZATION; PARALLEL ARCHITECTURES; SCHEDULING;

CLUSTERING ALGORITHMS;

EID: 84884704082     PISSN: 01665316     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.peva.2013.08.013     Document Type: Article

References (43)

1. Fair Scheduler, http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair- scheduler.html.
2. M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly, Dryad: distributed data-parallel programs from sequential building blocks, in: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, EuroSys'07, 2007.
3. J. Dean, and S. Ghemawat MapReduce: simplified data processing on large clusters Communications of the ACM 51 2008 107 113
4. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, E. Harris, Reining in the outliers in map-reduce clusters using Mantri, in: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, OSDI'10, 2010, pp. 1-16.
5. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg Quincy: fair scheduling for distributed computing clusters Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles SOSP'09 2009 ACM 261 276
6. M. Zaharia, D. Borthakur, J.S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica Job scheduling for multi-user MapReduce clusters, Tech. Rep. UCB/EECS-2009-55 2009 EECS Department, University of California Berkeley
7. A. Verma, L. Cherkasova, R. Campbell, Two sides of a coin: optimizing the schedule of MapReduce jobs to minimize their makespan and improve cluster performance, in: IEEE 20th International Symposium on Modeling, Analysis Simulation of Computer and Telecommunication Systems, MASCOTS, 2012.
8. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica PACMan: coordinated memory caching for parallel jobs Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation NSDI'12 2012 USENIX Association Berkeley, CA, USA
9. B. Palanisamy, A. Singh, L. Liu, B. Jain, Purlieus: locality-aware resource allocation for MapReduce in a cloud, in: Proceedings of the International Conference on Supercomputing, Seattle, WA, 2011.
10. Z. Guo, G. Fox, M. Zhou, Investigation of data locality in MapReduce, in: IEEE International Symposium on Cluster Computing and the Grid, 2012, pp. 419-426.
11. M. Hammoud, M. Sakr, Locality-aware reduce task scheduling for MapReduce, in: 2011 IEEE Third International Conference on Cloud Computing Technology and Science, CloudCom, 2011, pp. 570-576.
12. J. Tan, X. Meng, and L. Zhang Delay tails in MapReduce scheduling Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems SIGMETRICS'12 2012 ACM New York, NY, USA 5 16
13. T. Bressoud, M. Kozuch, Cluster fault-tolerance: an experimental evaluation of checkpointing and MapReduce through simulation, in: Cluster Computing and Workshops, CLUSTER'09, 2009.
14. M. Zaharia, A. Konwinski, A.D. Joseph, R. Katz, and I. Stoica Improving MapReduce performance in heterogeneous environments Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation OSDI'08 2008 USENIX Association Berkeley, CA, USA 29 42 URL: http://dl.acm.org/citation.cfm? id=1855741.1855744
15. S. Kavulya, J. Tan, R. Gandhi, P. Narasimhan, An analysis of traces from a production MapReduce cluster, in: Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, CCGRID'10, Washington, DC, USA, 2010, pp. 94-103.
16. J. Dai, J. Huang, S. Huang, B. Huang, Y. Liu, Hitune: dataflow-based performance analysis for big data cloud, in: Proceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference, USENIXATC'11, Berkeley, CA, USA, 2011, p. 7.
17. Capacity Scheduler, http://hadoop.apache.org/mapreduce/docs/r0.21.0/ capacity-scheduler.html.
18. G. Ananthanarayanan, C. Douglas, R. Ramakrishnan, S. Rao, I. Stoica, True elasticity in multi-tenant data-intensive compute clusters, in: Proceedings of the Third ACM Symposium on Cloud Computing, SoCC'12, 2012, pp. 24:1-24:7.
19. R.W. Wolff Stochastic Modeling and Theory of Queues 1997 Prentice Hall
20. L.A. Hall Approximability of flow shop scheduling Proceedings of the 36th Annual Symposium on Foundations of Computer Science FOCS'95 1995 IEEE Computer Society Washington, DC, USA 82 URL: http://dl.acm.org/citation.cfm?id=795662. 796282
21. B. Moseley, A. Dasgupta, R. Kumar, T. Sarlós, On scheduling in map-reduce and flow-shops, in: Proceedings of the 23rd ACM Symposium on Parallelism in Algorithms and Architectures, SPAA'11, 2011, pp. 289-298.
22. R. Stewart, and J. Singer Comparing fork/join and MapReduce Technical Report HW-MACS-TR-0096 2012 Department of Computer Science, Heriot-Watt University
23. E. Vianna, G. Comarela, T. Pontes, J. Almeida, V. Almeida, K. Wilkinson, H. Kuno, U. Dayal, Modeling the performance of the Hadoop online prototype, in: Proc. of Computer Architecture and High Performance Computing, 2011, pp. 152-159.
24. H.C. Zhao, C.H. Xia, Z. Liu, D. Towsley, A unified modeling framework for distributed resource allocation of general fork and join processing networks, in: Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, 2010.
25. L. Schrage A proof of the optimality of the shortest remaining processing time discipline Oper. Res. 16 3 1968 687 690 URL: http://www.jstor.org/stable/ 168596
26. S. Leonardi, and D. Raz Approximating total flow time on parallel machines Proceedings of the Twenty-Ninth Annual ACM Symposium on Theory of Computing STOC'97 1997 ACM New York, NY, USA 110 119 10.1145/258533.258562 URL: http://doi.acm.org/10.1145/258533.258562
27. J. Wilkes, More Google cluster data, Google research blog, November 2011, Posted at: http://googleresearch.blogspot.com/2011/11/more-google-cluster-data. html.
28. A. Wierman, M. Harchol-Balter, Classifying scheduling policies with respect to unfairness in an M/GI/1, in: Proc. of ACM SIGMETRICS, 2003.
29. D. Raz, H. Levy, B. Avi-Itzhak, A resource-allocation queueing fairness measure, in: Proc. of ACM Sigmetrics-Performance, 2004.
30. M. Gong, C. Williamson, Simulation evaluation of hybrid SRPT scheduling policies, in: Proc. of IEEE MASCOTS, 2004.
31. M. Gong, and C. Williamson Revisiting unfairness in web server scheduling Computer Networks 50 13 2006 2183 2203 (Pubitemid 43975021)
32. A. Wierman, M. Harchol-Balter, Classifying scheduling policies with respect to higher moments of conditional response time, in: Proc. of ACM SIGMETRICS, 2005.
33. A. Wierman Fairness and classifications ACM SIGMETRICS Performance Evaluation Review 34 4 2007 4 12
34. Hadoop, http://hadoop.apache.org.
35. N.M. van Dijk Queueing Networks and Product Forms - A Systems Approach Wiley-Interscience Series in Systems and Optimization 1993 Wiley
36. R. Graham, E. Lawler, J. Lenstra, and A. Kan Optimization and Approximation in Deterministic Sequencing and Scheduling: A Survey Annals of Discrete Mathematics vol. 5 1979 Elsevier 287 326
37. J.A. Hoogeveen, and T. Kawaguchi Minimizing total completion time in a two-machine flowshop: analysis of special cases Math. Oper. Res. 24 4 1999 887 910 (Pubitemid 30565507)
38. M.R. Garey, D.S. Johnson, and R. Sethi The complexity of flowshop and jobshop scheduling Math. Oper. Res. 1 2 1976 117 129
39. M.R. Garey, and D.S. Johnson Computers and Intractability: A Guide to the Theory of NP-Completeness 1979 W. H. Freeman & Co. New York, NY, USA
40. L. Cherkasova, Scheduling strategy to improve response time for web applications, in: Proceedings of the International Conference and Exhibition on High-Performance Computing and Networking, London, UK, 1998.
41. M. Harchol-Balter, B. Schroeder, N. Bansal, and M. Agrawal Size-based scheduling to improve web performance ACM Transactions on Computer Systems 21 2 2003
42. A. Verma, L. Cherkasova, R.H. Campbell, ARIA: automatic resource inference and allocation for MapReduce environments, in: Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC, 2011.
43. Z. Ren, X. Xu, J. Wan, W. Shi, and M. Zhou Workload characterization on a production Hadoop cluster: a case study on Taobao 2012 IEEE International Symposium on Workload Characterization IISWC 2012 IEEE 3 13