Shufflewatcher: Shuffle-aware scheduling in Multi-tenant MapReduce clusters

Proceedings of the 2014 USENIX Annual Technical Conference, USENIX ATC 2014

Volumn , Issue , 2014, Pages 1-12

  Ahmad, Faraz ^a,c   Chakradhar, Srimat T ^b   Raghunathan, Anand ^c   Vijaykumar, T N ^c  

^a TERADATA CORPORATION   (United States)

^b NEC LABORATORIES AMERICA   (United States)

^c Purdue University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FAULT TOLERANCE;

ALL-REDUCE; BISECTION BANDWIDTH; CLUSTER PERFORMANCE; FAIRNESS CONSTRAINTS; MAP-REDUCE; MAPREDUCE CLUSTERS; MULTI TENANTS; MULTIPLE USER; PERFORMANCE; TURN-AROUND TIME;

SCHEDULING;

EID: 85077444173     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (43)

1. H. Abu-Libdeh et al. Symbiotic routing in future data centers. In Proceedings of SIGCOMM, 2010.
2. F. Ahmad et al. Tarazu: Optimizing MapReduce on heterogeneous clusters. In Proceedings of ASPLOS, 2012.
3. F. Ahmad et al. MapReduce with Communication Overlap (MaRCO). In JPDC, 2012.
4. M. Al-Fares, A. Loukissas, and A. Vahdat. A scalable, commodity data center network architecture. In Proceedings of SIGCOMM, 2008.
5. Amazon EC2. http://aws.amazon.com/ec2.
6. Amazon Elastic MapReduce (Amazon EMR). http://aws.amazon.com/elasticmapreduce/.
7. G. Ananthanarayanan et al. Reining in the outliers in Map-Reduce clusters using Mantri. In Proceedings of OSDI, 2010.
8. Y. Bu et al. The Haloop approach to large-scale iterative data analysis. The VLDB Journal, 21(2):169-190, Apr. 2012.
9. Y. Chen et al. The case for evaluating MapReduce performance using workload suites. In Proceedings of MASCOTS, 2011.
10. M. Chowdhury et al. Managing data transfers in computer clusters with orchestra. In Proceedings of SIGCOMM, pages 98-109, 2011.
11. T. Condie et al. MapReduce online. In Proceedings of NSDI, 2010.
12. P. Costa et al. Camdoop: exploiting in-network aggregation for big data applications. In Proceedings of NSDI, 2012.
13. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Commun. ACM, Jan. 2008.
14. J. Ekanayake et al. Twister: a runtime for iterative MapReduce. In Proceedings of HPDC, 2010.
15. N. Farrington et al. Helios: a hybrid electrical/optical switch architecture for modular data centers. In Proceedings of SIGCOMM, 2010.
16. A. Ghodsi et al. Dominant Resource Fairness: Fair allocation of multiple resource types. In Proceedings of NSDI, 2011.
17. A. Greenberg et al. VL2: A scalable and flexible data center network. In Proceedings of SIGCOMM, 2009.
18. C. Guo et al. Bcube: a high performance, server-centric network architecture for modular data centers. In Proceedings of SIGCOMM, 2009.
19. C. Guo et al. Dcell: a scalable and fault-tolerant network structure for data centers. In Proceedings of SIGCOMM, 2008.
20. L. Gyarmati and T. A. Trinh. Scafida: a scale-free network inspired data center architecture. SIGCOMM Comput. Commun. Rev., 40(5):4-12.
21. Hadoop. http://hadoop.apache.org.
22. Hadoop On Demand. http://hadoop.apache.org/docs/r0.18.3/hod_admin_guide.html.
23. B. Hindman et al. Mesos: a platform for fine-grained resource sharing in the data center. In Proceedings of NSDI, 2011.
24. M. Isard et al. Dryad: Distributed data-parallel programs from sequential building blocks. In Proceedings of EuroSys, 2007.
25. M. Isard et al. Quincy: fair scheduling for distributed computing clusters. In Proceedings of SOSP, 2009.
26. M. Li et al. Cam: a topology aware minimum cost flow based resource manager for MapReduce applications in the cloud. In Proceedings of HPDC, 2012.
27. Torque Resource Manager. http://www.adaptivecomputing.com/products/open-source/torque/.
28. R. Niranjan Mysore et al. Portland: a scalable fault-tolerant layer 2 data center network fabric. In Proceedings of SIGCOMM, 2009.
29. B. Palanisamy et al. Purlieus: locality-aware resource allocation for MapReduce in a cloud. In Proceedings of SC, 2011.
30. Hadoop Capacity Scheduler. http://hadoop.apache.org/docs/r0.20.2/capacity_scheduler.html.
31. Hadoop Fair Scheduler. http://hadoop.apache.org/docs/r0.20.2/fair_scheduler.html.
32. A. Shieh et al. Sharing the data center network. In Proceedings of NSDI, 2011.
33. A. Singla et al. Jellyfish: Networking data centers randomly. In Proceedings of NSDI, 2012.
34. D. Thain, T. Tannenbaum, and M. Livny. Distributed computing in practice: the Condor experience: Research Articles, UK, Feb. 2005. John Wiley and Sons Ltd.
35. A. Vahdat et al. Scale-Out Networking in the Data Center. IEEE Micro, 30:29-41, July 2010.
36. A. Verma et al. Breaking the MapReduce Stage Barrier. In Proceedings of IEEE CLUSTER, 2010.
37. D. Weld. Lecture notes on MapReduce(based on Jeff Dean's slides). http://rakaposhi.eas.asu.edu/cse494/notes/s07-mapreduce.ppt, 2007.
38. Yarn. http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html.
39. M. Zaharia et al. Delay Scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of EuroSys, 2010.
40. M. Zaharia et al. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In Proceedings of NSDI, 2012.
41. M. Zaharia et al. Improving MapReduce performance in heterogeneous environments. In Proceedings of OSDI, 2008.
42. J. Mudigonda, P. Yalagandula, and J. C. Mogul. Taming the flying cable monster: a topology design and optimization framework for data-center networks. In Proceedings of USE-NIX ATC, 2011.
43. S. Rao, et.al. Sailfish: a framework for large scale data processing. In Proceedings of SoCC, 2012.