Hopper: Decentralized speculation-aware cluster scheduling at scale

SIGCOMM 2015 - Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

Volumn , Issue , 2015, Pages 379-392

  Ren, Xiaoqi ^a   Ananthanarayanan, Ganesh ^b   Wierman, Adam ^a   Yu, Minlan ^c  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b MICROSOFT RESEARCH   (United States)

^c UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Decentralized scheduling; Fairness; Speculation; Straggler

Indexed keywords

CONVOLUTIONAL CODES; HOPPERS;

CLUSTER SCHEDULING; DECENTRALIZED SCHEDULING; FAIRNESS; JOB SCHEDULING; SPECULATION; SPECULATIVE EXECUTION; STATE OF THE ART; STRAGGLER;

SCHEDULING;

EID: 84962303167     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2785956.2787481     Document Type: Conference Paper

References (50)

1. Apache Thrift. https://thrift.apache.org/.
2. Cloudera Impala. http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html.
3. Hadoop. http://hadoop.apache.org.
4. Hadoop Capacity Scheduler. http://hadoop.apache.org/docs/r1.2.1/capacityscheduler.html.
5. Hadoop Distributed File System. http://hadoop.apache.org/hdfs.
6. Hadoop Slowstart. https://issues.apache.org/jira/browse/MAPREDUCE-1184/.
7. Hive. http://wiki.apache.org/hadoop/Hive.
8. Hopper Technical Report. https://sites.google.com/site/sigcommhoppertechreport/.
9. Sparrow. https://github.com/radlab/sparrow.
10. The Next Generation of Apache Hadoop Map Reduce. http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/.
11. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris. Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters. In EuroSys, 2011.
12. G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica. Effective Straggler Mitigation: Attack of the Clones. In USENIX NSDI, 2013.
13. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. PACMan: Coordinated Memory Caching for Parallel Jobs. In USENIX NSDI, 2012.
14. G. Ananthanarayanan, M. Hung, X. Ren, I. Stoica, A. Wierman, and M. Yu. GRASS: Trimming Stragglers in Approximation Analytics. In USENIX NSDI, 2014.
15. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, E. Harris, and B. Saha. Reining in the Outliers in Map-Reduce Clusters Using Mantri. In USENIX OSDI, 2010.
16. E. Bortnikov, A. Frank, E. Hillel, and S. Rao. Predicting Execution Bottlenecks in Map-Reduce Clusters. In USENIX HotCloud, 2012.
17. E. Boutin, J. Ekanayake, W. Kin, B. Shi, J. Zhou, Z. Qian, M. Wu, and L. Zhou. Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing. In USENIX OSDI, 2014.
18. M. Bramson, Y. Lu, and B. Prabhakar. Randomized load balancing with general service time distributions. In Proceedings of Sigmetrics, pages 275-286, 2010.
19. R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and efficient Parallel Processing of Massive Data Sets. Proceedings of the VLDB Endowment, (2), 2008.
20. R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and efficient Parallel Processing of Massive Datasets. In VLDB, 2008.
21. H. Chen, J. Marden, and A. Wierman. On the Impact of Heterogeneity and Back-end Scheduling in Load Balancing Designs. In INFOCOM. IEEE, 2009.
22. J. Dean. Achieving Rapid Response Times in Large Online Services. In Berkeley AMPLab Cloud Seminar, 2012.
23. J. Dean and L. Barroso. The Tail at Scale. Communications of the ACM, (2), 2013.
24. J. Dean and S. Ghemawat. Map Reduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 2008.
25. F. Dogar, T. Karagiannis, H. Ballani, and A. Rowstron. Decentralized Task-aware Scheduling for Data Center Networks. In ACM SIGCOMM, 2014.
26. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica. Dominant Resource Fairness: Fair Allocation of Multiple Resource Types. In USENIX NSDI, 2011.
27. R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella. Multi-Resource Packing for Cluster Schedulers. In ACM SIGCOMM, 2014.
28. M. Harchol-Balter, B. Schroeder, N. Bansal, and M. Agrawal. Size-based scheduling to improve web performance. ACM Transactions on Computer Systems (TOCS), 21(2):207-233, 2003.
29. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. Joseph, R. Katz, S. Shenker, and I. Stoica. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In USENIX NSDI, 2011.
30. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair Scheduling for Distributed Computing Clusters. In ACM SOSP, 2009.
31. M. Lin, L. Zhang, A. Wierman, and J. Tan. Joint Optimization of Overlapping Phases in Map Reduce. Performance Evaluation, 2013.
32. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: Interactive Analysis of Web-Scale Datasets. In VLDB, 2010.
33. B. Moseley, A. Dasgupta, R. Kumar, and T. Sarlos. On Scheduling in Map-reduce and Flow-shops. In ACM SPAA, 2011.
34. K. Ousterhout, A. Panda, J. Rosen, S. Venkataraman, R. Xin, S. Ratnasamy, S. Shenker, and I. Stoica. The Case for Tiny Tasks in Compute Clusters. In USENIX HotOS, 2013.
35. K. Ousterhout, R. Rasti, S. Ratnasamy, S. Shenker, and B. Chun. Making Sense of Performance in Data Analytics Frameworks. In USENIX NSDI, 2015.
36. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. Sparrow: Distributed, Low Latency Scheduling. In ACM SOSP, 2013.
37. K. Pruhs, J. Sgall, and E. Torng. Online scheduling. Handbook of scheduling: Algorithms, models, and performance analysis, pages 15-1, 2004.
38. A. Richa, M. Mitzenmacher, and R. Sitaraman. The power of two random choices: A survey of techniques and results. Combinatorial Optimization, 2001.
39. L. Schrage. A proof of the optimality of the shortest remaining processing time discipline. Operations Research, 16(3):687-690, 1968.
40. B. Sharma, V. Chudnovsky, J. L. Hellerstein, R. Rifaat, and C. R. Das. Modeling and Synthesizing Task Placement Constraints in Google Compute Clusters. In ACM SOCC, 2011.
41. J. Tan, X. Meng, and L. Zhang. Delay Tails in MapReduce Scheduling. ACM SIGMETRICS Performance Evaluation Review, 2012.
42. Y. Wang, J. Tan, W. Yu, L. Zhang, and X. Meng. Preemptive ReduceTask Scheduling for Fast and Fair Job Completion. USENIX ICAC, 2013.
43. A. Wierman. Fairness and scheduling in single server queues. Surveys in Operations Research and Management Science, 16(1):39-48, 2011.
44. A. Wierman and M. Harchol-Balter. Classifying scheduling policies with respect to unfairness in an m/gi/1. In ACM SIGMETRICS Performance Evaluation Review, volume 31, pages 238-249. ACM, 2003.
45. J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh, K. Wu, and A. Balmin. FLEX: A Slot Allocation Scheduling Optimizer for Map Reduce Workloads. In Middleware 2010. Springer, 2010.
46. N. Yadwadkar, G. Ananthanarayanan, and R. Katz. Wrangler: Predictable and Faster Jobs using Fewer Resources. In ACM SoCC, 2014.
47. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Job scheduling for multi-user mapreduce clusters. In UC Berkeley Technical Report UCB/EECS-2009-55, 2009.
48. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. In ACM EuroSys, 2010.
49. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In USENIX NSDI, 2012.
50. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving MapReduce Performance in Heterogeneous Environments. In USENIX OSDI, 2008.