Jockey: Guaranteed job latency in data parallel clusters

EuroSys'12 - Proceedings of the EuroSys 2012 Conference

Volumn , Issue , 2012, Pages 99-112

  Ferguson, Andrew D ^a   Bodik, Peter ^b   Kandula, Srikanth ^b   Boutin, Eric ^c   Fonseca, Rodrigo ^a  

^a BROWN UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

^c MICROSOFT   (United States)

Author keywords

Data parallel; Deadline; Dryad; Dynamic adaptation; MapReduce; Scheduling; SLO

Indexed keywords

DATA PARALLEL; DEADLINE; DRYAD; DYNAMIC ADAPTATIONS; MAP-REDUCE; SLO;

DATA PROCESSING; SCHEDULING;

RESOURCE ALLOCATION;

EID: 84860561660     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2168836.2168847     Document Type: Conference Paper

References (27)

1. G. M. Amdahl. Validity of the single processor approach to achieving large scale computing capabilities. In Proc. AFIPS '67 (Spring), pages 483-485, New York, NY, USA, 1967.
2. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the Outliers in Map-Reduce Clusters using Mantri. In Proc. OSDI '10, Vancouver, Canada, 2010.
3. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris. Scarlett: Coping with Skewed Content Popularity in MapReduce Clusters. In Proc. EuroSys '11, Salzburg, Austria, 2011.
4. E. Anderson, D. Beyer, K. Chaudhuri, T. Kelly, N. Salazar, C. Santos, R. Swaminathan, R. Tarjan, J. Wiener, and Y. Zhou. Value-maximizing deadline scheduling and its application to animation rendering. In Proc. SPAA '05, Las Vegas, Nevada, USA, 2005.
5. D. Borthakur, J. Gray, J. S. Sarma, K. Muthukkaruppan, N. Spiegelberg, H. Kuang, K. Ranganathan, D. Molkov, A. Menon, S. Rash, R. Schmidt, and A. Aiyer. Apache Hadoop goes Realtime at Facebook. In Proc. SIGMOD '11, Athens, Greece, 2011.
6. R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and Efcient Parallel Processing of Massive Data Sets. Proc. VLDB, 1(2):1265-1276, 2008.
7. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: Easy, efcient data-parallel pipelines. In Proc. PLDI '10, Toronto, Ontario, Canada, 2010.
8. J. Dean and S. Ghemawat. MapReduce: Simplifed data processing on large clusters. Commun. ACM, 51(1):107-113, 2008.
9. S. Ghemawat, H. Gobiof, and S.-T. Leung. Te Google fle system. SIGOPS Oper. Syst. Rev., 37(5):29-43, 2003.
10. H. Herodotou, F. Dong, and S. Babu. No One (Cluster) Size Fits All: Automatic Cluster Sizing for Data-intensive Analytics. In Proc. SoCC '11, Cascais, Portugal, 2011.
11. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed Data-parallel Programs from Sequential Building Blocks. In Proc. EuroSys '07, Lisbon, Portugal, 2007.
12. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair Scheduling for Distributed Computing Clusters. In Proc. SOSP '09, Big Sky, Montana, USA, 2009.
13. A. Li, X. Yang, S. Kandula, and M. Zhang. CloudCmp: Comparing public cloud providers. In Proc. IMC '10, Melbourne, Australia, 2010.
14. C. Lumezanu, S. Bhola, and M. Astley. Online optimization for latency assignment in distributed real-time systems. In Proc. ICDCS '08, June 2008.
15. K. Morton, M. Balazinska, and D. Grossman. ParaTimer: a progress indicator for MapReduce DAGs. In Proc. SIGMOD '10, Indianapolis, IN, USA, 2010.
16. D. G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand. CIEL: a universal execution engine for distributed data-fow computing. In Proc. NSDI'11, Boston, MA, 2011.
17. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In Proc. SIGMOD '08, Vancouver, Canada, 2008.
18. B. Ravindran, P. Kachroo, and T. Hegazy. Adaptive resource management in asynchronous real-time distributed systems using feedback control functions. In Proc. of 5th Symposium on Autonomous Decentralized Systems, 2001.
19. T. Sandholm and K. Lai. Prediction-based enforcement of performance contracts. In Proc. GECON'07, Rennes, France, 2007.
20. T. Sandholm and K. Lai. A statistical approach to risk mitigation in computational markets. In Proc. HPDC '07, Monterey, CA, USA, 2007.
21. T. Sandholm and K. Lai. MapReduce optimization using regulated dynamic prioritization. In Proc. SIGMETRICS '09, Seattle, WA, USA, 2009.
22. A. Tusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckof, and R. Murthy. Hive: a warehousing solution over a map-reduce framework. Proc. VLDB Endow., 2: 1626-1629, August 2009.
23. B. Urgaonkar, P. Shenoy, A. Ch, and P. Goyal. Dynamic provisioning of multi-tier internet applications. In Proc. ICAC '05, 2005.
24. A. Verma, L. Cherkasova, and R. H. Campbell. SLO-Driven Right-Sizing and Resource Provisioning of MapReduce Jobs. In Proc. LADIS '11, 2011.
25. G. Wang and T. Ng. Te Impact of Virtualization on Network Performance of Amazon EC2 Data Center. In Proc. IEEE INFOCOM '10, March 2010.
26. D. Xuan, R. Bettati, J. Chen, W. Zhao, and C. Li. Utilization-Based Admission Control for Real-Time Applications. In Proc. ICPP '10, Washington, DC, USA, 2000.
27. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. In Proc. EuroSys '10, Paris, France, 2010.