Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters

Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

Volumn , Issue , 2013, Pages

  Cho, Brian ^a,c   Rahman, Muntasir ^a   Chajed, Tej ^a   Gupta, Indranil ^a   Abad, Cristina ^a,b   Roberts, Nathan ^b   Lin, Philbert ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b YAHOO INC   (United States)

^c Samsung   (United States)

Author keywords

Deadlines; Hadoop; Mapreduce; Priorities; Scheduling

Indexed keywords

CLOUD COMPUTING;

DEADLINES; DESIGN AND EVALUATIONS; HADOOP; HARD REAL-TIME; MAP-REDUCE; MAPREDUCE CLUSTERS; PRIORITIES; SYNTHETIC WORKLOADS;

SCHEDULING;

EID: 84893335405     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2523616.2523624     Document Type: Conference Paper

References (58)

1. A. Adya, J. Dunagan, and A. Wolman. Centrifuge: Integrated lease management and partitioning for cloud services. In Proc. Usenix Symposium on Networked Systems Design and Implementation (NSDI), 2010.
2. S. Agarwal, A. Panda, B. Mozafari, H. Milner, S. Madden, and I. Stoica. BlinkDB: queries with bounded errors and bounded response times on very large data. In Proc. ACM European Conference on Computer Systems (Eurosys), 2013.
3. M. Amirijoo, J. Hansson, and S. Son. Algorithms for managing QoS for real-time data services using imprecise computation. Real-Time and Embedded Computing, pages 136-157, 2004.
4. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. PACMan: coordinated memory caching for parallel jobs. In Proc. Usenix Symposium on Networked Systems Design and Implementation (NSDI), 2012.
5. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in map-reduce clusters using Mantri. In Proc. Usenix Symposium on Operating Systems Design and Implementation (OSDI), 2010.
6. G. Anantharanayanan, C. Douglas, R. Ramakrishnan, S. Rao, and I. Stoica. True elasticity in multitenant clusters through Amoeba. In Proc. ACM Symposium on Cloud Computing (SoCC), 2012.
7. A. Andrzejak, D. Kondo, and S. Yi. Decision model for cloud computing under SLA constraints. In Proc. IEEE Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 257-266, 2010.
8. O. Babaoglu, K. Marzullo, and F. B. Schneider. A formalization of priority inversion. Real-time Systems, 5(4): 285-303, Oct. 1993.
9. B. Calder, A. O. J. Wang, N. Nilakantan, A. Skjolsvold, S. McKelvie, Y. Xu, S. Srivastav, J. Wu, H. Simitci, J. Haridas, C. Uddaraju, H. Khatri, A. Edwards, V. Bedekar, S. Mainali, R. Abbasi, A. Agarwal, M. F. ul Haq, M. I. ul Haq, D. Bhardwaj, S. Dayanand, A. Adusumilli, M. McNett, S. Sankaran, K. Manivannan, and L. Rigas. Windows Azure Storage: A highly available cloud storage service with strong consistency. In Proc. ACM Symposium on Operating Systems Principles (SOSP), pages 143-157, 2011.
10. Y. Chen, A. Ganapathi, R. Griffith, and R. Katz. The case for evaluating Mapreduce performance using workload suites. In Proc. IEEE Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 390-399, Jul. 2011.
11. L. Cheng, Q. Q. Zhang, and R. Boutaba. Mitigating the negative impact of preemption on heterogeneous Mapreduce workloads. In Proc. Conference on Network and Service Management (CNSM), pages 189-197, 2011.
12. M. Chowdhury, M. Zaharia, J. Ma, M. I. Jordan, and I. Stoica. Managing data transfers in computer clusters with Orchestra. In Proc. ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM), pages 98-109, 2011.
13. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sear. Mapreduce online. In Proc. Usenix Symposium on Networked Systems Design and Implementation (NSDI), 2010.
14. F. Cruz, F. Maia, M. Matos, R. Oliveira, J. Paulo, J. Pereira, and R. Vilaca. MeT: workload aware elasticity for NoSQL. In Proc. ACM European Conference on Computer Systems (Eurosys), 2013.
15. S. Das, S. Nishimura, and D. A. A. E. Abbadi. Albatross: lightweight elasticity in shared storage databases for the cloud using live data migration. In Proc. Conference on Very Large Data Bases (VLDB), pages 494-505, 2010.
16. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Communications of the ACM (CACM), 51: 107-113, Jan. 2008.
17. M. L. Dertouzos and A. K. Mok. Multiprocessor on-line scheduling of hard-real-time tasks. IEEE Transactions on Software Engineering, 15(12), 1989.
18. Emulab. http://emulab.net.
19. A. D. Ferguson, P. Bodik, S. Kandula, E. Boutin, and R. Fonseca. Jockey: guaranteed job latency in data parallel clusters. In Proc. ACM European Conference on Computer Systems (EuroSys), pages 99-112, 2013.
20. A. Ganapathi, Y. Chen, A. Fox, R. Katz, and D. Patterson. Statistics-driven workload modeling for the cloud. In Proc. International Workshop on Self Managing Database Systems (SMDB), pages 87-92, 2010.
21. J. Goossens, S. Funk, and S. Baruah. Priority-driven scheduling of periodic task systems on multiprocessors. Real-time systems, 2(3), 2003.
22. Hadoop. http://hadoop.apache.org/.
23. Hadoop Capacity Scheduler. http://hadoop.apache.org/docs/stable/capacity- scheduler.html, 2013.
24. Fair Scheduler. http://hadoop.apache.org/docs/stable/fair-scheduler.html, 2013.
25. Y. Huai, R. Lee, S. Zhang, C. H. Xia, and X. Zhang. Dot: a matrix model for analyzing, optimizing and deploying software for big data analytics in distributed systems. In Proc. ACM Symposium on Cloud Computing (SoCC), 2011.
26. IBM Infosphere Platform. http://www-01.ibm.com/software/data/infosphere/, 2013.
27. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair Scheduling for Distributed Computing Clusters. In Proc. of ACM Symposium on Operating Systems Principles (SOSP), pages 261-276, 2009.
28. HADOOP-5726 JIRA: Remove pre-emption from the capacity scheduler code base. https://issues.apache.org/jira/browse/HADOOP-5726, 2009.
29. MAPREDUCE-5269 JIRA: Preemption of Reducer (and Shuffle) via checkpointing. https://issues.apache.org/jira/browse/MAPREDUCE-5269, 2013.
30. K. Kambatla, A. Pathak, and H. Pucha. Towards optimizing Hadoop provisioning in the cloud. In Proc. Usenix Workshop on Hot Topics in Cloud Computing (HotCloud), 2009.
31. S. Y. Ko, I. Hoque, B. Cho, and I. Gupta. Making cloud intermediate data fault-tolerant. In Proc. Symposium on Cloud Computing (SoCC), pages 181-192, 2010.
32. A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica. Mesos: a platform for fine-grained resource sharing in the data center. In Proc. Usenix Symposium on Networked Systems Design and Implementation (NSDI), 2011.
33. J. Lin and C. Dyer. Data-intensive text processing with Mapreduce. Morgan & Claypool Publishers, 2010.
34. C. L. Liu and J. W. Layland. Scheduling algorithms for multiprogramming in a hard-real-time environment. Journal of the ACM (JACM), 20: 46-61, Jan. 1993.
35. J. S. Liu. Real-time systems. Prentice Hall, 2000.
36. K. Morton, M. Balazinska, and D. Grossman. Paratimer: a progress indicator for Mapreduce DAGs. In Proc. ACM SIGMOD International Conference on Management of Data (SIGMOD), pages 507-518, 2010.
37. Netty. http://netty.io/, 2013.
38. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a not-so-foreign language for data processing. In Proc. of ACM SIGMOD International Conference on Management of Data (SIGMOD), pages 1099-1110, 2008.
39. Oozie. http://oozie.apache.org/, 2013.
40. L. Phan, Z. Zhang, B. Loo, and I. Lee. Real-time MapReduce scheduling. Technical Report MS-CIS-10-32, University of Pennsylvania, Feb. 2010.
41. R. Power and J. Li. Piccolo: Building fast, distributed programs with partitioned tables. In Proc. Usenix Symposium on Operarting Systems Design and Implementation (OSDI), 2010.
42. Z. Qian, Y. He, C. S. Z. Wu, H. Zhu, T. Zhang, L. Zhou, Y. Yu, and Z. Zhang. Reliable stream computation in the cloud. In Proc. ACM European Conference on Computer Systems (Eurosys), 2013.
43. S. Rao, R. Ramakrishnan, A. Silberstein, M. Ovsiannikov, and D. Reeves. Sailfish: a framework for large scale data processing. In Proc. ACM Symposium on Cloud Computing (SoCC), 2012.
44. T. Sandholm and K. Lai. Dynamic proportional share scheduling in Hadoop. In Proc. Conference on Job Scheduling Strategies for Parallel Processing (JSSPP), pages 110-131, 2010.
45. I. Santhoshkumar, G. Manimaran, and C. S. R. Murthy. A pre-run-time scheduling algorithm for object-based distributed real-time systems. Journal of Systems Architecture, 45(14), 1999.
46. M. Schwarzkopf, A. Konwinski, M. A. el Malek, and J. Wilkes. Omega: Flexible, scalable schedulers for large compute clusters. In Proc. ACM European Conference on Computer Systems (Eurosys), 2013.
47. D. Shue, M. J. Freedman, and A. Shaikh. Performance isolation and fairness for multi-tenant cloud storage. In Proc. Usenix Symposium on Operating Systems Design and Implementation (OSDI), pages 349-362, 2012.
48. Spark: Lightning-fast cluster computing. http://spark.incubator.apache. org/, 2013.
49. Storm: Distributed and fault-tolerant realtime computation. http://storm-project.net/, 2013.
50. A. S. Tanenbaum. The Operating System as a Resource Manager. In Modern Operating Systems, chapter 1.1.2. Pearson Prentice Hall, 3rd edition, 2008.
51. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: a warehousing solution over a Mapreduce framework. Proc. Very Large Data Base Endowment (PVLDB), 2: 1626-1629, Aug. 2009.
52. A. Verma, L. Cherkasova, and R. H. Campbell. ARIA: automatic resource inference and allocation for mapreduce environments. In Proc. International Conference on Autonomic Computing (ICAC), 2011.
53. A. Wang, S. Venkataraman, S. Alspaugh, R. Katz, and I. Stoica. Cake: enabling high-level SLOs on shared storage systems. In Proc. ACM Symposium on Cloud Computing (SoCC), 2012.
54. B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad, M. Newbold, M. Hibler, C. Barb, and A. Joglekar. An integrated experimental environment for distributed systems and networks. In Proc. Usenix Symposium on Operating Systems Design and Implementation (OSDI), pages 255-270, 2002.
55. A. Wieder, P. Bhatotia, A. Post, and R. Rodrigues. Orchestrating the deployment of computations in the cloud with Conductor. In Proc. Usenix Symposium on Networked Systems Design and Implementation (NSDI), 2012.
56. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language. In Proc. Usenix Symposium on Operating Systems Design and Implementation (OSDI), pages 1-14, 2008.
57. M. Zaharia, A. Konwinski, A. Joseph, R. Katz, and I. Stoica. Improving Mapreduce performance in heterogeneous environments. In Proc. Usenix Symposium on Operating Systems Design and Implementation (OSDI), pages 29-42, 2008.
58. M. Zaharia, D. B. J. S. Sarma, K. K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proc. ACM European Conference on Computer Systems (Eurosys), pages 265-278. ACM, 2010.