Deadline-based MapReduce workload management

IEEE Transactions on Network and Service Management

Volumn 10, Issue 2, 2013, Pages 231-244

  Polo, Jordà ^a,b   Becerra, Yolanda ^a,b   Carrera, David ^a,b   Steinder, Malgorzata ^c   Whalley, Ian ^c   Torres, Jordi ^a,b   Ayguadé, Eduard ^a,b  

^a UNIVERSITAT POLITÈCNICA DE CATALUNYA   (Spain)

^b BARCELONA SUPERCOMPUTING CENTER   (Spain)

^c IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

MapReduce; performance management; task scheduling

Indexed keywords

DYNAMIC ENVIRONMENTS; EXECUTION ENVIRONMENTS; MAP-REDUCE; PERFORMANCE MANAGEMENT; RELATIVE PERFORMANCE; SCHEDULING TECHNIQUES; SPECIALIZED PROCESSORS; TASK-SCHEDULING;

DIGITAL STORAGE; HARDWARE; SCHEDULING;

INFORMATION MANAGEMENT;

EID: 84879109590     PISSN: 19324537     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNSM.2012.122112.110163     Document Type: Article

References (27)

1. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," in Proc. 2004 Symposium on Operating System Design and Implementation, pp. 137-150.
2. B. Sharma, V. Chudnovsky, J. L. Hellerstein, R. Rifaat, and C. R. Das, "Modeling and synthesizing task placement constraints in Google compute clusters," in Proc. 2011 ACM Symposium on Cloud Computing, pp. 3:1-3:14.
3. Los Alamos National Laboratory, HPC: Roadrunner. Available: \url{http://www.lanl.gov/roadrunner/}
4. B.-G. Chun, G. Iannaccone, G. Iannaccone, R. Katz, G. Lee, and L. Niccolini, "An energy case for hybrid datacenters," in 2009 Power Aware Computing and Systems.
5. M. Gschwind, P. Hofstee, B. Flachs, M. Hopkins, Y. Watanabe, and T. Yamazaki, "A novel SIMD architecture for the cell heterogeneous chip-multiprocessor," 2005.
6. Y Becerra, V. Beltran, D. Carrera, M. Gonzalez, J. Torres, and E. Ayguadé, "Speeding up distributed MapReduce applications using hardware accelerators," in Proc. 2009 IEEE International Conference on Parallel Processing, pp. 42-49.
7. J. Polo, D. Carrera, Y. Becerra, M. Steinder, and I. Whalley, "Performance-driven task co-scheduling for MapReduce environments," in Proc. 2010 IEEE/IFIP Network Operations and Management Symposium, pp. 373-380.
8. J. Polo, D. Carrera, Y. Becerra, V. Beltran, J. Torres, and E. Ayguade, "Performance management of accelerated MapReduce workloads in heterogeneous clusters," in Proc. 2010 International Conference on Parallel Processing, pp. 653-662.
9. Apache Software Foundation, Hadoop MapReduce Tutorial. Available: http://hadoop.apache.org/mapreduce/
10. Sun Microsystems, Inc, Java Native Interface. Available: \url{http://java.sun.com/docs/books/jni}
11. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Job scheduling for multi-user Map-Reduce clusters," Tech. Rep. UCB/EECS-2009-55, 2009.
12. "Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," in Proc. 2010 European Conference on Computer Cystems, pp. 265-278.
13. J. Polo, C. Castillo, D. Carrera, Y Becerra, M. Steinder, I. Whalley, J. Torres, and E. Ayguadé, "Resource-aware adaptive scheduling for MapReduce clusters," in 2011 ACM/IFIP/USENIX International Middleware Conference.
14. D. Carrera, M. Steinder, I. Whalley, J. Torres, and E. Ayguadé, "Enabling resource sharing between transactional and batch workloads using dynamic application placement," in 2008 ACM/IFIP/USENIX International Conference on Middleware.
15. J. Polo, "Adaptive Scheduler," 2009. Available: https://issues.apache.org/jira/browse/MAPREDUCE-1380
16. D. G. Feitelson and L. Rudolph, "Parallel job scheduling: issues and approaches," in JSSPP, 1995, pp. 1-18.
17. P. R. Jelenkovic, X. Kang, and J. Tan, "Adaptive and scalable comparison scheduling," in 2007 SIGMETRICS.
18. A. Wierman and M. Nuyens, "Scheduling despite inexact job-size information," in Proc. 2008 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pp. 25-36.
19. Apache Hadoop On Demand. Available: {\scriptsizehttp://hadoop. apache.org/core/docs/r0. 20.0/hod \-user\-guide.html}
20. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," in 2007 ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, pp. 59-72.
21. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: fair scheduling for distributed computing clusters," in 2009 ACM SOSP.
22. J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh, K.-L. Wu, and A. Balmin, "FLEX: a slot allocation scheduling optimizer for MapReduce workloads," in Middleware 2010, vol. 6452, pp. 1-20.
23. A. Verma, L. Cherkasova, and R. H. Campbell, "ARIA: automatic resource inference and allocation for MapReduce environments," 2011 IEEE International Conference on Autonomic Computing.
24. T. Sandholm and K. Lai, "MapReduce optimization using regulated dynamic prioritization," in Proc. 2009 International Joint Conference on Measurement and Modeling of Computer Systems, pp. 299-310.
25. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving MapReduce performance in heterogeneous environments," in Proc. 2008 USENIX Conference on Operating Systems Design and Implementation, pp. 29-42.
26. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris, "Reining in the outliers in Map-Reduce clusters using Mantri," in Proc. 2010 USENIX Conference on Operating Systems Design and Implementation, pp. 1-16.
27. A. C. Murthy, C. Douglas, M. Konar, O. O'Malley, S. Radia, S. Agarwal, and V. K V, Architecture of next generation Apache Hadoop MapReduce framework. Available: \url{https://issues.apache.org/jira/secure/attachment/12486023/}