A throughput optimal algorithm for map task scheduling in MapReduce with data locality

Performance Evaluation Review

Volumn 40, Issue 4, 2013, Pages 33-42

  Wang, Weina ^a   Zhu, Kai ^a   Ying, Lei ^a   Tan, Jian ^b   Zhang, Li ^b  

^a AZ 85281   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTING CLUSTERS; JOIN-THE-SHORTEST-QUEUE; LARGE-SCALE DATASETS; MAXIMIZE THROUGHPUT; MAXWEIGHT POLICIES; NON-PREEMPTIVE TASKS; STOCHASTIC NETWORKS; THROUGHPUT-OPTIMAL;

OPTIMIZATION; THROUGHPUT;

CLUSTERING ALGORITHMS;

EID: 84883467278     PISSN: 01635999     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2479942.2479947     Document Type: Article

References (13)

1. Hadoop. http://hadoop.apache.org.
2. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris. Scarlett: coping with skewed content popularity in mapreduce clusters. In Proc. European Conf Computer Systems (EuroSys), pages 287-300, Salzburg, Austria, 2011.
3. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. Pacman: coordinated memory caching for parallel jobs. In Proc. Conf. Networked Systems Design and Implementations (USENIX), pages 20-20, 2012.
4. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. ACM Commun., 51(1):107-113, Jan. 2008.
5. S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system. In Proc. ACM Symp. Operating Systems Principles (SOSP), pages 29-43, Bolton Landing, NY, 2003.
6. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: fair scheduling for distributed computing clusters. In Proc. ACM Symp. Operating Systems Principles (SOSP), pages 261-276, Big Sky, MT, 2009.
7. S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan. An analysis of traces from a production mapreduce cluster. In Proc. IEEE/ACM Int. Conf. Cluster, Cloud and Grid Computing (CCGRID), pages 94-103, Melbourne, Australia, 2010.
8. S. T. Maguluri and R. Srikant. Scheduling jobs with unknown duration in clouds. In Proc. IEEE Int. Conf. Computer Communications (INFOCOM), Turin, Italy, 2013.
9. K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The hadoop distributed file system. In IEEE Symp. Mass Storage Systems and Technologies (MSST), pages 1-10, Incline Villiage, NV, May 2010.
10. L. Tassiulas and A. Ephremides. Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. IEEE Trans. Autom. Control, 4:1936-1948, Dec. 1992.
11. L. Tassiulas and A. Ephremides. Dynamic server allocation to parallel queues with randomly varying connectivity. IEEE Trans. Inf. Theory, 39:466-478, Mar. 1993.
12. T. White. Hadoop: The definitive guide. Yahoo Press, 2010.
13. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In Proc. European Conf. Computer Systems (EuroSys), pages 265-278, Paris, France, 2010.