Maestro: Replica-aware map scheduling for MapReduce

Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012

Volumn , Issue , 2012, Pages 435-442

  Ibrahim, Shadi ^a,b   Jin, Hai ^a   Lu, Lu ^a   He, Bingsheng ^c   Antoniu, Gabriel ^b   Wu, Song ^a  

^a HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

^b INRIA   (France)

^c NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

cloud computing; Hadoop; MapReduce; replication; scheduling

Indexed keywords

DATA DISTRIBUTION; DATA-INTENSIVE COMPUTING; DISTRIBUTED FRAMEWORK; EXECUTION TIME; HADOOP; INPUT DATAS; LOCAL MAP; MAP TASK; MAP-REDUCE; NETWORK TRAFFIC; NODE-BASED; NOVEL SCHEDULING ALGORITHMS; PERFORMANCE DEGRADATION; PROGRAMMING MODELS; REMOTE DATA; REPLICATION; RESEARCH EFFORTS; RUN-TIME SCHEDULING; SPECULATIVE EXECUTION;

CLOUD COMPUTING; COMPUTER PROGRAMMING; DEGRADATION; GRID COMPUTING; INPUT OUTPUT PROGRAMS;

SCHEDULING;

EID: 84863659799     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CCGrid.2012.122     Document Type: Conference Paper

References (27)

1. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks,"in Proceedings of the 2nd ACM European Conference on Computer Systems (EuroSys '07), Lisbon, Portugal, 2007, pp. 59-72.
2. J. Dean and S. Ghemawat, "Mapreduce: simplified data processing on large clusters," in Proceedings of the 6th USENIX conference on Symposium on Opearting Systems Design & Implementation (OSDI '04), San Francisco, CA, USA, 2004, pp. 137-150.
3. The Apache Hadoop Project, http://www.hadoop.org.
4. Amazon Elastic Compute Cloud (EC2), http://aws.amazon.com/ec2/.
5. D. Gottfrid, "Self-service, Prorated Super Computing Fun!"http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super- computing-fun/.
6. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, and L. Qi, "Cloudlet: towards mapreduce implementation on virtual machines," in Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing (HPDC-18), Garching, Germany, 2009, pp. 65-66.
7. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving mapreduce performance in heterogeneous environments,"in Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI'08), San Diego, California, 2008, pp. 29-42.
8. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig latin: a not-so-foreign language for data processing," in Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (SIGMOD'08), Vancouver, Canada, 2008, pp. 1099-1110.
9. B. He, M. Yang, Z. Guo, R. Chen, B. Su, W. Lin, and L. Zhou, "Comet: batched stream processing for data intensive distributed computing," in Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC'10), Indianapolis, Indiana, USA, 2010, pp. 63-74.
10. W. Fang, B. He, Q. Luo, and N. K. Govindaraju, "Mars: Accelerating mapreduce with graphics processors," IEEE Trans. Parallel Distrib. Syst., Vol. 22, pp. 608-620, 2011.
11. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google file system,"SIGOPS - Operating Systems Review, Vol. 37, No. 5, pp. 29-43, 2003.
12. Y. Jégou, S. Lantéri, J. Leduc, N. Melab, G. Mornet, R. Namyst, P. Primet, B. Quetier, O. Richard, E.-G. Talbi, and T. Iréa, "Grid'5000: a large scale and highly reconfigurable experimental Grid testbed."International Journal of High Performance Computing Applications, Vol. 20, No. 4, pp. 481-494, 2006.
13. Grid 5000 Project, https://www.grid5000.fr/mediawiki/index.php.
14. Xensource Homepage,, http://www.xensource.com/.
15. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, "Xen and the art of virtualization," in Proceedings of the nineteenth ACM Symposium on Operating Systems Principles (SOSP'03), Bolton Landing, NY, USA, 2003, pp. 164-177.
16. R. Jain, D.-M. Chiu, and W. Hawe, "A quantitative measure of fairness and discrimination for resource allocation in shared computer systems," DEC Research Report TR-301, 1984.
17. S. Ibrahim, H. Jin, L. Lu, B. He, and S. Wu, "Adaptive disk i/o scheduling for mapreduce in virtualized environment," in Proceedings of the 2011 International Conference on Parallel Processing (ICPP'11), Taipei, Taiwan, 2011, pp. 335-344.
18. B. He, M. Yang, Z. Guo, R. Chen, W. Lin, B. Su, H. Wang, and L. Zhou, "Wave computing in the cloud," in Proceedings of the 12th USENIX Workshop on Hot Topics in Operating Systems (HotOS'09), Monte Verità, Switzerland, 2009.
19. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears, "Mapreduce online," in Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation (NSDI'10), San Jose, California, 2010.
20. S. Seo, I. Jang, K. Woo, I. Kim, J.-S. Kim, and S. Maeng, "Hpmr: Prefetching and pre-shuffling in shared mapreduce computation environment,"in Proceedings of the 2009 IEEE International Conference on Cluster Computing (CLUSTER'07), New Orleans, Louisiana, USA.
21. S. Ibrahim, H. Jin, L. Lu, S. Wu, B. He, and L. Qi, "Leen: Locality/fairness-aware key partitioning for mapreduce in the cloud,"in Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CLOUDCOM'10), Indianapolis, USA, 2010, pp. 17-24.
22. S. Ibrahim, B. He, and H. Jin, "Towards pay-as-you-consume cloud computing," in Proceedings of the 2011 IEEE International Conference on Services Computing (SCC'11), Washington, DC, USA, 2011, pp. 370-377.
23. J. Xie, S. Yin, X. Ruan, Z. Ding, Y. Tian, J. Majors, A. Manzanares, and X. Qin, "Improving mapreduce performance through data placement in heterogeneous hadoop clusters," in Proceedings of the 24th IEEE International Symposium on Parallel and Distributed Workshop (IPDPSw'10), Atlanta, Georgia, USA, 2010, pp. 1-9.
24. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris, "Scarlett: coping with skewed content popularity in mapreduce clusters," in Proceedings of the sixth ACM European Conference on Computer Systems (EuroSys'11), Salzburg, Austria, 2011, pp. 287-300.
25. H. Wang, Q. Jing, R. Chen, B. He, Z. Qian, and L. Zhou, "Distributed systems meet economics: pricing in the cloud," in Proceedings of the 2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud'10), Boston, MA, 2010.
26. N. Chohan, C. Castillo, M. Spreitzer, M. Steinder, A. Tantawi, and C. Krintz, "See spot run: using spot instances for mapreduce workflows," in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, Boston, MA, 2010.
27. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," in Proceedings of the 5th ACM European Conference on Computer Systems (EuroSys'10), Paris, France, 2010, pp. 265-278.