LEEN: Locality/fairness-aware key partitioning for MapReduce in the cloud

Proceedings - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010

Volumn , Issue , 2010, Pages 17-24

  Ibrahim, Shadi ^a   Jin, Hai ^a   Lu, Lu ^a   Wu, Song ^a   He, Bingsheng ^b   Qi, Li ^c  

^a HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c China Development Bank   (China)

Author keywords

Cloud computing; Hadoop; MapReduce; Partationing skew

Indexed keywords

DATA DISTRIBUTION; FAIR DISTRIBUTION; HADOOP; MAPREDUCE; NOVEL ALGORITHM; PARTATIONING SKEW; PERFORMANCE DEGRADATION; PERFORMANCE IMPROVEMENTS;

ALGORITHMS; COMPUTER SYSTEMS; DATA TRANSFER;

CLOUD COMPUTING;

EID: 79952402115     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CloudCom.2010.25     Document Type: Conference Paper

References (26)

1. J. Dean and S. Ghemawat, "Mapreduce: simplified data processing on large clusters", Proc. Usenix Symp. Opearting Systems Design & Implementation (OSDI 2004), Dec. 2004, pp.137-150.
2. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks", Proc. European Conf. Computer Systems (EuroSys 2007), ACM Press, Mar. 2007, pp.59-72.
3. B. S. He, W. B. Fang, Q. Luo, N. K. Govindaraju, and T. Y. Wang, "Mars: a MapReduce framework on graphics processors", Proc. conf. Parallel Architectures and Compilation Techniques (PACT 2008), ACM Press, Oct. 2008, pp.260-269.
4. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis, "Evaluating MapReduce for Multi-core and Multiprocessor Systems", Proc. Symp. High-Performance Computer Architecture (HPCA-13), ACM Press, Feb. 2008, pp.13-24.
5. Hadoop, http://lucene.apache.org/hadoop
6. Yahoo!, Yahoo! Developer Network, http://developer.yahoo.com/blogs/ hadoop/2008/02/yahoo-worlds-largest-production-hadoop.html
7. Hadoop, Applications powered by Hadoop: http://wiki.apache.org/hadoop/ PoweredB
8. S. Ghemawat, H. Gobioff, and S. T. Leung, "The Google file system", Proc. ACM Symp. Operating Systems Principles (SOSP 2003), ACM Press, Oct. 2003, pp.29-43.
9. T. Condie, N. Conway, P. Alvaro, M. J. Hellerstein, K. Elmeleegy, and R. Sears, "MapReduce Online", Proc. Usenix Symp. Networked Systems Design and Implementation (NSDI 2010), Apr. 2010. pp.313-328.
10. S. Seo, I. Jang, K. C. Woo, I. Kim, J. S. Kim, and S. Maeng, "HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment", Proc. IEEE Conf. Cluster Computing (Cluster 2009), IEEE Press, Aug. 2009, pp.1-8.
11. P. C. Chen, Y. L. Su, J. B. Chang, and C. K. Shieh, "Variable-Sized Map and Locality-Aware Reduce on Public-Resource Grids", Proc. Conf. Grid and Pervasive Computing (GPC 2010), May 2010, pp.234-243.
12. D. DeWitt and M. Stonebraker, "MapReduce: A major step backwards", http://databasecolumn.vertica.com/database-innovation/ mapreduce-a-major-step-backwards/, 2008.
13. J. D. DeWitt and J. Gary, "Parallel Database System: The Future of High Performance Database Systems", Commun. ACM, Vol.35, No.6, pp.85-98, June 1992.
14. X. Qiu, J. Ekanayake, S. Beason, T. Gunarathne, G. Fox, R. Barga, and D. Gannon, "Cloud technologies for bioinformatics applications", Proc. ACM Work. Many-Task Computing on Grids and Supercomputers (MTAGS 2009), ACM Press, Nov. 2009.
15. J. Lin, "The Curse of Zipf and Limits to Parallelization: A Look at the Stragglers Problem in MapReduce", Proc. Work. Large-Scale Distributed Systems for Information Retrieval (LSDS-IR'09), Jul. 2009.
16. Y. C. Kwon, M. Balazinska, B. Howe, and J. Rolia, "Skew-resistant parallel processing of feature-extracting scientific user-defined functions", Proc. ACM Symp. Cloud Computing (SOCC 2010), ACM Press, Jun. 2010, pp.75-86.
17. Iptables: http://en.wikipedia.org/wiki/Iptables.2009
18. R. Jain, D. Chiu, and W. Hawe, "A Quantitative Measure Of Fairness And Discrimination For Resource Allocation In Shared Computer Systems", DEC Research Report TR-301, September 1984.
19. Amazon Elastic Compute Cloud, http://aws.amazon.com/ec2/
20. Xen Hypervisor Homepage, http://www.xen.org/
21. S. Chen and S. W. Schlosser, "Map-Reduce Meets Wider Varieties of Applications", IRP-TR-08-05, Technical Report, Intel Research Pittsburgh, May 2008.
22. S. Ibrahim, H. Jin, B. Cheng, H. Cao, W. Song, and L. Qi, "Cloudlet: Towards MapReduce implementation on Virtual machines", Proc. ACM Symp. High Performance Distributed Computing (HPDC 2009), ACM Press, Jun. 2009, pp.65-66.
23. S. Ibrahim, H. Jin, L. Lu, L. Qi, S. Wu, and X. Shi, "Evaluating MapReduce on Virtual Machines: The Hadoop Case", Proc. Conf. Cloud Computing (CloudCom 2009), Springer LNCS, Dec 2009, pp.519-528.
24. G. Y. Wang, A. Butt, P. Pandey, and K. Gupta, "A Simulation Approach to Evaluating Design Decisions in MapReduce", Proc. IEEE Symp. Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS 2009), IEEE Press, Sep. 2009, pp.1-11.
25. B. S. He, M. Yang, Z. Y. Guo, R. S. Chen, W. Lin, B. Su, and L. D. Zhou, "Comet: Batched Stream Processing for Data Intensive Distributed Computing", Proc. ACM Symp. Cloud Computing (SOCC 2010), ACM Press, Jun. 2010, pp. 63-74.
26. B. S He, M. Yang, Z. Y. Guo, R. S. Chen, W. Lin, B. Su, H. Y. Wang, and L. D. Zhou, "Wave Computing in the Cloud", Proc. Work. Hot Topics in Operating Systems (HotOS 2009), May 2009.