Performance optimization for short MapReduce job execution in Hadoop

Proceedings - 2nd International Conference on Cloud and Green Computing and 2nd International Conference on Social Computing and Its Applications, CGC/SCA 2012

Volumn , Issue , 2012, Pages 688-694

  Yan, Jinshuang ^a   Yang, Xiaoliang ^a   Gu, Rong ^a   Yuan, Chunfeng ^a   Huang, Yihua ^a  

^a NANJING UNIVERSITY   (China)

Author keywords

job execution; MapReduce; parallel computing; performance optimization

Indexed keywords

COMMUNICATION MECHANISMS; DATA INTENSIVE; EVENT NOTIFICATION; EXECUTION PERFORMANCE; FUNDAMENTAL DESIGN; HADOOP MAPREDUCE; HIGH-THROUGHPUT; INSTANT MESSAGES; JOB EXECUTION; LARGE-SCALE DATASETS; MAP-REDUCE; OPTIMIZATION METHOD; PARALLEL COM- PUTING; PERFORMANCE LIMITATIONS; PERFORMANCE OPTIMIZATIONS; QUICK RESPONSE; SPEED UP; TASK ASSIGNMENT; TEST APPLICATIONS; TIME COST;

COMMUNICATION; OPTIMIZATION; PARALLEL ARCHITECTURES; PARALLEL PROCESSING SYSTEMS;

MULTIPROCESSING SYSTEMS;

EID: 84874620940     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CGC.2012.40     Document Type: Conference Paper

References (16)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, 2008, pp. 107.
2. MPI (Message Passing Interface). Available: http://wwwunix.mcs.anl.gov/ mpi/
3. PVM (Parallel Virtual Machine). Available: http://www.csm.ornl.gov/pvm/
4. Apahce Software Foundation, Hadoop, Available: http://hadoop.apache.org/
5. D. Gillick, A. Faria, and J. Denero, "Mapreduce: Distributed computing for machine learning," 2008. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/summary10.1.1.111.9204G
6. Fair Scheduler Guide. Available: http://hadoop.apache.org/common/docs/r0. 20.2/fair-scheduler.html
7. Matei Zaharia, Andy Konwinski, Anthony D.Joseph, Randy Katz, Ion Stoica. Improving MapReduce Performance in Heterogeneous Environments. 8th USENIX Symposium on Operating Systems Design and Implementation., p.29-42, December 08-10, 2008, San Diego, California.
8. Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, Ion Stoica. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. Proceedings of the 5th European conference on Computer systems. Pages: 265-278, 2010.
9. Sangwon Seo, Ingook Jang, Kyungchang Woo, Inkyo Kim, Jin-Soo Kim, Seungryoul Maeng. HPMR:Prefetching and Pre-Shuffling in Shared MapReduce Computation Environment. IEEE International Conference on Cluster Computing and Workshops, 2009
10. Shubin Zhang, Jizhong Han, Zhiyong Liu, Kai Wang, Shengzhong Feng. Accelerating MapReduce with Distributed Memory Cache. IEEE, 2009 15th International Conference on Parallel and Distributed Systems. 2009.
11. th International Conference on Data Engineering. 2010.
12. Shivnath Babu. Towards automatic optimization of MapReduce programs. International Conference on Management of Data, Proceedings of the 1st ACM symposium on Cloud computing. 2010.
13. R. Pike, S. Dorward, R. Griesemer, S. Quinlan. "Interpreting the Data: Parallel Analysis with Sawzall," Scientific Programming Journal, vol. 13, no. 4, Oct. 2005, pp. 227-298.
14. Apache Software Foundation, Hive, Available:http://hadoop.apache.org/ hive/.
15. I. S. Jacobs and C. P. Bean, "Fine particles, thin films and exchange anisotropy," in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271-350.
16. Xiao-Liang Yang, Yu-Long Liu, Chun-Feng Yuan, Yi-Hua Huang, "parallelization of BLAST with MapReduce for Long Sequence Alignment," International Symposium on Parallel Architectures, Algorithms and Programming (PAAP), 2011, pp.241-246.