Can MPI benefit Hadoop and MapReduce applications?

Proceedings of the International Conference on Parallel Processing Workshops

Volumn , Issue , 2011, Pages 371-379

  Lu, Xiaoyi ^a,b   Wang, Bing ^a,b   Zha, Li ^a   Xu, Zhiwei ^a  

^a INSTITUTE OF COMPUTING TECHNOLOGY   (China)

^b UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

Hadoop; MapReduce; MPI; MPI D

Indexed keywords

APPLICATION PERFORMANCE; BENCHMARK TESTS; COMMUNICATION INFRASTRUCTURE; COMMUNICATION OVERHEADS; COMMUNICATION PRIMITIVES; DATA-INTENSIVE COMPUTING; HADOOP; HIGH-PERFORMANCE COMPUTING; MAP-REDUCE; MESSAGE PASSING INTERFACE; MPI; MPI-D; SPEED-UPS; TECHNICAL CHALLENGES;

BENCHMARKING; COMMUNICATION; COMPUTER SOFTWARE SELECTION AND EVALUATION; COST REDUCTION; MESSAGE PASSING; STANDARDS;

MULTIPROCESSING SYSTEMS;

EID: 80155191154     PISSN: 15302016     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICPPW.2011.56     Document Type: Conference Paper

References (18)

1. R. E. Bryant, "Data intensive scalable computing: Harnessing the power of cloud computing," http://www.cs.cmu.edu/~bryant/pubdir/ discoverview09.pdf, Feb. 2009, revised manuscript (2009) based on his talk at the 2007 Federated Computing Research Conference.
2. Y. He, R. Lee, Y. Huai, Z. Shao, N. Jain, X. Zhang, and Z. Xu, "RCFile: a fast and space-efficient data placement structure in MapReduce-based warehouse systems," in Proceedings of the 27th IEEE International Conference on Data Engineering (ICDE 2011). IEEE, Apr. 2011, pp. 1199-1208.
3. "Apache hadoop," http://hadoop.apache.org/.
4. T. White, Hadoop: The Definitive Guide. O'Reilly Media, Inc., Oct. 2010.
5. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," Commun. ACM, vol. 51, no. 1, pp. 107-113, Jan. 2008.
6. "Message passing interface (MPI) forum home page," http://www.mpi-forum.org/.
7. W. Gropp, E. Lusk, N. Doss, and A. Skjellum, "A highperformance, portable implementation of the MPI message passing interface standard." Parallel Computing, vol. 22, no. 6, pp. 789-828, Sep. 1996.
8. "MPICH2 : High-performance and widely portable MPI," http://www.mcs.anl.gov/research/projects/mpich2/.
9. R. Thakur, R. Rabenseifner, and W. Gropp, "Optimization of collective communication operations in MPICH," International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49-66, 2005.
10. E. Gabriel, G. Fagg, G. Bosilca, T. Angskun, J. Dongarra, J. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R. Castain, D. Daniel, R. Graham, and T. Woodall, "Open MPI: goals, concept, and design of a next generation MPI implementation," in Recent Advances in Parallel Virtual Machine and Message Passing Interface (Euro PVM/MPI 2004), ser. Lecture Notes in Computer Science, D. Kranzlmller, P. Kacsuk, and J. Dongarra, Eds. Berlin / Heidelberg: Springer, 2004, vol. 3241, pp. 97-104.
11. K. Hwang and Z. Xu, Scalable Parallel Computing: Technology, Architecture, Programming. NewYork, NY, USA: McGraw-Hill, Inc., 1998.
12. "Jetty WebServer," http://jetty.codehaus.org/jetty/.
13. "Gridmix," http://hadoop.apache.org/mapreduce/docs/current/ gridmix.html.
14. T. Hoefler, A. Lumsdaine, and J. Dongarra, "Towards efficient MapReduce using MPI," in Recent Advances in Parallel Virtual Machine and Message Passing Interface (Euro PVM/MPI 2009), ser. Lecture Notes in Computer Science, M. Ropo, J. Westerholm, and J. Dongarra, Eds. Berlin, Heidelberg: Springer, Sep. 2009, vol. 5759, pp. 240-249.
15. S. J. Plimpton and K. D. Devine, "MapReduce in MPI for large-scale graph algorithms," A special issue of Parallel Computing, pp. 1-39, 2011.
16. "MapReduce-MPI library," http://www.sandia.gov/~sjplimp/ mapreduce.html.
17. S. Sur, H. Wang, J. Huang, X. Ouyang, and D. K. Panda, "Can High-Performance interconnects benefit hadoop distributed file system?" in Proceedings of the Workshop on Micro Architectural support for Virtualization, Data Center Computing, and Clouds (MASVDC'10), in Conjunction with the 43rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO 2010), Atlanta, GA, USA, Dec. 2010.
18. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S. Bae, J. Qiu, and G. Fox, "Twister: a runtime for iterative MapReduce," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (HPDC 2010). Chicago, Illinois: ACM, 2010, pp. 810-818.