DataMPI: Extending MPI to hadoop-like big data computing

Proceedings of the International Parallel and Distributed Processing Symposium, IPDPS

Volumn , Issue , 2014, Pages 829-838

  Lu, Xiaoyi ^a   Liang, Fan ^a   Wang, Bing ^a   Zha, Li ^a   Xu, Zhiwei ^a  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

Author keywords

Big Data; DataMPI; Hadoop; MapReduce; MPI

Indexed keywords

DISTRIBUTED COMPUTER SYSTEMS; DISTRIBUTED PARAMETER NETWORKS; ITERATIVE METHODS; SPECIFICATIONS;

DATAMPI; DISTRIBUTED COMPUTATIONS; EFFICIENT COMMUNICATIONS; HADOOP; HIGH PERFORMANCE COMPUTING; MAP-REDUCE; MPI; TIME CONSUMING TECHNIQUES;

BIG DATA;

EID: 84906654880     PISSN: 15302075     EISSN: 23321237     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2014.90     Document Type: Conference Paper

References (26)

1. "Message Passing Interface Forum," http://www.mpi-forum.org.
2. "MPICH2," http://www.mcs.anl.gov/research/projects/mpich2/.
3. "MVAPICH," http://mvapich.cse.ohio-state.edu/.
4. E. Gabriel, G. E. Fagg, G. Bosilca, T. Angskun, J. J. Dongarra, J. M. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R. H. Castain, D. J. Daniel, R. L. Graham, and T. S. Woodall, "Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation," in Proc. of the 11th European PVM/MPI Users' Group Meeting, ser. EuroMPI '04, Budapest, Hungary, 2004.
5. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008.
6. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears, "MapReduce Online," in Proc. of the 7th USENIX Conference on Networked Systems Design and Implementation, ser. NSDI '10, Berkeley, CA, USA, 2010.
7. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox, "Twister: A Runtime for Iterative MapReduce," in Proc. of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC '10, New York, NY, USA, 2010.
8. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-parallel Programs from Sequential Building Blocks," in Proc. of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems, ser. EuroSys '07, New York, NY, USA, 2007.
9. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster Computing with Working Sets," in Proc. of the 2nd USENIX Conference on Hot Topics in Cloud Computing, ser. HotCloud '10, Berkeley, CA, USA, 2010.
10. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari, "S4: Distributed Stream Computing Platform," in Proc. of the 2010 IEEE International Conference on Data Mining Workshops, ser. ICDMW '10, Sydney, Australia, 2010.
11. "The Apache Hadoop Project," http://hadoop.apache.org.
12. X. Lu, B. Wang, L. Zha, and Z. Xu, "Can MPI Benefit Hadoop and MapReduce Applications?" in Proc. of the 40th International Conference on Parallel Processing Workshops, ser. ICPPW '11, Taipei, China, 2011.
13. S. Brin and L. Page, "The Anatomy of A Large-scale Hypertextual Web Search Engine," in Proc. of the 7th International Conference on World Wide Web, ser. WWW7, Amsterdam, The Netherlands, 1998.
14. J. B. MacQueen, "Some Methods for Classification and Analysis of MultiVariate Observations," in Proc. of the 5th Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA, 1967.
15. Tony Hey, Stewart Tansley, and Kristin Tolle, The Fourth Paradigm: Data-Intensive Scientific Discovery. Redmond, Washington: Microsoft Research, 2009.
16. S. Ibrahim, H. Jin, L. Lu, S. Wu, B. He, and L. Qi, "LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud," in Proc. of the 2nd International Conference on Cloud Computing Technology and Science, ser. CLOUDCOM '10, Washington, DC, USA, 2010.
17. M. Hammoud and M. F. Sakr, "Locality-Aware Reduce Task Scheduling for MapReduce," in Proc. of the 3rd International Conference on Cloud Computing Technology and Science, ser. CLOUDCOM '11, Washington, DC, USA, 2011.
18. "mpiJava," http://www.hpjava.org/mpiJava.html.
19. "The Apache Mahout Project," http://mahout.apache.org.
20. T. Hoefler, A. Lumsdaine, and J. Dongarra, "Towards Efficient MapReduce Using MPI," in Proc. of the 16th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, ser. EuroMPI '09, Berlin, Heidelberg, 2009.
21. S. J. Plimpton and K. D. Devine, "MapReduce in MPI for Large-scale Graph Algorithms," Parallel Computing, vol. 37, no. 9, pp. 610-632, 2011.
22. Y. Wang, X. Que, W. Yu, D. Goldenberg, and D. Sehgal, "Hadoop Acceleration Through Network Levitated Merge," in Proc. of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC '11, New York, NY, USA, 2011.
23. M. Wasi-ur Rahman, N. S. Islam, X. Lu, J. Jose, H. Subramoni, H. Wang, and D. K. Panda, "High-Performance RDMA-based Design of Hadoop MapReduce over InfiniBand," in Proc. of the 27th International Symposium on Parallel and Distributed Processing Workshops and PhD Forum, ser. IPDPSW '13, Washington, DC, USA, 2013.
24. N. S. Islam, M. W. Rahman, J. Jose, R. Rajachandrasekar, H. Wang, H. Subramoni, C. Murthy, and D. K. Panda, "High Performance RDMA-based Design of HDFS over InfiniBand," in Proc. of the International Conference on High Performance Computing, Networking, Storage and Analysis, ser. SC '12, Los Alamitos, CA, USA, 2012.
25. X. Lu, N. S. Islam, M. Wasi-Ur-Rahman, J. Jose, H. Subramoni, H. Wang, and D. K. Panda, "High-Performance Design of Hadoop RPC with RDMA over InfiniBand," in Proc. of the 42nd International Conference on Parallel Processing, ser. ICPP '13, Lyon, France, 2013.
26. "The DataMPI Project," http://datampi.org.