High-performance RDMA-based design of hadoop MapReduce over infiniband

Proceedings - IEEE 27th International Parallel and Distributed Processing Symposium Workshops and PhD Forum, IPDPSW 2013

Volumn , Issue , 2013, Pages 1908-1917

  Wasi Ur Rahman, Md ^a   Islam, Nusrat Sharmin ^a   Lu, Xiaoyi ^a   Jose, Jithin ^a   Subramoni, Hari ^a   Wang, Hao ^a   Panda, Dhabaleswar K Dk ^a  

^a Ohio State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMMUNICATION FRAMEWORK; DATA-INTENSIVE APPLICATION; DESIGN ELEMENTS; HADOOP MAPREDUCE; PARAMETER-TUNING; PERFORMANCE IMPROVEMENTS; PROGRAMMING MODELS; REMOTE DIRECT MEMORY ACCESS;

FAULT TOLERANCE;

EID: 84899739143     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPSW.2013.238     Document Type: Conference Paper

References (24)

1. Y. Wang, X. Que, W. Yu, D. Goldenberg, and D. Sehgal, "Hadoop Acceleration through Network Levitated Merge," in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC '11, 2011.
2. Apache Hadoop, http://hadoop.apache.org/.
3. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," in Operating Systems Design and Implementation (OSDI), 2004.
4. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System," in IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010.
5. J. Appavoo, A. Waterland, D. Da Silva, V. Uhlig, B. Rosenburg, E. Van Hensbergen, J. Stoess, R. Wisniewski, and U. Steinberg, "Providing A Cloud Network Infrastructure on A Supercomputer," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC '10. New York, NY, USA: ACM, 2010, pp. 385-394.
6. Greenplum Analytics Workbench, http://www.greenplum.com/news/greenplum- Analyticsworkbench.
7. S. Sur, H. Wang, J. Huang, X. Ouyang, and D. K. Panda, "Can High Performance Interconnects Benefit Hadoop Distributed File System" in Workshop on Micro Architectural Support for Virtualization, Data Center Computing, and Clouds, in Conjunction with MICRO 2010, Atlanta, GA, 2010.
8. J. Jose, H. Subramoni, M. Luo, M. Zhang, J. Huang, M. W. Rahman, N. S. Islam, X. Ouyang, H. Wang, S. Sur, and D. K. Panda, "Memcached Design on High Performance RDMA Capable Interconnects," in International Conference on Parallel Processing (ICPP), Sept 2011.
9. J. Huang, X. Ouyang, J. Jose, M.W. Rahman, H.Wang, M. Luo, H. Subramoni, C. Murthy, and D. K. Panda, "High-Performance Design of HBase with RDMA over InfiniBand," in IEEE International Parallel and Distributed Processing Symposium (IPDPS'12).
10. N. S. Islam, M. W. Rahman, J. Jose, R. Rajachandrasekar, H. Wang, H. Subramoni, C. Murthy, and D. K. Panda, "High Performance RDMA-based Design of HDFS over InfiniBand," in The International Conference for High Performance Computing, Networking, Storage and Analysis (SC), November 2012.
11. J. Jose, M. Luo, S. Sur, and D. K. Panda, "Unifying UPC and MPI Runtimes: Experience with MVAPICH," in Fourth Conference on Partitioned Global Address Space Programming Model (PGAS), Oct 2010.
12. Hadoop Map Reduce, "The Apache Hadoop Project," http://hadoop.apache.org/mapreduce/.
13. Sort, http://wiki.apache.org/hadoop/Sort.
14. RandomWriter, http://wiki.apache.org/hadoop/RandomWriter.
15. Infiniband Trade Association, http://www.infinibandta. org.
16. OpenFabrics Alliance, http://www.openfabrics.org/.
17. P. Balaji, H. V. Shah, and D. K. Panda, "Sockets vs RDMA Interface over 10-Gigabit Networks: An In-depth analysis of the Memory Traffic Bottleneck," in Workshop on Remote Direct Memory Access (RDMA): Applications, Implementations, and Technologies (RAIT), in conjunction with IEEE Cluster, 2004.
18. RDMA Consortium, "Architectural Specifications for RDMA over TCP/IP," http://www.rdmaconsortium.org/.
19. B. Fitzpatrick, "Distributed Caching with Memcached," Linux Journal, vol. 2004, pp. 5-, August 2004. [Online]. Available: http://portal.acm.org/citation.cfm id=1012889.1012894
20. Apache HBase, "The Apache Hadoop Project," http://hbase.apache. org/.
21. MVAPICH2: MPI over InfiniBand, 10GigE/iWARP and RoCE, http://mvapich.cse.ohio-state.edu/.
22. Mellanox Technologies, "Unstructured Data Accelerator," http://www.mellanox.com/page/products dyn product family=144.
23. S. Seo, I. Jang, K. Woo, I. Kim, J.-S. Kim, and S. Maeng, "HPMR: Prefetching and Pre-shuffling in Shared MapReduce Computation Environment," in Cluster Computing and Workshops, 2009. CLUSTER '09. IEEE International Conference on, Sep. 2009, pp. 1-8.
24. X. Lu, B. Wang, L. Zha, and Z. Xu, "Can MPI Benefit Hadoop and MapReduce Applications" in IEEE 40th International Conference on Parallel Processing Workshops (ICPPW), 2011.