High-Performance design of hadoop RPC with RDMA over InfiniBand

Proceedings of the International Conference on Parallel Processing

Volumn , Issue , 2013, Pages 641-650

  Lu, Xiaoyi ^a   Islam, Nusrat S ^a   Md , Wasi Ur Rahman ^a   Jose, Jithin ^a   Subramoni, Hari ^a   Wang, Hao ^a   Panda, Dhabaleswar K ^a  

^a Ohio State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMMUNICATION MECHANISMS; HIGH PERFORMANCE COMPUTING; HIGH PERFORMANCE NETWORKS; HIGH-PERFORMANCE DESIGN; INFINIBAND NETWORKS; INTEGRATED EVALUATION; LOW-LATENCY NETWORKS; PERFORMANCE EVALUATIONS;

BENCHMARKING;

EID: 84893300725     PISSN: 01903918     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICPP.2013.78     Document Type: Conference Paper

References (23)

1. Apache Hadoop, http://hadoop.apache.org/.
2. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters, " in Operating Systems Design and Implementation (OSDI), 2004.
3. T. White, Hadoop: The Definitive Guide. O'Reilly Media, Inc., Oct. 2010.
4. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System, " in IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010.
5. Apache HBase, http://hbase.apache.org.
6. N. S. Islam, M. W. Rahman, J. Jose, R. Rajachandrasekar, H. Wang, H. Subramoni, C. Murthy, and D. K. Panda, "High Performance RDMA-based Design of HDFS over InfiniBand, " in The International Conference for High Performance Computing, Networking, Storage and Analysis (SC), November 2012.
7. J. Huang, X. Ouyang, J. Jose, M. W. Rahman, H. Wang, M. Luo, H. Subramoni, C. Murthy, and D. K. Panda, "High-Performance Design of HBase with RDMA over InfiniBand, " in IEEE Int'l Parallel and Distributed Processing Symposium (IPDPS 2011), May 2011.
8. Y. Wang, X. Que, W. Yu, D. Goldenberg, and D. Sehgal, "Hadoop Acceleration Through Network Levitated Merge, " in Proceedings of the International Conference on High Performance Computing Networking, Storage and Analysis (SC'11), 2011.
9. M. W. Rahman, N. S. Islam, X. Lu, J. Jose, H. Subramoni, H. Wang, and D. K. Panda, "High-Performance RDMA-based Design of Hadoop MapReduce over InfiniBand, " in IEEE 27th International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2013.
10. M. Schatz, "CloudBurst: Highly Sensitive Short Read Mapping with MapReduce, " http://sourceforge.net/apps/mediawiki/cloudburstbio/index.php? title=CloudBurst.
11. The Apache Software Foundation, "The Apache MINA Project, " http://mina.apache.org.
12. X. Lu, M.W. Rahman, N. Islam, and D. K. Panda, "A Micro-benchmark Suite for Evaluating Hadoop RPC on High-Performance Networks, " in 3rd Workshop on Big Data Benchmarking (WBDB), 2013.
13. S. Ghemawat and P. Menage, "TCMalloc : Thread-Caching Malloc, " http://googperftools.sourceforge.net/doc/tcmalloc.html.
14. J. Jose, M. Luo, S. Sur, and D. K. Panda, "Unifying UPC and MPI Runtimes: Experience with MVAPICH, " in Fourth Conference on Partitioned Global Address Space Programming Model (PGAS), Oct 2010.
15. J. Jose, H. Subramoni, M. Luo, M. Zhang, J. Huang, M. W. Rahman, N. S. Islam, X. Ouyang, H. Wang, S. Sur, and D. K. Panda, "Memcached Design on High Performance RDMA Capable Interconnects, " in International Conference on Parallel Processing (ICPP), Sept 2011.
16. B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, "Benchmarking Cloud Serving Systems with YCSB, " in The Proceedings of the ACM Symposium on Cloud Computing (SoCC 2010), Indianapolis, Indiana, June 10-11 2010.
17. M. W. Rahman, J. Huang, J. Jose, X. Ouyang, H. Wang, N. Islam, H. Subramoni, C. Murthy, and D. K. Panda, "Understanding the Communication Characteristics in HBase: What are the Fundamental Bottlenecks?" in IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), April 2012.
18. J. Maassen, R. V. Nieuwpoort, R. Veldema, H. Bal, T. Kielmann, C. Jacobs, and R. Hofman, "Efficient Java RMI for Parallel Programming, " ACM Transaction on Programming Languages and Systems, vol. 23, no. 6, pp. 747-775, November 2001.
19. N. A. Ghazaleh, M. J. Lewis, and M. Govindaraju, "Differential Serialization for Optimized SOAP Performance, " in Proceedings of the 13th International Symposium on High-Performance Distributed Computing (HPDC-13), 2004.
20. N. A. Ghazaleh and M. J. Lewis, "Differential Deserialization for Optimized SOAP Performance, " in Proceedings of the International Conference on High Performance Computing Networking, Storage and Analysis (SC'05), 2005.
21. X. Lu, B. Wang, L. Zha, and Z. Xu, "Can MPI Benefit Hadoop and MapReduce Applications?" in IEEE 40th International Conference on Parallel Processing Workshops (ICPPW), 2011.
22. S. Sur, H. Wang, J. Huang, X. Ouyang, and D. K. Panda, "Can High Performance Interconnects Benefit Hadoop Distributed File System?" in Workshop on Micro Architectural Support for Virtualization, Data Center Computing, and Clouds, in Conjunction with MICRO 2010, Atlanta, GA, December 5 2010.
23. "Hadoop-RDMA: High-Performance Design of Hadoop over RDMA-enabled Interconnects, " http://hadoop-rdma.cse.ohio-state.edu.