MARIANE: Using MApReduce in HPC environments

Future Generation Computer Systems

Volumn 36, Issue , 2014, Pages 379-388

  Fadika, Zacharia ^a   Dede, Elif ^a   Govindaraju, Madhusudhan ^a   Ramakrishnan, Lavanya ^b  

^a BINGHAMTON UNIVERSITY   (United States)

^b LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

Author keywords

Data intensive; Hadoop; MapReduce; Scientific computing

Indexed keywords

COMPUTER SOFTWARE; NATURAL SCIENCES COMPUTING;

DATA INTENSIVE; DISTRIBUTED FILE SYSTEMS; HADOOP; HADOOP DISTRIBUTED FILE SYSTEM (HDFS); HIGH PERFORMANCE COMPUTING (HPC); MAP-REDUCE; MAPREDUCE FRAMEWORKS; PROGRAMMING MODELS;

FILE ORGANIZATION;

EID: 84899935089     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2013.12.007     Document Type: Article

References (33)

1. J. Dean, and S. Ghemawat MapReduce: simplified data processing on large clusters Commun. ACM 51 1 2008 107 113 10.1145/1327452.1327492
2. Apache Hadoop. URL http://hadoop.apache.org.
3. K. Shvachko, H. Kuang, S. Radia, R. Chansler, The hadoop distributed file system, in: Mass Storage Systems and Technologies, MSST, 2010 IEEE 26th Symposium on, 2010, pp. 1-10. http://dx.doi.org/10.1109/MSST.2010.5496972.
4. S.R. Soltis, G.M. Erickson, K.W. Preslan, M.T. O'keefe, T.M. Ruwart, The global file system: a file system for shared disk storage, 1997.
5. R. Sandberg, D. Goldberg, S. Kleiman, D. Walsh, B. Lyon, Design and implementation or the sun network filesystem, 1985.
6. Lustre file system. URL http://wiki.lustre.org/index.php/Main-Page.
7. National energy research scientific computing center. URL http://www.nersc.gov.
8. Z. Fadika, E. Dede, M. Govindaraju, and L. Ramakrishnan Mariane: mapreduce implementation adapted for hpc environments Proceedings of the 2011 IEEE/ACM 12th International Conference on Grid Computing GRID'11 2011 IEEE Computer Society Washington, DC, USA 82 89 10.1109/Grid.2011.20
9. S. Ghemawat, H. Gobioff, and S.-T. Leung The google file system Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles SOSP'03 2003 ACM New York, NY, USA 29 43 URL http://doi.acm.org/10.1145/945445. 945450 (Pubitemid 40929686)
10. G. Mackey, S. Sehrish, J. Lopez, J. Bent, S. Habib, J. Wang, Introducing mapreduce to high end computing, in: Petascale Data Workshop, held in conjunction with SC08.
11. Fermilab Computing Division, FermiGrid. URL http://fermigrid.fnal.gov/.
12. NYSGrid. URL www.nysgrid.org.
13. Open science grid. URL http://www.opensciencegrid.org.
14. TeraGrid information services. URL http://info.teragrid.org/.
15. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, and L. Qi Cloudlet: towards mapreduce implementation on virtual machines HPDC'09: Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing 2009 ACM New York, NY, USA 65 66 http://doi.acm.org/10.1145/1551609.1551624
16. F. Schmuck, R. Haskin, GPFS: a shared-disk file system for large computing clusters, in: Proceedings of the 2002 Conference on File and Storage Technologies, FAST, 2002, pp. 231-244.
17. Z. Fadika, M. Govindaraju, Lemo-mr: low overhead and elastic mapreduce implementation optimized for memory and cpu-intensive applications, in: IEEE International Conference on Cloud Computing Technology and Science, 2010, pp. 1-8.
18. Google.com, Google Web APIs. http://www.google.com/apis/.
19. A. Thusoo, J.S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, R. Murthy, Hive - a petabyte scale data warehouse using hadoop, in: ICDE [19], pp. 996-1005.
20. AxisJava, The Apache project, 2002. URL http://ws.apache.org/axis/.
21. M.R. Head, M. Govindaraju, A. Slominski, P. Liu, N. Abu-Ghazaleh, R. van Engelen, K. Chiu, and M.J. Lewis A benchmark suite for SOAP-based communication in grid web services SC'05: Proceedings of the 2005 ACM/IEEE Conference on Supercomputing 2005 IEEE Computer Society Washington, DC, USA 19 10.1109/SC.2005.2 (Pubitemid 44911002)
22. W. Zhang, and R.A. van Engelen A table-driven streaming XML parsing methodology for high-performance web services ICWS'06: Proceedings of the IEEE International Conference on Web Services 2006 IEEE Computer Society Los Alamitos, CA, USA 197 204 10.1109/ICWS.2006.15 (Pubitemid 351209854)
23. Z. Fadika, M. Govindaraju, S.R. Canon, L. Ramakrishnan, Evaluating hadoop for data-intensive scientific operations, in: IEEE CLOUD, 2012, pp. 67-74.
24. Z. Fadika, M.R. Head, M. Govindaraju, Parallel and distributed approach for processing large-scale XML datasets, in: 10th IEEE/ACM International Conference on Grid Computing, 2009, pp. 5-7.
25. Y. Gu, and R.L. Grossman Sector and sphere: the design and implementation of a high-performance data cloud Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 1897 2009 2429 2445 10.1098/rsta.2009.0053
26. L. Heshan, A. Ma, and M. Feng Moon: mapreduce on opportunistic environments HPDC'10: the ACM International Symposium on High Performance Distributed Computing 2010 ACM
27. D. Thain, and M. Livny Building reliable clients and servers I. Foster, C. Kesselman, The Grid: Blueprint for a New Computing Infrastructure 2003 Morgan Kaufmann
28. Amazon, Amazon elastic compute cloud. URL http://aws.amazon.com/ec2.
29. Microsoft research. URL http://www.microsoft.com/windowsazure/.
30. Pervasive Technology Institute, Indiana University. URL http://www.iterativemapreduce.org/.
31. Microsoft research. URL http://research.microsoft.com/en-us/projects/ Dryad/.
32. Towards Efficient MapReduce Using MPI, Vol. 5759 2009 Springer-Verlag Espoo, Finland
33. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M. Ranzato, A.W. Senior, P.A. Tucker, K. Yang, A.Y. Ng, Large scale distributed deep networks, in: NIPS, 2012, pp. 1232-1240.