MARIANE: MApReduce implementation adapted for HPC environments

Proceedings - 2011 12th IEEE/ACM International Conference on Grid Computing, Grid 2011

Volumn , Issue , 2011, Pages 82-89

  Fadika, Zacharia ^a   Dede, Elif ^a   Govindaraju, Madhusudhan ^a   Ramakrishnan, Lavanya ^b  

^a State University of New York at Binghamton   (United States)

^b LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

Author keywords

Hadoop; MapReduce; MARIANE

Indexed keywords

DATA INTENSIVE; DISTRIBUTED ENVIRONMENTS; DISTRIBUTED FILE SYSTEMS; ENERGY RESEARCH; FILE SYSTEMS; HADOOP; MAGELLAN; MAP-REDUCE; MARIANE; OPEN SOURCE IMPLEMENTATION; PERFORMANCE GAIN; PROGRAMMING MODELS; TERAGRID;

COMPUTER PROGRAMMING; COMPUTER SOFTWARE; EQUIPMENT TESTING; OPEN SYSTEMS;

GRID COMPUTING;

EID: 83455264037     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/Grid.2011.20     Document Type: Conference Paper

References (30)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008.
2. Apache Hadoop. [Online]. Available: http://hadoop.apache.org
3. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The hadoop distributed file system," in Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on, May 2010, pp. 1-10.
4. National Energy Research Scientific Computing Center. [Online]. Available: http://www.nersc.gov
5. NYSGrid. [Online]. Available: www.nysgrid.org
6. Open Science Grid. [Online]. Available: http://www.opensciencegrid. org.
7. TeraGrid Information Services. [Online]. Available: http://info.teragrid. org/
8. S. R. Soltis, G. M. Erickson, K. W. Preslan, M. T. O'keefe, and T. M. Ruwart, "The global file system: A file system for shared disk storage,"1997.
9. R. Sandberg, D. Goldberg, S. Kleiman, D. Walsh, and B. Lyon, "Design and implementation or the sun network filesystem," 1985.
10. Lavanya Ramakrishnan, Piotr T Zbiegiel, Scott Campbell, Rick Bradshaw, Richard Shane Canon, Susan Coghlan, Iwona Sakrejda, Narayan Desai, Tina Declerck, Anping Liu, "Magellan: Experiences from a Science Cloud," in ScienceCloud 2011: 2nd Workshop on Scientific Cloud Computing, San Jose, California, USA, 2011.
11. Lustre File System. [Online]. Available: http://wiki.lustre.org/index. php/ Main Page
12. ORACLE Cluster File System. [Online]. Available: http://oss.oracle.com/ projects/ocfs/
13. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The google file system,"in Proceedings of the nineteenth ACM symposium on Operating systems principles, ser. SOSP '03. New York, NY, USA: ACM, 2003, pp. 29-43. [Online]. Available: http://doi.acm.org/10.1145/945445.945450
14. G. Mackey, S. Sehrish, J. Lopez, J. Bent, S. Habib, and J. Wang, "Introducing mapreduce to high end computing in Petascale Data," in Storage Workshop Held in conjunction with SC08.
15. Message Passing Interface Forum, "MPI: A Message-Passing Interface Standard," 1994.
16. Pervasive Technology Institute, Indiana University. [Online]. Available: http://www.iterativemapreduce.org/
17. Microsoft Research. [Online]. Available: http://research.microsoft.com/ en-us/projects/Dryad/
18. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, and L. Qi, "Cloudlet: towards mapreduce implementation on virtual machines," in HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing. New York, NY, USA: ACM, 2009, pp. 65-66.
19. F. Schmuck and R. Haskin, "Gpfs: A shared-disk file system for large computing clusters," in In Proceedings of the 2002 Conference on File and Storage Technologies (FAST, 2002, pp. 231-244.
20. Amazon, "Amazon Elastic Compute Cloud." [Online]. Available: http://aws.amazon.com/ec2
21. Z. Fadika and M. Govindaraju, "Lemo-mr: Low overhead and elastic mapreduce implementation optimized for memory and cpu-intensive applications," Cloud Computing Technology and Science, IEEE International Conference on, vol. 0, pp. 1-8, 2010.
22. Microsoft Research. [Online]. Available: http://www.microsoft.com/ windowsazure/
23. Google.com, "Google Web APIs," http://www.google.com/apis/.
24. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Z. 0002, S. Anthony, H. Liu, and R. Murthy, "Hive - a petabyte scale data warehouse using hadoop," in ICDE, 2010, pp. 996-1005.
25. AxisJava, "The Apache Project," 2002, http://ws.apache.org/ axis/.
26. M. R. Head, M. Govindaraju, A. Slominski, P. Liu, N. Abu-Ghazaleh, R. van Engelen, K. Chiu, and M. J. Lewis, "A Benchmark Suite for SOAP-based Communication in Grid Web Services," in SC '05: Proceedings of the 2005 ACM/IEEE conference on Supercomputing. Washington, DC, USA: IEEE Computer Society, 2005, p. 19.
27. W. Zhang and R. A. van Engelen, "A Table-Driven Streaming XML Parsing Methodology for High-Performance Web Services," in ICWS '06: Proceedings of the IEEE International Conference on Web Services. Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 197-204.
28. Y. Gu and R. L. Grossman, "Sector and sphere: the design and implementation of a high-performance data cloud." Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, vol. 367, no. 1897, pp. 2429-2445, June 2009. [Online]. Available: http://dx.doi.org/10.1098/rsta.2009.0053
29. Z. Fadika and M. Govindaraju, "Delma: Dynamically elastic mapreduce framework for cpu-intensive applications," in CCGRID, 2011, pp. 454-463.
30. G. Orenstein, "Digging Deeper Into Data With Hadoop," Available at http://gigaom.com/2009/06/07/digging-deeper-into-data-with-hadoop, 2009.