MARISSA: MApReduce implementation for streaming science applications

2012 IEEE 8th International Conference on E-Science, e-Science 2012

Volumn , Issue , 2012, Pages

  Dede, E ^a   Fadika, Z ^a   Hartog, J ^a   Govindaraju, M ^a   Ramakrishnan, L ^b   Gunter, D ^b   Canon, R ^b  

^a BINGHAMTON UNIVERSITY   (United States)

^b LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALTERNATIVE FRAMEWORK; BIG DATUM; FILE SYSTEMS; HADOOP STREAMING; JAVA APPLICATIONS; MAP-REDUCE; MAPREDUCE MODELS; PERFORMANCE GAIN; SCIENCE APPLICATIONS; SCIENTIFIC APPLICATIONS; SCIENTIFIC DISCIPLINE; SCIENTIFIC SOFTWARES; SPACE EXPLORATIONS;

COMPUTER PROGRAMMING; COMPUTER SOFTWARE; FILE ORGANIZATION; PROTEIN FOLDING; SPACE RESEARCH;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 84873684979     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/eScience.2012.6404432     Document Type: Conference Paper

References (39)

1. Apache Hadoop. http://hadoop.apache.org.
2. European organization for nuclear research. http://public.web.cern.ch/ public/en/lhc/lhc-en.html.
3. Lustre File System.
4. Microsoft Research.
5. National Energy Research Scientific Computing Center. http://www.nersc.gov.
6. NYSGrid. www.nysgrid.org.
7. Open Science Grid. http://www.opensciencegrid.org.
8. Pervasive Technology Institute, Indiana University. http://www. iterativemapreduce.org/.
9. SALSA Group, Indiana University .
10. TeraGrid Information Services.
11. S. Ahern, A. Shoshani, K.-L. Ma, A. Choudhary, T. Critchlow, S. Kalasky, K.-L. Ma, V. Pascucci, J. Ahrens, E. W. Bethel, H. Childs, J. Huang, K. Joy, Q. Koziol, G. Lofstead, J. Meredith, K. Moreland, G. Ostrouchov, M. Papka, V. Vishwanath, M. Wold, N. Wright, and K. Wu. Scientific discovery at the exascale: Report from the doe ascr 2011 workshop on exascale data management, analysis, and visualization. http://www.olcf.ornl.gov/wp-content/uploads/2011/01/Exascale- ASCR-Analysis.pdf, 2011.
12. AMAZON. Amazon Elastic MapReduce.
13. Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst. Haloop: efficient iterative data processing on large clusters. Proc. VLDB Endow., 3(1-2):285-296, Sept. 2010.
14. J. B. Buck, N. Watkins, J. LeFevre, K. Ioannidou, C. Maltzahn, N. Polyzotis, and S. A. Brandt. Scihadoop: array-based query processing in hadoop. In SC, page 66, 2011.
15. S. Canon, S. Cholia, J. Shalf, K. Jackson, L. Ramakrishnan, and V. Markowitz. A performance comparison of massively parallel sequence matching computations on cloud computing platforms and hpc clusters using hadoop. In Using Clouds for Parallel Computations in Systems Biology Workshop, Held at SC09, 2009.
16. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Commun. ACM, 51(1):107-113, Jan. 2008.
17. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox. Twister: a runtime for iterative mapreduce. In HPDC, pages 810-818, 2010.
18. J. Ekanayake, S. Pallickara, and G. Fox. MapReduce for Data Intensive Scientific Analyses. In IEEE Fourth International Conference on eScience, pages 277-284, December 2008.
19. Z. Fadika, E. Dede, M. Govindaraju, and L. Ramakrishnan. Mariane: Mapreduce implementation adapted for hpc environments. Grid Computing, IEEE/ACM International Workshop on, 0:82-89, 2011.
20. Z. Fadika and M. Govindaraju. Lemo-mr: Low overhead and elastic mapreduce implementation optimized for memory and cpu-intensive applications. Cloud Computing Technology and Science, IEEE International Conference on, 0:1-8, 2010.
21. Z. Fadika and M. Govindaraju. Delma: Dynamically elastic mapreduce framework for cpu-intensive applications. In CCGRID, pages 454-463, 2011.
22. Z. Fadika, M. Govindaraju, S. Canon, and L. Ramakrishnan. Evaluting hadoop for data-intensive scientific operations. IEEE Cloud Computing, 2012.
23. S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP '03, pages 29-43, New York, NY, USA, 2003. ACM.
24. T. Gunarathne, B. Zhang, T.-L. Wu, and J. Qiu. Portable parallel programming on cloud and hpc: Scientific applications of twister4azure. In UCC, pages 97-104, 2011.
25. K. Gupta, R. Jain, I. Koltsidas, H. Pucha, P. Sarkar, M. Seaman, and D. Subhraveti. Gpfs-snc: An enterprise storage framework for virtual-machine clouds. IBM Journal of Research and Development, 55(6):2:1-2:10, nov.-dec. 2011.
26. B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang. Mars: a mapreduce framework on graphics processors. In Proceedings of the 17th international conference on Parallel architectures and compilation techniques, PACT '08, pages 260-269, New York, NY, USA, 2008. ACM.
27. I. Korf, M. Yandell, and J. Bedell. BLAST. O'Reilly & Associates, Inc., Sebastopol, CA, USA, 2003.
28. Lavanya Ramakrishnan, Piotr T Zbiegiel, Scott Campbell, Rick Bradshaw, Richard Shane Canon, Susan Coghlan, Iwona Sakrejda, Narayan Desai, Tina Declerck, Anping Liu. Magellan: Experiences from a Science Cloud. In ScienceCloud 2011: 2nd Workshop on Scientific Cloud Computing, San Jose, California, USA, 2011.
29. W. Lu, J. Jackson, and R. Barga. Azureblast: a case study of developing science applications on the cloud. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10, pages 413-420, New York, NY, USA, 2010. ACM.
30. G. Mackey, S. Sehrish, J. Lopez, J. Bent, S. Habib, and J. Wang. Introducing Map-Reduce to High End Computing. In Petascale Data Storage Workshop at SC08, Austin, Texas, Nov. 2008.
31. P. Nguyen and M. Halem. A mapreduce workflow system for architecting scientific data intensive applications. In Proceeding of the 2nd international workshop on Software engineering for cloud computing, SECLOUD '11, pages 57-63, New York, NY, USA, 2011. ACM.
32. L. Ramakrishnan, K. R. Jackson, S. Canon, S. Cholia, and J. Shalf. Defining future platform requirements for e-science clouds. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC '10, pages 101-106, New York, NY, USA, 2010. ACM.
33. L. Ramakrishnan, P. T. Zbiegel, S. Campbell, R. Bradshaw, R. S. Canon, S. Coghlan, I. Sakrejda, N. Desai, T. Declerck, and A. Liu. Magellan: experiences from a science cloud. In Proceedings of the 2nd international workshop on Scientific cloud computing, ScienceCloud '11, pages 49-58, New York, NY, USA, 2011. ACM.
34. F. Schmuck and R. Haskin. Gpfs: A shared-disk file system for large computing clusters. In In Proceedings of the 2002 Conference on File and Storage Technologies (FAST, pages 231-244, 2002.
35. S. W. S. Shimin Chen. Map-reduce meets wider varieties of applications. May 2008.
36. K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The hadoop distributed file system. In Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on, pages 1 -10, May 2010.
37. S. R. Soltis, G. M. Erickson, K. W. Preslan, M. T. O'keefe, and T. M. Ruwart. The global file system: A file system for shared disk storage, 1997.
38. J. Talbot, R. M. Yoo, and C. Kozyrakis. Phoenix++: modular mapreduce for shared-memory systems. In Proceedings of the second international workshop on MapReduce and its applications, MapReduce '11, pages 9-16, New York, NY, USA, 2011. ACM.
39. C. Zhang, H. De Sterck, A. Aboulnaga, H. Djambazian, and R. Sladek. Case study of scientific data processing on a cloud using hadoop. High Performance Computing Systems and Applications, 5976(2):400-415, 2010.