Evaluating Hadoop for data-intensive scientific operations

Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012

Volumn , Issue , 2012, Pages 67-74

  Fadika, Zacharia ^a   Govindaraju, Madhusudhan ^a   Canon, Richard ^b   Ramakrishnan, Lavanya ^b  

^a BINGHAMTON UNIVERSITY   (United States)

^b LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA OPERATIONS; DATA-INTENSIVE APPLICATION; DATA-INTENSIVE COMPUTING; DESIGN CONSIDERATIONS; FILE SYSTEMS; HIGH PERFORMANCE COMPUTING (HPC); LARGE DATA VOLUMES; MAP-REDUCE; OPEN SOURCE IMPLEMENTATION; PARALLELIZATIONS; PERFORMANCE CHARACTERISTICS; PERFORMANCE TRADE-OFF; SCIENTIFIC APPLICATIONS; SCIENTIFIC DATA; SOFTWARE STACKS;

CLOUD COMPUTING; DATA PROCESSING; SENSOR NETWORKS;

FAULT TOLERANCE;

EID: 84866750535     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLOUD.2012.118     Document Type: Conference Paper

References (28)

1. Mcllanox Hadoop http://www.mcllanox.com/content/pages.php?pg=hadoop.
2. D. E. G. Abhishek Verma, Xavier Llor'a. Scaling simple and compact genetic algorithms using mapreduce. October 2009.
3. S. Ahern, A. Shoshani, K.-L. Ma, A. Choudhary, T. Critchlow, S. Kalasky, K.-L. Ma, V. Pascucci, J. Ahrens, E. W. Bethel, H. Childs, J. Huang, K. Joy, Q. Koziol, G. Lofstead, J. Meredith, K. Moreland, G. Ostrouchov, M. Papka, V. Vishwanath, M. Wold, N. Wright, and K. Wu. Scientific discovery at the exascale: Report from the doe ascr 2011 workshop on exascale data management, anatysis, and visualization, http://www.olcf.ornl.gov/wp-content/uploads/2011/01/Exascale- ASCR-Analysis.pdf, 2011.
4. V. Akaselik, J. Bielak, G. Biros, I. Epanomeritakis, A. Fernandez, O. Ghattas, E. J. Kim, J. Lopez, T. Tu, and J. Urbanic. High resolution forward and inverse earthquake modeling on terascale computers. In In SC2003, 2003.
5. R. Ananthanarayanan, K. Gupta, P. Pandey, H. Pucha, P. Sarkar, M. Shah, and R. Tewari. Cloud analytics: do we really need to reinvent the storage stack? In Proceedings of the 2009 conference on Hot topics in cloud computing, HotCloud, 2009.
6. J. B. Buck, N. Watkins, J. LeFevre, K. Ioannidou, C. Maltzahn, N. Polyzotis, and S. A. Brandt. Scihadoop: array-based query processing in hadoop. In SC, page 66, 2011.
7. A. Cary, Z. Sun, V. Hristidis, and N. Rishe. Experiences on processing spatial data with mapreduce. In SSDBM 2009: Proceedings of the 21st International Conference on Scientific and Statistical Database Management, pages 302-319. Berlin, Heidelberg, 2009. Springer-Verlag.
8. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51(1):107-113, 2008.
9. J. Ekanayake, S. Pallickara. and G. Fox. MapReduce for Data Intensive Scientific Analyses. In IEEE Fourth International Conference on eScience, pages 277-284, December 2008.
10. Z. Fadika, E. Dede, M. Govindaraju, and L. Ramakrishnan. Benchmarking mapreduce implementations for application usage scenarios. Grid 2011: 12th IEEE/ACM International Gonference on Grid Computing, 0:1-8, 2011.
11. Y. Gu and R. L. Grossman. Lessons learned from a year's worth of benchmarks of large data clouds. In Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS '09, pages 3:1-3:6, New York, NY, USA, 2009. ACM.
12. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A self-tuning system for big data analytics. In CIDR, pages 261-272, 2011.
13. S. Huang, J. Huang, J. Dai, T. Xie, and B. H. 0002. The hibeneh benchmark suite: Characterization of the mapreduce-based data analysis. In ICDE Workshops, pages 41-51. IEEE, 2010.
14. D. Jiang, B. C. Ooi. L. Shi, and S. Wu. The perfonnance of mapreduce: an in-depth study. Proc. VLDB Endow, 3:472-483, September 2010.
15. A. Kalyanaraman, W. R. Cannon, B. Latt, and D. J. Baxter. Mapreduce implementation of a hybrid spectral library-database search method for large-scale peptide identification. Bioinformatics, 2011.
16. K. Kim, K. Jeon, H. Han, S.-g. Kim, H. Jung, and H. Y. Yeom. Mrbench: A benchmark for mapreduce framework. In Proceedings of the 2008 14th IEEE Internationnl Conference on Parallel and Distributed Systems, pages 11-18, 2008.
17. M. Kontagora and H. Gonzalez-Velez, Benchmarking a mapreduce enviromnent on a full virtualisation platform. In Proceedings of the 2010 International Gonference on Complex, Intelligent nnd Software Intensive Systems, CISIS '10, pages 433-438, 2010.
18. G. Mackey, S. Sehrish, J. Lopez, J. Bent, S. Habib, and J. Wang. Introducing Map-Reduce to High End Computing. In Petascale Data Storage Workshop at SC08, Austin, Texas, Nov. 2008.
19. P. Nguyen and M. Halem. A mapreduce workflow system for architecting scientific data intensive applications. In Proceeding of the 2nd international workshop on Software engineering for cloud computing, SECLOUD '11, pages 57-63, New York, NY, USA, 2011. ACM.
20. L. Ramakrishnan, K. R. Jackson, S. Canon, S. Cholia, and J. Shalf. Defining future platform requirements for e-science clouds. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC '10, pages 101-106, New York, NY, USA, 2010. ACM.
21. L. Raniakrislman, P. T. Zbiegel, S. Campbell, R. Bradshaw, R. S. Canon, S. Coghlan, I. Sakrejda, N. Desai, T. Declerck, and A. Liu. Magellan: Experiences from a science cloud. ScienceCloud, pages 49-58, 2011.
22. M. C. Schatz. CloudBurst: highly sensitive read mapping with MapReduce. Bioinformatics, pages 1363-1369, June 2009.
23. J. Shafer, S. Rixner, and A. Cox. The hadoop distributed filesystem: Balancing portabihty mid performance. In Pemformance Analysis of Systems Software (ISPASS), 2010 IEEE International Symposium on, pages 122-133, march 2010.
24. K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The hadnop distributed file system. In Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on, pages 1-10, May 2010.
25. M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden, F. Paulson, A. Pavlo, and A. Rasin. Mapreduce and parallel dbmss: friends or foes? Commun. ACM, 53:64-71, Jan. 2010.
26. S. Sur, H. Wang, J. Huang, X. Ouyang, and D. K. Panda. Can high-performance interconnects benefit hadoop distributed file system? Workshop on Micro Architectural Support for Virtualization, Data Center Computing, and Clouds, in Conjunction with MICRO 2010, Dec 2010.
27. Y. S. Tan, J. Tan, E. S. Chng, B.-S. Lee, J. Li, S. Date, H. P. Chak, X. Xiao, and A. Narishige. Hadoop framework: impact of data organization on performance. Software: Practice and Experience, 2011.
28. C. Zhang, H. De Sterck, A. Aboulnaga. H. Djambazian, and R. Sladek. Case study of scientific data processing on a cloud using hadoop. High Performance Computing Systems and Applications, 5976(2):400-415, 2010.