SciHadoop: Array-based query processing in hadoop

Proceedings of 2011 SC - International Conference for High Performance Computing, Networking, Storage and Analysis

Volumn , Issue , 2011, Pages

  Buck, Joe B ^a   Watkins, Noah ^a   Lefevre, Jeff ^a   Ioannidou, Kleoni ^a   Maltzahn, Carlos ^a   Polyzotis, Neoklis ^a   Brandt, Scott ^a  

^a University of California Santa Cruz   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AGGREGATE QUERIES; BINARY FILES; BYTE STREAMS; COMMERCIAL APPLICATIONS; DATA DEPENDENCIES; DATA SETS; INPUT DATAS; LOGICAL DATA; PLUG-INS; RUNTIMES; SCIENTIFIC APPLICATIONS; SCIENTIFIC DATA;

COMPUTER SOFTWARE SELECTION AND EVALUATION; DATA TRANSFER; MODELS; OPTIMIZATION; QUERY PROCESSING;

DATA REDUCTION;

EID: 83155182857     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2063384.2063473     Document Type: Conference Paper

References (20)

1. Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, Avi Silberschatz, and Alexander Rasin. HadoopDB: An architectural hybrid of mapreduce and dbms technologies for analytical workloads. In Proceedings of Very Large Data Bases (VLDB'09), Lyon, France, August 24-28 2009.
2. Paul G. Brown. Overview of SciDB: large scale array storage, processing and analysis. In Proceedings of the 2010 International Conference on Management of Data, SIGMOD'10, pages 963-968, New York, NY, USA, 2010. ACM.
3. Joe Buck, Noah Watkins, Jeff Lefevre, Kleoni Ioannidou, Carlos Maltzahn, Neoklis Polyzotis, and Scott A. Brandt. SciHadoop: Array-based query processing in hadoop. Technical Report UCSC-SOE-11-04, UCSC, 2011.
4. Jeffrey Dean and Sanjay Ghemawat. Mapreduce: simplified data processing on large clusters. In OSDI'04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, pages 10-10, Berkeley, CA, USA, 2004. USENIX Association.
5. Thilina Gunarathne, Tak-Lon Wu, Judy Qiu, and Geoffrey Fox. Mapreduce in the clouds for science. Cloud Computing Technology and Science, IEEE International Conference on, 0:565-572, 2010.
6. HBase homepage. http://http://hbase.apache.org/.
7. Ming-Yee Iu and Willy Zwaenepoel. HadoopToSQL: a mapreduce query optimizer. In Proceedings of the 5th European Conference on Computer Systems (EuroSys'10), pages 251-264, 2010.
8. Leonid Libkin, Rona Machlin, and Limsoon Wong. A query language for multidimensional arrays: design, implementation, and optimization techniques. In Proceedings of the 1996 ACM SIGMOD international conference on Management of data, SIGMOD'96, pages 228-239, New York, NY, USA, 1996. ACM. (Pubitemid 126440771)
9. S. Loebman, D. Nunley, Yong-Chul Kwon, B. Howe, M. Balazinska, and J.P. Gardner. Analyzing massive astrophysical datasets: Can pig/hadoop or a relational dbms help? In Cluster Computing and Workshops, 2009. CLUSTER'09. IEEE International Conference on, pages 1-10, 31 2009-sept. 4 2009.
10. Rona Machlin. Index-based multidimensional array queries: safety and equivalence. In Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, PODS'07, pages 175-184, New York, NY, USA, 2007. ACM. (Pubitemid 47620892)
11. Samuel Madden, Michael J. Franklin, Joseph M. Hellerstein, and Wei Hong. Tag: a tiny aggregation service for ad-hoc sensor networks. In OSDI 02, OSDI'02, pages 131-146, New York, NY, USA, 2002. ACM.
12. NetCDF operator (NCO) homepage. http://nco.sourceforge.net/.
13. Sangwon Seo, Edward J. Yoon, Jaehong Kim, Seongwook Jin, and Seungryoul Maeng. Hama: An efficient matrix computation with the mapreduce framework. In Proceedings of the 2nd IEEE International Conference on Cloud Computing Technology and Science, IEEE CloudCom 2010. IEEE, December 2010.
14. Konstantin V. Shvachko. HDFS scalability: the limits to growth.;login, 35(2):6-16, April 2010.
15. Emad Soroush, Magdalena Balazinska, and Daniel Wang. Arraystore: a storage manager for complex parallel array processing. In Proceedings of the 2011 international conference on Management of data, SIGMOD'11, pages 253-264, New York, NY, USA, 2011. ACM.
16. Michael Stonebraker, Jacek Becla, David Dewitt, Kian-Tat Lim, David Maier, Oliver Ratzesberger, and Stan Zdonik. Requirements for science data bases and SciDB. In CIDR 09, 2009.
17. Alex van Ballegooij, Roberto Cornacchia, Arjen de Vries, and Martin Kersten. Distribution rules for array database queries. In Kim Andersen, John Debenham, and Roland Wagner, editors, Database and Expert Systems Applications, volume 3588 of Lecture Notes in Computer Science, pages 55-64. Springer Berlin/Heidelberg, 2005. (Pubitemid 41450441)
18. Daniel L. Wang, Charles S. Zender, and Stephen F. Jenks. Clustered workow execution of retargeted data analysis scripts. In CCGRID 2008, 2008.
19. Jianwu Wang, Daniel Crawl, and Ilkay Altintas. Kepler + Hadoop: A general architecture facilitating data-intensive applications in scientific workow systems. In SC-WORKS, 2009.
20. Hui Zhao, SiYun Ai, ZhenHua Lv, and Bo Li. Parallel accessing massive NetCDF data based on MapReduce. In Web Information Systems and Mining, Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2010.