SciHive: Array-based query processing with HiveQL

Proceedings - 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2013

Volumn , Issue , 2013, Pages 887-894

  Geng, Yifeng ^a   Huang, Xiaomeng ^a   Zhu, Meiqi ^a   Ruan, Huabin ^a   Yang, Guangwen ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

array data; big data; Hive; MapReduce; query optimization

Indexed keywords

ARRAY DATA; BIG DATUM; HIVE; MAP-REDUCE; QUERY OPTIMIZATION;

OPTIMIZATION; QUERY LANGUAGES;

LOADING;

EID: 84893483814     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/TrustCom.2013.108     Document Type: Conference Paper

References (21)

1. http://public.web.cern.ch/public/en/lhc/Computing-en.html
2. http://cmip-pcmdi.llnl.gov/cmip5/index.html
3. Hive homepage. http://hive.apache.org/
4. Hadoop homepage. http://Hadoop.apache.org/
5. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters", in Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation-Volume 6, 2004, p. 10-10.
6. http://www.unidata.ucar.edu/software/netcdf/
7. J. Gray, D. T. Liu, M. Nieto-Santisteban, A. Szalay, D. J. DeWitt, and G. Heber, "Scientific data management in the coming decade", SIGMOD Rec., vol. 34, no. 4, pp. 34-41, 2005.
8. http://www.unidata.ucar.edu/software/netcdf-java/
9. http://meta.iap.ac.cn/index.php?option=com-mtree&task= viewlink&link-id=13&Itemid=1000&Itemid=0
10. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+U DF
11. S. Loebman, D. Nunley, Y.-C. Kwon, B. Howe, M. Balazinska, and J. P. Gardner, "Analyzing massive astrophysical datasets: Can Pig/Hadoop or a relational DBMS help?", in Cluster Computing and Workshops, 2009. CLUSTER '09. IEEE International Conference on, 2009, pp. 1-10.
12. J. Ekanayake, S. Pallickara, and G. Fox, "MapReduce for Data Intensive Scientific Analyses", in eScience, 2008. eScience '08. IEEE Fourth International Conference on, 2008, pp. 277-284.
13. J. Wang, D. Crawl, and I. Altintas, "Kepler + Hadoop: a general architecture facilitating data-intensive applications in scientific workflow systems", in Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, New York, NY, USA, 2009, pp. 12:1-12:8.
14. S. Seo, E. J. Yoon, J. Kim, S. Jin, J.-S. Kim, and S. Maeng, "HAMA: An Efficient Matrix Computation with the MapReduce Framework", in Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on, 2010, pp. 721-726.
15. J. B. Buck, N. Watkins, J. LeFevre, K. Ioannidou, C. Maltzahn, N. Polyzotis, and S. Brandt, "SciHadoop: array-based query processing in Hadoop", in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, New York, NY, USA, 2011, pp. 66:1-66:11.
16. H. Zhao, S. Ai, Z. Lv, and B. Li, "Parallel accessing massive NetCDF data based on mapreduce", in Proceedings of the 2010 international conference on Web information systems and mining, Berlin, Heidelberg, 2010, pp. 425-431.
17. Yi Wang, Wei Jiang, Gagan Agrawal, "SciMATE: A Novel MapReduce-Like Framework for Multiple Scientific Data Formats," ccgrid, pp.443-450, 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012), 2012
18. D. L. Wang, C. S. Zender, and S. F. Jenks, "Clustered Workflow Execution of Retargeted Data Analysis Scripts", in Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid, Washington, DC, USA, 2008, pp. 449-458.
19. E. Soroush, M. Balazinska, and D. Wang, "ArrayStore: a storage manager for complex parallel array processing", in Proceedings of the 2011 international conference on Management of data, New York, NY, USA, 2011, pp. 253-264.
20. A. van Ballegooij, R. Cornacchia, A. de Vries, and M. Kersten, "Distribution Rules for Array Database Queries", in Database and Expert Systems Applications, vol. 3588, K. Andersen, J. Debenham, and R. Wagner, Eds. Springer Berlin / Heidelberg, 2005, pp. 55-64.
21. J. Chou, K. Wu, and P. Prabhat, "FastQuery: A Parallel Indexing System for Scientific Data", in 2011 IEEE International Conference on Cluster Computing (CLUSTER), 2011, pp 455-464.