Major technical advancements in Apache Hive

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn , Issue , 2014, Pages 1235-1246

  Huai, Yin ^a   Chauhan, Ashutosh ^b   Gates, Alan ^b   Hagleitner, Gunther ^b   Hanson, Eric N ^c   O'Malley, Owen ^b   Pandey, Jitendra ^b   Yuan, Yuan ^a   Lee, Rubao ^a   Zhang, Xiaodong ^a  

^a OHIO STATE UNIVERSITY   (United States)

^b Hortonworks Inc   (United States)

^c MICROSOFT   (United States)

Author keywords

Data warehouse; Databases; Hadoop; Hive; MapReduce

Indexed keywords

COMPUTER SOFTWARE; DATA WAREHOUSES; DATABASE SYSTEMS; DIGITAL STORAGE;

DATA WAREHOUSE SYSTEMS; HADOOP; HIVE; MAP-REDUCE; RESOURCE UTILIZATIONS; RUN-TIME PERFORMANCE; STORAGE EFFICIENCY; TECHNICAL ADVANCEMENT;

BIG DATA;

EID: 84904296452     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2588555.2595630     Document Type: Conference Paper

References (51)

1. https://hadoop.apache.org/.
2. https://cwiki.apache.org/confluence/display/Hive/ Setting+up+HiveServer2.
3. https://tez.incubator.apache.org/.
4. https://hbase.apache.org/.
5. https://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/ hive/ql/io/orc/WriterImpl.java?view=log.
6. https://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/ hive/ql/optimizer/correlation/CorrelationOptimizer.java?view=log.
7. https://cwiki.apache.org/confluence/display/Hive/Correlation+Optimizer.
8. https://issues.apache.org/jira/browse/HIVE-4160.
9. https://issues.apache.org/jira/secure/attachment/12603710/ Hive-Vectorized-Query-Execution-Design-rev11.pdf.
10. http://www.tpc.org/tpch/.
11. http://www.tpc.org/tpcds/.
12. http://avro.apache.org/docs/current/trevni/spec.html.
13. https://github.com/Parquet/parquet-format.
14. https://incubator.apache.org/drill/.
15. https://github.com/cloudera/impala.
16. http://www.slideshare.net/ApacheDrill/oscon-2013-apache-drill-workshop- part-2.
17. https://hive.apache.org/.
18. A. Ailamaki, D. J. DeWitt, M. D. Hill, and M. Skounakis. Weaving Relations for Cache Performance. In VLDB, 2001.
19. P. A. Boncz, M. Zukowski, and N. Nes. MonetDB/X100: Hyper-Pipelining Query Execution. In CIDR, 2005.
20. Y. Cao, G. C. Das, C.-Y. Chan, and K.-L. Tan. Optimizing Complex Queries with Multiple Relation Instances. In SIGMOD, 2008.
21. G. P. Copeland and S. N. Khoshafian. A Decomposition Storage Model. In SIGMOD, 1985.
22. P. Cudre-Mauroux, H. Kimura, K.-T. Lim, J. Rogers, S. Madden, M. Stonebraker, S. B. Zdonik, and P. G. Brown. SS-DB: A Standard Science DBMS Benchmark. http://www-conf.slac.stanford.edu/xldb10/docs/ssdb-benchmark.pdf.
23. A. Floratou, J. M. Patel, E. J. Shekita, and S. Tata. Column-Oriented Storage Techniques for MapReduce. In VLDB, 2011.
24. S. Guo, J. Xiong, W. Wang, and R. Lee. Mastiff: A Mapreduce-based System for Time-Based Big Data Analytics. In CLUSTER, 2012.
25. S. Harizopoulos, V. Shkapenyuk, and A. Ailamaki. QPipe: A Simultaneously Pipelined Relational Query Engine. In SIGMOD, 2005.
26. W. Hasan and R. Motwani. Coloring Away Communication in Parallel Query Optimization. In VLDB, 1995.
27. Y. He, R. Lee, Y. Huai, Z. Shao, N. Jain, X. Zhang, and Z. Xu. RCFile: A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems. In ICDE, 2011.
28. Y. Huai, S. Ma, R. Lee, O. O'Malley, and X. Zhang. Understanding Insights into the Basic Structure and Essential Issues of Table Placement Methods in Clusters. In VLDB, 2013.
29. A. Jindal, J.-A. Quiané-Ruiz, and J. Dittrich. Trojan Data Layouts: Right Shoes for a Running Elephant. In SOCC, 2011.
30. A. Lamb, M. Fuller, R. Varadarajan, N. Tran, B. Vandier, L. Doshi, and C. Bear. The Vertica Analytic Database: C-Store 7 Years Later. In VLDB, 2012.
31. R. Lee, T. Luo, Y. Huai, F. Wang, Y. He, and X. Zhang. YSmart: Yet Another SQL-to-MapReduce Translator. In ICDCS, 2011.
32. H. Lim, H. Herodotou, and S. Babu. Stubby: A Transformation-based Optimizer for Mapreduce Workflows. In VLDB, 2012.
33. Y. Lin, D. Agrawal, C. Chen, B. C. Ooi, and S.Wu. Llama: Leveraging Columnar Storage for Scalable Join Processing in the Mapreduce Framework. In SIGMOD, 2011.
34. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: Interactive Analysis of Web-Scale Datasets. In VLDB, 2010.
35. T. Neumann. Efficiently Compiling Efficient Query Plans for Modern Hardware. In VLDB, 2011.
36. T. Neumann and G. Moerkotte. A Combined Framework for Grouping and Order Optimization. In VLDB, 2004.
37. T. Nykiel, M. Potamias, C. Mishra, G. Kollios, and N. Koudas. MRShare: Sharing Across Multiple Queries in Mapreduce. In VLDB, 2010.
38. S. Padmanabhan, T. Malkemus, R. C. Agarwal, and A. Jhingran. Block Oriented Processing of Relational Database Operations in Modern Computer Architectures. In ICDE, 2001.
39. J. Parikh. Data Infrastructure at Web Scale. http://www.vldb.org/2013/ video/keynote1.flv.
40. P. G. Selinger, M. M. Astrahan, D. D. Chamberlin, R. A. Lorie, and T. G. Price. Access Path Selection in a Relational Database Management System. In SIGMOD, 1979.
41. T. K. Sellis. Multiple-Query Optimization. ACM Trans. Database Syst., 13(1):23-52, Mar. 1988.
42. D. Simmen, E. Shekita, and T. Malkemus. Fundamental Techniques for Order Optimization. In SIGMOD, 1996.
43. J. Sompolski, M. Zukowski, and P. A. Boncz. Vectorization vs. Compilation in Query Execution. In DaMoN, 2011.
44. M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. O'Neil, P. O'Neil, A. Rasin, N. Tran, and S. Zdonik. C-Store: A Column-oriented DBMS. In VLDB, 2005.
45. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive-A Petabyte Scale Data Warehouse Using Hadoop. In ICDE, 2010.
46. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache Hadoop YARN: Yet Another Resource Negotiator. In SoCC, 2013.
47. X. Wang, C. Olston, A. D. Sarma, and R. Burns. CoScan: Cooperative Scan Sharing in the Cloud. In SoCC, 2011.
48. J. Zhou, P.-Å. Larson, and R. Chaiken. Incorporating Partitioning and Parallel Plans into the SCOPE Optimizer. In ICDE, 2010.
49. J. Zhou and K. A. Ross. A Multi-resolution Block Storage Model for Database Design. In IDEAS, 2003.
50. J. Zhou and K. A. Ross. Buffering Database Operations for Enhanced Instruction Cache Performance. In SIGMOD, 2004.
51. M. Zukowski, S. Héman, N. Nes, and P. Boncz. Cooperative Scans: Dynamic Bandwidth Sharing in a DBMS. In VLDB, 2007.