The design and implementation of modern column-oriented database systems

Foundations and Trends in Databases

Volumn 5, Issue 3, 2012, Pages 197-280

  Abadi, Daniel ^a   Boncz, Peter ^b   Harizopoulos, Stavros ^c   Idreos, Stratos ^d   Madden, Samuel ^e  

^a YALE UNIVERSITY   (United States)

^b AWS CWI   (United States)

^c Amiato Inc   (United States)

^d HARVARD UNIVERSITY   (United States)

^e MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE INDEXING; ARCHITECTURAL FEATURES; COLUMN-ORIENTED DATABASE; DESIGN AND IMPLEMENTATIONS; JOIN PROCESSING; RECENT RESEARCHES; RESEARCH PROTOTYPE; VECTORIZATION;

INDEXING (MATERIALS WORKING); QUERY PROCESSING;

DATABASE SYSTEMS;

EID: 84889853931     PISSN: 19317883     EISSN: 19317891     Source Type: Journal    
DOI: 10.1561/1900000024     Document Type: Article

References (101)

1. Daniel J. Abadi, Samuel Madden, and Nabil Hachem. Column-stores vs. row-stores: how different are they really? In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 967-980, 2008.
2. Daniel J. Abadi, Samuel R. Madden, and Miguel Ferreira. Integrating compression and execution in column-oriented database systems. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 671-682, 2006. (Pubitemid 46946548)
3. Daniel J. Abadi, Daniel S. Myers, David J. DeWitt, and Samuel R. Madden. Materialization strategies in a column-oriented DBMS. In Proceedings of the International Conference on Data Endineering (ICDE), pages 466-475, 2007. (Pubitemid 47422050)
4. R. Abdel Kader, P. Boncz, S. Manegold, and M. van Keulen. ROX: run-time optimization of xqueries. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 615-626. ACM, 2009.
5. Anastassia Ailamaki, David J. DeWitt, Mark D. Hill, and Marios Skounakis. Weaving relations for cache performance. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 169-180, 2001.
6. Martina-Cezara Albutiu, Alfons Kemper, and Thomas Neumann. Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems. Proceedings of the Very Large Data Bases Endowment (PVLDB), 5(10):1064-1075, 2012.
7. Sihem Amer-Yahia and Theodore Johnson. Optimizing queries on compressed bitmaps. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 329-338, 2000.
8. G. Antoshenkov. Byte-aligned data compression. U.S. Patent Number 5, 363, 098, 1994.
9. Cagri Balkesen, Gustavo Alonso, Jens Teubner, and M. Tamer Özsu. Multi-Core, Main-Memory Joins: Sort vs. Hash Revisited. Proceedings of the Very Large Data Bases Endowment (PVLDB), 7(1):85-96, 2013.
10. Cagri Balkesen, Jens Teubner, Gustavo Alonso, and M. Tamer Özsu. Main-memory hash joins on multi-core CPUs: Tuning to the underlying hardware. In Proceedings of the International Conference on Data Endineering (ICDE), pages 362-373, 2013.
11. Ronald Barber, Peter Bendel, Marco Czech, Oliver Draese, Frederick Ho, Namik Hrle, Stratos Idreos, Min-Soo Kim, Oliver Koeth, Jae-Gil Lee, Tianchao Tim Li, Guy M. Lohman, Konstantinos Morfonios, René Müller, Keshava Murthy, Ippokratis Pandis, Lin Qiao, Vijayshankar Raman, Richard Sidle, Knut Stolze, and Sandor Szabo. Business Analytics in (a) Blink. IEEE Data Eng. Bull., 35(1):9-14, 2012.
12. Don S. Batory. On searching transposed files. ACM Transactions on Database Systems, 4(4):531-544, 1979.
13. Philip A. Bernstein and Dah-Ming W. Chiu. Using semi-joins to solve relational queries. J. ACM, 28(1):25-40, 1981.
14. R. Bestgen and T. McKinley. Taming the business intelligence monster. IBM Systems Magazine, 2007.
15. Carsten Binnig, Stefan Hildenbrand, and Franz Färber. Dictionarybased order-preserving string compression for main memory column stores. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 283-296, 2009.
16. Peter Boncz. Monet: A next-generation DBMS kernel for queryintensive applications. University of Amsterdam, PhD Thesis, 2002.
17. Peter Boncz, Stefan Manegold, and Martin Kersten. Database architecture optimized for the new bottleneck: Memory access. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 54-65, 1999.
18. Peter Boncz, Marcin Zukowski, and Niels Nes. Monet DB/X100: Hyperpipelining query execution. In Proceedings of the biennial Conference on Innovative Data Systems Research (CIDR), 2005.
19. Peter A. Boncz and Martin L. Kersten. MIL primitives for querying a fragmented world. VLDB Journal, 8(2):101-119, 1999.
20. Nicolas Bruno. Teaching an old elephant new tricks. In Proceedings of the biennial Conference on Innovative Data Systems Research (CIDR), 2009.
21. Surajit Chaudhuri and Vivek R. Narasayya. An Efficient Cost-Driven Index Selection Tool for Microsoft SQL Server. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 146-155, 1997.
22. George P. Copeland and Setrag N. Khoshafian. A decomposition storage model. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 268-279, 1985.
23. D. J. Dewitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen. The gamma database machine project. IEEE Transactions on Knowledge and Data Engineering, 2(1):44-62, 1990.
24. David DeWitt. From 1 to 1000 mips, November 2009. PASS Summit 2009 Keynote.
25. Amr El-Helw, Kenneth A. Ross, Bishwaranjan Bhattacharjee, Christian A. Lang, and George A. Mihaila. Column-oriented query processing for row stores. In Proceedings of the International Workshop On Data Warehousing and OLAP, pages 67-74, 2011.
26. Franz Färber, Norman May, Wolfgang Lehner, Philipp Große, Ingo Müller, Hannes Rauhe, and Jonathan Dees. The SAP HANA Database - An Architecture Overview. IEEE Data Eng. Bull., 35(1):28-33, 2012.
27. Avrilia Floratou, Jignesh M. Patel, Eugene J. Shekita, and Sandeep Tata. Column-Oriented Storage Techniques for MapReduce. Proceedings of the Very Large Data Bases Endowment (PVLDB), 4(7):419-429, 2011.
28. Clark D. French. "One Size Fits All" Database Architectures Do Not Work for DDS. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 449-450, 1995.
29. Clark D. French. Teaching an OLTP Database Kernel Advanced Data Warehousing Techniques. In Proceedings of the International Conference on Data Endineering (ICDE), pages 194-198, 1997.
30. G.Graefe and L.Shapiro. Data compression and database performance. In ACM/IEEE-CS Symp. On Applied Computing, pages 22 -27, April 1991.
31. Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. Compressing relations and indexes. In Proceedings of the International Conference on Data Endineering (ICDE), pages 370-379, 1998.
32. Goetz Graefe. Query evaluation techniques for large databases. ACM Computing Surveys, 25(2):73-170, 1993. (Pubitemid 23712749)
33. Goetz Graefe. Efficient columnar storage in b-trees. SIGMOD Rec., 36(1):3-6, 2007.
34. Goetz Graefe. Modern B-Tree Techniques. Foundations and Trends in Databases, 3(4):203-402, 2011.
35. Goetz Graefe, Felix Halim, Stratos Idreos, Harumi Kuno, and Stefan Manegold. Concurrency Control for Adaptive Indexing. Proceedings of the Very Large Data Bases Endowment (PVLDB), 5(7):656-667, 2012.
36. Goetz Graefe, Stratos Idreos, Harumi Kuno, and Stefan Manegold. Benchmarking Adaptive Indexing. In Proceedings of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC), pages 169-184, 2010.
37. Felix Halim, Stratos Idreos, Panagiotis Karras, and Roland H. C. Yap. Stochastic Database Cracking: Towards Robust Adaptive Indexing in Main-Memory Column-Stores. Proceedings of the Very Large Data Bases Endowment (PVLDB), 5(6):502-513, 2012.
38. Alan Halverson, Jennifer L. Beckmann, Jeffrey F. Naughton, and David J. Dewitt. A Comparison of C-Store and Row-Store in a Common Framework. Technical Report TR1570, University of Wisconsin- Madison, 2006.
39. Richard A. Hankins and Jignesh M. Patel. Data morphing: An adaptive, cache-conscious storage technique. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 417-428, 2003.
40. Stavros Harizopoulos, Velen Liang, Daniel J. Abadi, and Samuel R. Madden. Performance tradeoffs in read-optimized databases. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 487-498, 2006.
41. S. Héman, M. Zukowski, N.J. Nes, L. Sidirourgos, and P. Boncz. Positional update handling in column stores. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 543-554, 2010.
42. William Hodak. Exadata hybrid columnar compression. Oracle Whitepaper, 2009. http://www.oracle.com/technetwork/database/exadata/index.html.
43. Allison L. Holloway, Vijayshankar Raman, Garret Swart, and David J. DeWitt. How to barter bits for chronons: compression and bandwidth trade offs for database scans. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 389-400, 2007. (Pubitemid 47630823)
44. Stratos Idreos. Database Cracking: Towards Auto-tuning Database Kernels. CWI, PhD Thesis, 2010.
45. Stratos Idreos, Ioannis Alagiannis, Ryan Johnson, and Anastasia Ailamaki. Here are my Data Files. Here are my Queries. Where are my Results? In Proceedings of the biennial Conference on Innovative Data Systems Research (CIDR), pages 57-68, 2011.
46. Stratos Idreos, Fabian Groffen, Niels Nes, Stefan Manegold, Sjoerd Mullender, and Martin L Kersten. MonetDB: Two Decades of Research in Column-oriented Database Architectures. IEEE Data Eng. Bull., 35(1):40-45, 2012.
47. Stratos Idreos, Raghav Kaushik, Vivek R. Narasayya, and Ravishankar Ramamurthy. Estimating the compression fraction of an index using sampling. In Proceedings of the International Conference on Data Endineering (ICDE), pages 441-444, 2010.
48. Stratos Idreos, Martin L. Kersten, and Stefan Manegold. Database cracking. In Proceedings of the biennial Conference on Innovative Data Systems Research (CIDR), pages 68-78, 2007.
49. Stratos Idreos, Martin L. Kersten, and Stefan Manegold. Updating a cracked database. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 413-424, 2007. (Pubitemid 47630825)
50. Stratos Idreos, Martin L. Kersten, and Stefan Manegold. Self-organizing tuple reconstruction in column stores. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 297-308, 2009.
51. Stratos Idreos, Stefan Manegold, Harumi Kuno, and Goetz Graefe. Merging What's Cracked, Cracking What's Merged: Adaptive Indexing in Main-Memory Column-Stores. Proceedings of the Very Large Data Bases Endowment (PVLDB), 4(9):585-597, 2011.
52. Ryan Johnson, Vijayshankar Raman, Richard Sidle, and Garret Swart. Row-wise parallel predicate evaluation. Proceedings of the Very Large Data Bases Endowment (PVLDB), 1(1):622-634, 2008.
53. Theodore Johnson. Performance measurements of compressed bitmap indices. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 278-289, 1999.
54. Ilkka Karasalo and Per Svensson. The design of cantor: A new system for data analysis. In Proceedings of the 3rd international workshop on Statistical and scientific database management, pages 224-244, 1986.
55. Illka Karasalo and Per Svensson. An overview of cantor: A new system for data analysis. In Proceedings of the 2nd international Workshop on Statistical Database Management (SSDBM), pages 315-324, 1983. (Pubitemid 17628649)
56. Alfons Kemper and Thomas Neumann. HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots. In Proceedings of the International Conference on Data Endineering (ICDE), pages 195-206, 2011.
57. Alfons Kemper, Thomas Neumann, Florian Funke, Viktor Leis, and Henrik Mühe. HyPer: Adapting Columnar Main-Memory Data Management for Transactional AND Query Processing. IEEE Data Eng. Bull., 35(1):46-51, 2012.
58. Setrag Khoshafian, George Copeland, Thomas Jagodis, Haran Boral, and Patrick Valduriez. A query processing strategy for the decomposed storage model. In Proceedings of the International Conference on Data Endineering (ICDE), pages 636-643, 1987. (Pubitemid 17580391)
59. Setrag Khoshafian and Patrick Valduriez. Parallel execution strategies for declustered databases. In Proceedings of the International Workshop on Database Machines, pages 458-471, 1987.
60. Andrew Lamb, Matt Fuller, Ramakrishna Varadarajan, Nga Tran, Ben Vandier, Lyric Doshi, and Chuck Bear. The vertica analytic database: C-store 7 years later. Proceedings of the Very Large Data Bases Endowment (PVLDB), 5(12):1790-1801, 2012.
61. P.Å. Larson, C. Clinciu, E.N. Hanson, A. Oks, S.L. Price, S. Rangarajan, A. Surna, and Q. Zhou. Sql server column store indexes. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 1177- 1184, 2011.
62. Per-Åke Larson, Cipri Clinciu, Campbell Fraser, Eric N. Hanson, Mostafa Mokhtar, Michal Nowakiewicz, Vassilis Papadimos, Susan L. Price, Srikumar Rangarajan, Remus Rusanu, and Mayukh Saubhasik. Enhancements to sql server column stores. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 1159-1168, 2013.
63. Per-Åke Larson, Eric N. Hanson, and Susan L. Price. Columnar Storage in SQL Server 2012. IEEE Data Eng. Bull., 35(1):15-20, 2012.
64. Zhe Li and Kenneth A. Ross. Fast joins using join indices. VLDB Journal, 8:1-24, April 1999.
65. R.A. Lorie and A.J. Symonds. A relational access method for interactive applications. In Courant Computer Science Symposia, Vol. 6: Data Base Systems. Prentice Hall, 1971.
66. Roger MacNicol and Blaine French. Sybase IQ multiplex - designed for analytics. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 1227-1230, 2004.
67. S. Manegold, P. Boncz, N. Nes, and M. Kersten. Cache-conscious radixdecluster projections. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 684-695, 2004.
68. A. Moffat and J. Zobel. Compression and fast indexing for multigigabyte text databases. Australian Computer Journal, 26(1):1-9, 1994.
69. C. Mohan, Donald J. Haderle, Yun Wang, and Josephine M. Cheng. Single Table Access Using Multiple Indexes: Optimization, Execution, and Concurrency Control Techniques. pages 29-43, 1990.
70. Thomas Neumann. Efficiently Compiling Efficient Query Plans forModern Hardware. Proceedings of the Very Large Data Bases Endowment (PVLDB), 4(9):539-550, 2011.
71. Patrick O'Neil and Dallan Quass. Improved query performance with variant indexes. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 38-49, 1997. (Pubitemid 127456772)
72. Patrick E. O'Neil. Model 204 architecture and performance. In Proceeding of the International Workshop on High Performance Transaction Systems, pages 40-59, 1987.
73. Patrick E. O'Neil, Elizabeth J. O'Neil, and Xuedong Chen. The Star Schema Benchmark (SSB). http://www.cs.umb.edu/~poneil/StarSchemaB.PDF.
74. S. Padmanabhan, T. Malkemus, R. Agarwal, and A. Jhingran. Block oriented processing of relational database operations in modern computer architectures. In Proceedings of the International Conference on Data Endineering (ICDE), pages 567-574, 2001. (Pubitemid 32457160)
75. Sriram Padmanabhan, Bishwaranjan Bhattacharjee, Timothy Malkemus, Leslie Cranston, and Matthew Huras. Multi-Dimensional Clustering: A New Data Layout Scheme in DB2. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 637-641, 2003.
76. M. Poess and D. Potapov. Data compression in oracle. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 937-947, 2003.
77. Bogdan Raducanu, Peter A. Boncz, and Marcin Zukowski. Micro adaptivity in vectorwise. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 1231-1242, 2013.
78. Ravishankar Ramamurthy, David Dewitt, and Qi Su. A case for fractured mirrors. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 89 - 101, 2002. (Pubitemid 37017922)
79. V. Raman, G. Attaluri, R. Barber, N. Chainani, D. Kalmuk, V. KulandaiSamy, J. Leenstra, S. Lightstone, S. Liu, G. M. Lohman, T. Malkemus, R. Mueller, I. Pandis, B. Schiefer, D. Sharpe, R. Sidle, A. Storm, and L. Zhang. DB2 with BLU Acceleration: So much more than just a column store. Proceedings of the Very Large Data Bases Endowment (PVLDB), 6(11), 2013.
80. Vijayshankar Raman, Lin Qiao, Wei Han, Inderpal Narang, Ying-Lin Chen, Kou-Horng Yang, and Fen-Ling Ling. Lazy, adaptive rid-list intersection, and its application to index anding. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 773-784, 2007. (Pubitemid 47630852)
81. Vijayshankar Raman, Garret Swart, Lin Qiao, Frederick Reiss, Vijay Dialani, Donald Kossmann, Inderpal Narang, and Richard Sidle. Constant-Time Query Processing. In Proceedings of the International Conference on Data Endineering (ICDE), pages 60-69, 2008.
82. Mark A. Roth and Scott J. Van Horn. Database compression. SIGMOD Rec., 22(3):31-39, 1993.
83. Felix Martin Schuhknecht, Alekh Jindal, and Jens Dittrich. The Uncracked Pieces in Database Cracking. Proceedings of the Very Large Data Bases Endowment (PVLDB), 7(2), 2013.
84. Minglong Shao, Jiri Schindler, Steven W. Schlosser, Anastassia Ailamaki, and Gregory R. Ganger. Clotho: Decoupling memory page layout from storage organization. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 696-707, 2004.
85. Ambuj Shatdal, Chander Kant, and Jeffrey F. Naughton. Cache Conscious Algorithms for Relational Query Processing. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 510-521, 1994.
86. Lefteris Sidirourgos and Martin L. Kersten. Column imprints: A secondary index structure. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 893-904, 2013.
87. Michael Stonebraker. The case for partial indexes. SIGMOD Record, 18(4):4-11, 1989.
88. Michael Stonebraker, Daniel J. Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Samuel R. Madden, Elizabeth J. O'Neil, Patrick E. O'Neil, Alexander Rasin, Nga Tran, and Stan B. Zdonik. C-Store: A Column-Oriented DBMS. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 553-564, 2005. (Pubitemid 43991033)
89. Dimitris Tsirogiannis, Stavros Harizopoulos, Mehul A. Shah, Janet L. Wiener, and Goetz Graefe. Query processing techniques for solid state drives. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 59-72, 2009.
90. Stephen Weyl, James Fries, Gio Wiederhold, and Frank Germano. A modular self-describing clinical databank system. Computers and Biomedical Research, 8(3):279 - 293, 1975.
91. K. Wu, E. Otoo, and A. Shoshani. Compressed bitmap indices for efficient query processing. Technical Report LBNL-47807, 2001.
92. K. Wu, E. Otoo, and A. Shoshani. Compressing bitmap indexes for faster search operations. In Proceedings of the International Conference on Scientific and Statistical Database Management (SSDBM), pages 99- 108, 2002.
93. K.Wu, E. Otoo, A. Shoshani, and H. Nordberg. Notes on design and implementation of compressed bit vectors. Technical Report LBNL/PUB- 3161, 2001.
94. Jingren Zhou and Kenneth A. Ross. A Multi-Resolution Block Storage Model for Database Design. In Proceedings of the International Database Engineering and Applications Symposium (IDEAS), pages 22- 33, 2003.
95. Jingren Zhou and Kenneth A. Ross. Buffering Database Operations for Enhanced Instruction Cache Performance. In Proceedings of the ACM SIGMOD Conference on Management of Data, pages 191-202, 2004.
96. M. Zukowski. Balancing vectorized query execution with bandwidthoptimized storage. University of Amsterdam, PhD Thesis, 2009.
97. M. Zukowski, S. Héman, and P. Boncz. Architecture-conscious hashing. In Proceedings of the International Workshop on Data Management on New Hardware (DAMON), 2006.
98. M. Zukowski, S. Héman, N. Nes, and P. Boncz. Cooperative scans: dynamic bandwidth sharing in a DBMS. In Proceedings of the International Conference on Very Large Data Bases (VLDB), pages 723-734, 2007.
99. Marcin Zukowski and Peter A. Boncz. Vectorwise: Beyond column stores. IEEE Data Eng. Bull., 35(1):21-27, 2012.
100. Marcin Zukowski, Sandor Heman, Niels Nes, and Peter Boncz. Super- Scalar RAM-CPU Cache Compression. In Proceedings of the International Conference on Data Endineering (ICDE), 2006.
101. Marcin Zukowski, Niels Nes, and Peter Boncz. DSM vs. NSM: CPU performance tradeoffs in block-oriented query processing. In Proceedings of the International Workshop on Data Management on New Hardware (DAMON), pages 47-54, 2008.