Memory-efficient Group-by-aggregate using compressed buffer trees

Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

Volumn , Issue , 2013, Pages

  Amur, Hrishikesh ^b   Richter, Wolfgang ^a   Andersen, David G ^a   Kaminsky, Michael ^c   Schwan, Karsten ^b   Balachandran, Athula ^a   Zawadzki, Erik ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^c INTEL CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AGGREGATES; BATCH DATA PROCESSING; CLOUD COMPUTING; DATA HANDLING; FORESTRY;

ANALYTICS SYSTEMS; COMPRESSED DATA STRUCTURES; MEMORY CAPACITY; MEMORY EFFICIENCY; MEMORY EFFICIENT; PROCESSING IN MEMORY; PROCESSING MODEL; RAPID GROWTH;

TREES (MATHEMATICS);

COMPUTATION; DATA PROCESSING; FORESTRY;

EID: 84893324422     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2523616.2523625     Document Type: Conference Paper

References (51)

1. Project Gutenberg. www.gutenberg.org.
2. LZO. oberhumer.com/opensource/lzo.
3. Storm. storm-project.net.
4. ØMQ (zeroMQ). zeromq.org.
5. B. Abali, H. Franke, X. Shen, D. E. Poff, and T. B. Smith. Performance of Hardware Compressed Main Memory. In Proceedings of the 7th International Symposium on HighPerformance Computer Architecture, HPCA'01, pages 73-81. IEEE Computer Society, 2001.
6. A. Aggarwal and S. Vitter, Jeffrey. The Input/Output Complexity of Sorting and Related Problems. Communications of the ACM, 31(9): 1116-1127, Sept. 1988.
7. L. Arge. The Buffer Tree: A Technique for Designing Batched External Data Structures. Algorithmica, 37(1): 1-24, 2003.
8. M. A. Bender, M. Farach-Colton, J. T. Fineman, Y. R. Fogel, B. C. Kuszmaul, and J. Nelson. Cache-Oblivious Streaming B-Trees. In Proceedings of the Nineteenth Annual ACM Symposium on Parallel Algorithms and Architectures, SPAA'07, pages 81-92. ACM, 2007.
9. K. Beyer and R. Ramakrishnan. Bottom-up Computation of Sparse and Iceberg CUBE. In Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data, SIGMOD'99, pages 359-370. ACM, 1999.
10. W. B. Cavnar and J. M. Trenkle. N-Gram-Based Text Categorization. In In Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, pages 161-175, 1994.
11. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst., 26(2): 4:1-4:26, June 2008.
12. B. Chattopadhyay, L. Lin, W. Liu, S. Mittal, P. Aragonda, V. Lychagina, Y. Kwon, and M. Wong. Tenzing: A SQL Implementation on the Mapreduce Framework. In Proceedings of the VLDB Endowment, Volume 4, pages 1318-1327, 2011.
13. S. Chaudhuri. An Overview of Query Optimization in Relational Systems. In Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, PODS'98, pages 34-43. ACM, 1998.
14. R. Chen, H. Chen, and B. Zang. Tiled-Mapreduce: Optimizing Resource Usages of Data-Parallel Applications on Multicore with Tiling. In Proceedings of the 19th International Conference on Parallel Architectures and Compilation Techniques, PACT'10, pages 523-534. ACM, 2010.
15. Z. Chen, J. Gehrke, and F. Korn. Query Optimization in Compressed Database Systems. In Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data, SIGMOD'01, pages 271-282. ACM, 2001.
16. J. Evans. A Scalable Concurrent malloc(3) Implementation for FreeBSD.
17. R. Fagin, J. Nievergelt, N. Pippenger, and H. R. Strong. Extendible Hashing - a Fast Access Method for Dynamic Files. ACM Trans. Database Syst., 4(3): 315-344, Sept. 1979.
18. B. Gedik, H. Andrade, K.-L. Wu, P. S. Yu, and M. Doo. SPADE: the System S Declarative Stream Processing Engine. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD'08, pages 1123-1134. ACM, 2008.
19. S. Ghemawat and P. Menage. Tcmalloc: Thread-caching malloc. goog-perftools.sourceforge.net/doc/tcmalloc.html.
20. Google. Snappy. code.google.com/p/snappy.
21. Google. Sparsehash. code.google.com/p/sparsehash.
22. G. Graefe. Query Evaluation Techniques for Large Databases. ACM Comput. Surv., 25(2): 73-169, June 1993.
23. J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh. Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals. Data Min. Knowl. Discov., 1(1): 29-53, Jan. 1997.
24. J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online Aggregation. In Proceedings of the 1997 ACM SIGMOD International Conference on Management of Data, SIGMOD'97, pages 171-182. ACM, 1997.
25. S. Huddleston and K. Mehlhorn. A New Data Structure for Representing Sorted Lists. Acta Informatica, 17(2): 157-184, 1982.
26. Intel Corporation. Intel Threading Building Blocks. www. threadingbuildingblocks.org.
27. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In EuroSys'07: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, pages 59-72. ACM, 2007.
28. A. Islam and D. Inkpen. Real-Word Spelling Correction using Google Web IT 3-grams. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Volume 3 of EMNLP'09, pages 1241-1249. Association for Computational Linguistics, 2009.
29. W. Jiang, V. T. Ravi, and G. Agrawal. A Map-Reduce System with an Alternate API for Multi-core Environments. In Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, CC-GRID'10, pages 84-93. IEEE Computer Society, 2010.
30. Y. Kotidis and N. Roussopoulos. An Alternative Storage Organization for ROLAP Aggregate Views Based on Cubetrees. In Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data, SIGMOD'98, pages 249-258. ACM, 1998.
31. A. Lakshman and P. Malik. Cassandra: A Decentralized Structured Storage System. SIGOPS Operating Systems Review, 44(2): 35-40, Apr. 2010.
32. W. Lam, L. Liu, S. Prasad, A. Rajaraman, Z. Vacheri, and A. Doan. Muppet: MapReduce-Style Processing of Fast Data. Proc. VLDB Endow., 5(12): 1814-1825, Aug. 2012.
33. B. Li, E. Mazur, Y. Diao, A. McGregor, and P. Shenoy. A Platform for Scalable One-Pass Analytics using MapReduce. In Proceedings of the 2011 International Conference on Management of Data, SIGMOD'11, pages 985-996. ACM, 2011.
34. J. Li, D. Rotem, and J. Srivastava. Aggregation Algorithms for Very Large Compressed Data Warehouses. In Proceedings of the 25th International Conference on Very Large Data Bases, VLDB'99, pages 651-662. Morgan Kaufmann Publishers Inc., 1999.
35. K. Lim, J. Chang, T. Mudge, P. Ranganathan, S. K. Reinhardt, and T. F. Wenisch. Disaggregated Memory for Expansion and Sharing in Blade Servers. In Proceedings of the 36th Annual International Symposium on Computer Architecture, ISCA'09, pages 267-278. ACM, 2009.
36. H. Liu. huanliu.wordpress.com/2011/01/24/the-true-cost-of-an-ecu/.
37. Y. Mao, R. Morris, and F. Kaashoek. Optimizing Mapreduce for Multicore Architectures. Technical Report MIT-CSAIL-TR-2010-020, MIT, 2010.
38. J. B. Mariòo, R. E. Banchs, J. M. Crego, A. de Gispert, P. Lambert, J. A. R. Fonollosa, and M. R. Costajussà. N-gram-based Machine Translation. Comput. Linguist., 32(4): 527-549, Dec. 2006.
39. P. Melsted and J. Pritchard. Efficient Counting of k-mers in DNA Sequences using a Bloom Filter. BMC Bioinfor-matics, 12(1): 1-7, 2011.
40. V. Monga and B. L. Evans. Robust Perceptual Image Hashing Using Feature Points. In PROC. IEEE Conference on Image Processing, pages 677-680, 2004.
41. P. O'Neil, E. Cheng, D. Gawlick, and E. O'Neil. The Log-Structured Merge-Tree (LSM-Tree). Acta Informatica, 33 (4): 351-385, June 1996.
42. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis. Evaluating MapReduce for Multi-core and Multiprocessor Systems. In Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, HPCA'07, pages 13-24. IEEE Computer Society, 2007.
43. L. Rizzo. A Very Fast Algorithm for RAM Compression. SIGOPS Oper. Syst. Rev., 31(2): 36-45, Apr. 1997.
44. M. Rosenblum and J. K. Ousterhout. The Design and Implementation of a Log-Structured File System. ACM Trans. Comput. Syst., 10(1): 26-52, Feb. 1992.
45. R. Sears and R. Ramakrishnan. bLSM: a General Purpose Log Structured Merge Tree. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD'12, pages 217-228. ACM, 2012.
46. J. Talbot, R. M. Yoo, and C. Kozyrakis. Phoenix++: Modular MapReduce for Shared-Memory Systems. In Proceedings of the Second International Workshop on MapReduce and its Applications, MapReduce'11, pages 9-16. ACM, 2011.
47. A. Torralba, R. Fergus, and W. Freeman. 80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(11): 1958-1970, Nov. 2008.
48. Y. Yu, P. K. Gunda, and M. Isard. Distributed Aggregation for Data-Parallel Computing: Interfaces and Implementations. In Proceedings of the ACM SIGOPS 22nd symposium on Operating Systems Principles. ACM, 2009.
49. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster Computing with Working Sets. In Proceedings of the 2nd USENIX conference on Hot Topics in Cloud Computing, HotCloud'10, pages 10-10. USENIX Association, 2010.
50. J. Ziv and A. Lempel. A Universal Algorithm for Sequential Data Compression. IEEE Trans. Inf. Theor., 23(3): 337-343, Sept. 1977.
51. J. Ziv and A. Lempel. Compression of Individual Sequences Via Variable-Rate Coding. Information Theory, IEEE Transactions on, 24(5): 530-536, 1978.