Spark SQL: Relational data processing in spark

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn 2015-May, Issue , 2015, Pages 1383-1394

  Armbrust, Michael ^a   Xin, Reynold S ^a   Lian, Cheng ^a   Huai, Yin ^a   Liu, Davies ^a   Bradley, Joseph K ^a   Meng, Xiangrui ^a   Kaftan, Tomer ^c   Frankliny, Michael J ^a,c   Ghodsi, Ali ^a   Zaharia, Matei ^a,b  

^a Databricks Inc   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Data warehouse; Databases; Hadoop; Machine learning; Spark

Indexed keywords

APPLICATION PROGRAMMING INTERFACES (API); ARTIFICIAL INTELLIGENCE; CATALYSTS; DATA WAREHOUSES; DATABASE SYSTEMS; DIGITAL STORAGE; ELECTRIC SPARKS; FUNCTIONAL PROGRAMMING; LEARNING SYSTEMS;

CONTROL CODES; EXTENSION POINTS; EXTERNAL DATABASE; HADOOP; PROGRAMMING MODELS; RELATIONAL DATA; RELATIONAL PROCESSING; SCHEMA INFERENCE;

DATA HANDLING;

EID: 84953884359     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2723372.2742797     Document Type: Conference Paper

References (40)

1. A. Abouzied, D. J. Abadi, and A. Silberschatz. Invisible loading: Access-driven data transfer from raw files into database systems. In EDBT, 2013.
2. A. Alexandrov et al. The Stratosphere platform for big data analytics. The VLDB Journal, 23(6):939-964, Dec. 2014.
3. AMPLab big data benchmark. https://amplab.cs.berkeley.edu/benchmark.
4. Apache Avro project. http://avro.apache.org.
5. Apache Parquet project. http://parquet.incubator.apache.org.
6. Apache Spark project. http://spark.apache.org.
7. M. Armbrust, N. Lanham, S. Tu, A. Fox, M. J. Franklin, and D. A. Patterson. The case for PIQL: a performance insightful query language. In SOCC, 2010.
8. A. Behm et al. Asterix: towards a scalable, semistructured data platform for evolving-world models. Distributed and Parallel Databases, 29(3):185-216, 2011.
9. G. J. Bex, F. Neven, and S. Vansummeren. Inferring XML schema definitions from XML data. In VLDB, 2007.
10. BigDF project. https://github.com/AyasdiOpenSource/bigdf.
11. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: Easy, efficient data-parallel pipelines. In PLDI, 2010.
12. J. Cohen, B. Dolan, M. Dunlap, J. Hellerstein, and C. Welton. MAD skills: new analysis practices for big data. VLDB, 2009.
13. DDF project. http://ddf.io.
14. B. Emir, M. Odersky, and J. Williams. Matching objects with patterns. In ECOOP 2007 - Object-Oriented Programming, volume 4609 of LNCS, pages 273-298. Springer, 2007.
15. J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. Franklin, and I. Stoica. GraphX: Graph processing in a distributed dataflow framework. In OSDI, 2014.
16. G. Graefe. The Cascades framework for query optimization. IEEE Data Engineering Bulletin, 18(3), 1995.
17. G. Graefe and D. DeWitt. The EXODUS optimizer generator. In SIGMOD, 1987.
18. J. Hegewald, F. Naumann, and M. Weis. XStruct: efficient schema extraction from multiple and large XML documents. In ICDE Workshops, 2006.
19. Hive data definition language. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL.
20. M. Isard and Y. Yu. Distributed data-parallel computing using a high-level programming language. In SIGMOD, 2009.
21. Jackson JSON processor. http://jackson. codehaus.org.
22. Y. Klonatos, C. Koch, T. Rompf, and H. Chafi. Building efficient query engines in a high-level language. PVLDB, 7(10):853-864, 2014.
23. M. Kornacker et al. Impala: A modern, open-source SQL engine for Hadoop. In CIDR, 2015.
24. Y. Low et al. Distributed GraphLab: a framework for machine learning and data mining in the cloud. VLDB, 2012.
25. S. Melnik et al. Dremel: interactive analysis of web-scale datasets. Proc. VLDB Endow., 3:330-339, Sept 2010.
26. X. Meng, J. Bradley, E. Sparks, and S. Venkataraman. ML pipelines: a new high-level API for MLlib. https://databricks.com/blog/2015/01/07/ml-pipelines-a-newhighlevel-api-for-mllib.html.
27. S. Nestorov, S. Abiteboul, and R. Motwani. Extracting schema from semistructured data. In ICDM, 1998.
28. F. A. Nothaft, M. Massie, T. Danford, Z. Zhang, U. Laserson, C. Yeksigian, J. Kottalam, A. Ahuja, J. Hammerbacher, M. Linderman, M. J. Franklin, A. D. Joseph, and D. A. Patterson. Rethinking data-intensive science using scalable analytics systems. In SIGMOD, 2015.
29. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a not-so-foreign language for data processing. In SIGMOD, 2008.
30. pandas Python data analysis library. http://pandas.pydata.org.
31. A. Pavlo et al. A comparison of approaches to large-scale data analysis. In SIGMOD, 2009.
32. R project for statistical computing. http://www.r-project.org.
33. scikit-learn: machine learning in Python. http://scikit-learn.org.
34. D. Shabalin, E. Burmako, and M. Odersky. Quasiquotes for Scala, a technical report. Technical Report 185242, École Polytechnique Fédérale de Lausanne, 2013.
35. D. Tahara, T. Diamond, and D. J. Abadi. Sinew: A SQL system for multi-structured data. In SIGMOD, 2014.
36. A. Thusoo et al. Hive-a petabyte scale data warehouse using Hadoop. In ICDE, 2010.
37. P. Wadler. Monads for functional programming. In Advanced Functional Programming, pages 24-52. Springer, 1995.
38. R. S. Xin, J. Rosen, M. Zaharia, M. J. Franklin, S. Shenker, and I. Stoica. Shark: SQL and rich analytics at scale. In SIGMOD, 2013.
39. M. Zaharia et al. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In NSDI, 2012.
40. K. Zeng et al. G-OLA: Generalized online aggregation for interactive analysis on big data. In SIGMOD, 2015.