MLlib: Machine learning in Apache Spark

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Meng, Xiangrui ^a   Bradley, Joseph ^a   Yavuz, Burak ^a   Sparks, Evan ^b   Venkataraman, Shivaram ^b   Liu, Davies ^a   Freeman, Jeremy ^c   Tsai, D B ^d   Amde, Manish ^e   Owen, Sean ^f   Xin, Doris ^g   Xin, Reynold ^a   Franklin, Michael J ^b   Zadeh, Reza ^h   Zaharia, Matei ⁱ   Talwalkar, Ameet ^j  

^a Databricks Inc   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c HOWARD HUGHES MEDICAL INSTITUTE   (United States)

^d NETFLIX INC   (United States)

^e Origami Logic   (United States)

^f Cloudera UK   (United Kingdom)

^g UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^h STANFORD UNIVERSITY   (United States)

ⁱ MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^j UNIVERSITY OF CALIFORNIA   (United States)

more..

Author keywords

Apache spark; Distributed algorithms; Scalable machine learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; DATA HANDLING; HIGH LEVEL LANGUAGES; ITERATIVE METHODS; LINEAR ALGEBRA; PARALLEL ALGORITHMS;

DISTRIBUTED MACHINE LEARNING; LARGE-SCALE DATA PROCESSING; LEARNING SETTINGS; OPEN SOURCE COMMUNITIES; OPEN SOURCE PLATFORMS; OPEN SOURCES; RAPID GROWTH; SCALABLE MACHINE LEARNING;

LEARNING SYSTEMS;

EID: 84979900694     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (27)

1. Michael Armbrust, Reynold Xin, Cheng Lian, Yin Yuai, Davies Liu, Joseph Bradley, Xiangrui Meng, Tomer Kaftan, Michael Franklin, Ali Ghodsi, and Matei Zaharia. Spark SQL: Relational data processing in spark. In ACM Special Interest Group on Management of Data, 2015.
2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3, 2003.
3. Joseph Bradley. Topic modeling with LDA: MLlib meets GraphX. https://databricks.com/?p=3135, 2015.
4. Mikio Braun. jblas. http://jblas.org/, 2015.
5. Breeze. Breeze. https://github.com/scalanlp/breeze/wiki, 2015.
6. Lars Buitinck et al. API design for machine learning software: experiences from the scikit-learn project. arXiv:1309.0238, 2013.
7. Spark Community. Spark community. https://spark.apache.org/community.html, 2015.
8. Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Simplified data processing on large clusters. In USENIX Symposium on Operating Systems Design and Implementation, 2004.
9. Jeremy Freeman. Introducing streaming k-means in spark 1.2. https://databricks.com/?p=2382, 2015.
10. Joseph E. Gonzalez, Reynold S. Xin, Ankur Dave, Daniel Crankshaw, Michael J. Franklin, and Ion Stoica. Graphx: Graph processing in a distributed dataflow framework. In Conference on Operating Systems Design and Implementation, 2014.
11. Alex Guazzelli, Michael Zeller, Wen-Ching Lin, and Graham Williams. PMML: An open standard for sharing models. The R Journal, 1(1), 2009.
12. Sam Halliday. netlib-java. https://github.com/fommil/netlib-java, 2015.
13. Tim Kraska, Ameet Talwalkar, John Duchi, Rean Griffith, Michael Franklin, and Michael Jordan. MLbase: A Distributed Machine-learning System. In Conference on Innovative Data Systems Research, 2013.
14. Mahout. Apache mahout. http://mahout.apache.org/, 2014.
15. Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In ACM Recommender Systems Conference, 2013.
16. Xiangrui Meng, Joseph Bradley, Evan Sparks, and Shivaram Venkataraman. Ml pipelines: A new high-level api for MLlib. https://databricks.com/?p=2473, 2015.
17. MLlib. MLlib user guide. https://spark.apache.org/docs/latest/mllib-guide.html, 2015.
18. NumPy. Numpy. http://www.numpy.org/, 2015.
19. Spark Packages. Spark packages. https://spark-packages.org, 2015.
20. Biswanath Panda, Joshua S. Herbach, Sugato Basu, and Roberto J. Bayardo. Planet: Massively parallel learning of tree ensembles with mapreduce. In International Conference on Very Large Databases, 2009.
21. Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
22. Evan R. Sparks, Ameet Talwalkar, Virginia Smith, Jey Kottalam, Xinghao Pan, Joseph E. Gonzalez, Michael J. Franklin, Michael I. Jordan, and Tim Kraska. MLI: An API for Distributed Machine Learning. In International Conference on Data Mining, 2013.
23. Evan R Sparks, Ameet Talwalkar, Daniel Haas, Michael J. Franklin, Michael I. Jordan, and Tim Kraska. Automating model search for large scale machine learning. Symposium on Cloud Computing, 2015.
24. Ameet Talwalkar. BerkeleyX CS190-1x: Scalable machine learning. https://www.edx.org/course/scalable-machine-learning-uc-berkeleyx-cs190-1x, 2015.
25. Burak Yavuz and Xiangrui Meng. Spark 1.1: MLlib performance improvements. https://databricks.com/?p=1393, 2014.
26. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In USENIX Symposium on Networked Systems Design and Implementation, 2012.
27. Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. Discretized streams: Fault-tolerant streaming computation at scale. In Symposium on Operating Systems Principles, 2013.