MLlib: Machine Learning in Apache Spark

By Xiangrui Meng, Joseph K. Bradley, B. Yavuz, Evan R. Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, Sean Owen and 6 more. 2015

1,861 citations recorded by the source corpus.

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

Subfields: Artificial Intelligence, Computer Networks and Communications, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Michael I. Jordan
  3. Yoshua Bengio
  4. Percy Liang
  5. Quoc V. Le
  6. Martin Jaggi
  7. Mohit Bansal
  8. Ruslan Salakhutdinov

In Computer Networks and Communications

  1. Ion Stoica
  2. Scott Shenker
  3. M. Franklin
  4. Jean-Pierre Hubaux
  5. Shusen Wang
  6. Virginia Smith
  7. Matei A. Zaharia
  8. Ameet Talwalkar

In ML Systems & Hardware

  1. Ion Stoica
  2. Michael I. Jordan
  3. Joseph E. Gonzalez
  4. M. Franklin
  5. Matei A. Zaharia
  6. Scott Shenker
  7. Ameet Talwalkar
  8. Stephen J. Wright

Citations in the corpus