MLlib: Machine Learning in Apache Spark
By Xiangrui Meng, Joseph K. Bradley, B. Yavuz, Evan R. Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, Sean Owen and 6 more. 2015
1,861 citations recorded by the source corpus.
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.
Subfields: Artificial Intelligence, Computer Networks and Communications, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Michael I. Jordan
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Martin Jaggi
- Mohit Bansal
- Ruslan Salakhutdinov
In Computer Networks and Communications
- Ion Stoica
- Scott Shenker
- M. Franklin
- Jean-Pierre Hubaux
- Shusen Wang
- Virginia Smith
- Matei A. Zaharia
- Ameet Talwalkar
In ML Systems & Hardware
- Ion Stoica
- Michael I. Jordan
- Joseph E. Gonzalez
- M. Franklin
- Matei A. Zaharia
- Scott Shenker
- Ameet Talwalkar
- Stephen J. Wright
Citations in the corpus
- IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, background): “FL stands in contrast to established distributed training systems [17, 21], where all data is transmitted to a central data center and is subsequently distributed among cluster nodes to train in parallel.”
- On the Convergence of FedAvg on Non-IID Data (2019, methodology): “Note that FedAvg with E = 1 and full batch size is exactly the Full Gradient Descent; with a proper and fixed learning rate, its global convergence to the optimum is guaranteed Nesterov (2013). However, Theorem 4 shows that FedAvg with E > 1 and full batch size cannot possibly converge to the optimu”
- CoCoA: A General Framework for Communication-Efficient Distributed Optimization (2016, methodology): “5.0 (Meng et al., 2016).”
- A Hybrid Approach to Privacy-Preserving Federated Learning (2018, background): “For example, MLlib from Apache Spark assumes a trusted central node to coordinate distributed learning processes [28].”
- Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Examples include Spark MLib [83], Mahout(33) and SystemML [16].”
- Non-stochastic Best Arm Identification and Hyperparameter Optimization (2015, background): “We ignore this technicality in our experiments. such as LibSVM (Chang & Lin, 2011), scikit-learn (Pedregosa et al., 2011) and MLlib (Kraska et al., 2013).”
- GIANT: Globally Improved Approximate Newton Method for Distributed Optimization (2017, methodology): “In this light, several modern distributed computing architectures, e.g., MapReduce (Dean and Ghemawat, 2008), Apache Spark (Zaharia et al., 2010, Meng et al., 2016), GraphLab (Low et al., 2012), and Parameter Server (Li et al., 2014), have been designed to operate on and learn from data at massive…”
- Clipper: A Low-Latency Online Prediction Serving System (2016, methodology): “We implemented Clipper in Rust and added support for several of the most widely used machine learning frameworks: Apache Spark MLLib [40], Scikit-Learn [51], Caffe [31], TensorFlow [1], and HTK [63].”
- Low latency RNN inference with cellular batching (2018, background): “Clipper [17] is a general-purpose serving system that supports a variety of machine learning system back-ends, such as TensorFlow, Spark MLlib [29], and Caffe [25].”
- Probabilistic Demand Forecasting at Scale (2017, methodology): “Many approaches [20, 27] build on variants of the popular pipeline abstraction in sci-kit learn [21].”
- KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics (2016, methodology): “To assemble such pipelines, developers typically piece together domain specific libraries1 for feature extraction and general purpose numerical optimization packages [34, 44] for supervised learning.”
- A Berkeley View of Systems Challenges for AI (2017, background): “…the growing demand for machine learning (ML), researchers and practitioners built libraries on top of these systems to satisfy this demand [8, 52, 75]. recent successes of deep learning (DL) have spurred a new wave of specialized software systems have emerged to scale out these workloads on CPU…”
- A comparative evaluation of outlier detection algorithms: Experiments and analyses (2018, neutral): “Distributed implementations, streaming or mini-batch training are a prerequisite to deal with large datasets, and several methods have been extended to support these features, e.g. dpmm, k-means, svm or gmm on Spark MLlib [20].”
- The Role of Big Data Analytics in Industrial Internet of Things (2019, neutral): “IIoT clusters transfer data streams to distributed analytics servers that run Apaches Hadoop [83], MapReduce [84] and Spark [85] technologies for data processing and BDA.”