Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

By Matei A. Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, M. Franklin, Scott Shenker, Ion Stoica. 2012

4,836 citations recorded by the source corpus.

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, neutral): “Spark [75] extends DryadLINQ with the ability to cache previously computed datasets in memory, and is therefore better suited to iterative machine learning algorithms (such as k -means clustering and logistic regression) when the input data ﬁt in memory.”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “Modern large-scale machine learning applications require scaling stochastic optimization algorithms [30, 28, 15, 11] to distributed computational architectures [10, 39, 12, 20, 19, 38] or multicore systems [26, 9, 22, 25].”
Adding vs. Averaging in Distributed Primal-Dual Optimization (2015, methodology): “We present experiments on several large real-world datasets distributed across multiple machines, running on Apache Spark .”
Communication-Efficient Distributed Dual Coordinate Ascent (2014, methodology): “We assume that there existsΘ ∈ [0, 1) such that for any givenα, LOCALDUAL METHOD when run on blockk alone returns a (possibly random) update∆α[k] such that E[ǫD,k((α[1], . . . ,α[k−1],α[k] +∆α[k],α[k+1], . . . ,α[K]))] ≤ Θ · ǫD,k(α).”
MLlib: Machine Learning in Apache Spark (2015, methodology): “Keywords: Scalable Machine Learning, Distributed Algorithms, Apache Spark”
Ray: A Distributed Framework for Emerging AI Applications (2017, methodology): “To meet the performance requirements, Ray distributes two components that are typically centralized in existing frameworks [64, 28, 40]: (1) the task scheduler and (2) a state (si+1) (observation)”
A reliable effective terascale linear learning system (2011, background): “However, the abstraction is rather ill-suited for machine learning algorithms as several researchers in the field have observed (Low et al., 2010; Zaharia et al., 2011), because it does not easily allow iterative algorithms, such as typical optimization algorithms used to solve the problem (1).”
Tune: A Research Platform for Distributed Model Selection and Training (2018, methodology): “In contrast to popular distributed frameworks such as Spark (Zaharia et al. (2012)), or MPI (Gabriel et al. (2004)), Ray oﬀers a more ﬂexible programming model.”
Clipper: A Low-Latency Online Prediction Serving System (2016, methodology): “We implemented Clipper in Rust and added support for several of the most widely used machine learning frameworks: Apache Spark MLLib [40], Scikit-Learn [51], Caffe [31], TensorFlow [1], and HTK [63].”
Flexible Scheduling of Distributed Analytic Applications (2016, methodology): “Our algorithm is evaluated using trace-driven simulations, with large-scale real system traces: our flexible scheduler outperforms a baseline approach across a variety of metrics, including application turnaround times, and resource allocation efficiency.”
kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data (2017, background): “Among them, Spark [20] highlights as one of the most flexible and powerful engines to performed faster distributed computing in big data by using in-memory primitives.”
Hopper: Decentralized Speculation-aware Cluster Scheduling at Scale (2015, background): “The scheduler places requests at the workers for its tasks; if a task has locality constraints, its requests are only placed on the workers meeting its constraints [13, 40, 49].”
Occupy the cloud: distributed computing for the 99% (2017, methodology): “For example, PySpark, arguably one of the easier to use platforms, runs on top of Spark [48] (written in Scala) which interoperates and is closely coupled with Hadoop File System [39] (written in Java), Yarn [44] (Java again), and the JVM.”
MillWheel: Fault-Tolerant Stream Processing at Internet Scale (2013, methodology): “Speciﬁcally, this model depends heavily on RDDs [33], which limit users to rollback-capable operators.”
A Berkeley View of Systems Challenges for AI (2017, methodology): “Google developed MapReduce [32] and Google File System [43], followed shortly by the open-source counterpart, Apache Hadoop [7]. came a plethora of systems [46, 55, 60, 67, 114], that aimed to improve speed, scale, and ease of use.”
BigBench V2: The New and Improved BigBench (2017, methodology): “Apache Spark [4] [34] has became a popular alternative to the MapReduce framework, promising faster processing and offering advanced analytical capabilities by SparkSQL [35].”
Strategies and Principles of Distributed Machine Learning on Big Data (2015, methodology): “This presents ML practitioners with a choice between the aforementioned monolithic yet high-performance “towers” (specialized systems that require substantial engineering to maintain and upgrade), or the more general-purpose yet slower “platforms” such as Hadoop and Spark (which are relatively easy ”
Tupleware: Distributed Machine Learning on Small Clusters (2014, background): “Tupleware also has commonalities with the programming models proposed by Spark [36] and Stratosphere [21].”