Spark: Cluster Computing with Working Sets

By Matei A. Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, Ion Stoica. 2010

5,012 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, methodology): “Spark [102] has been established as a popular open source framework for implementing distributed iterative algorithms, and includes several of the algorithms mentioned in this section.”
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server (2013, background): “(2) Building high-throughput distributed ML architectures or algorithm implementations that feature signiﬁcant systems contributions but relatively less theoretical analysis, such as GraphLab [18], Spark [27], Pregel [19], and YahooLDA [2].”
SparkNet: Training Deep Networks in Spark (2015, background): “SparkNet builds on Apache Spark (Zaharia et al., 2010) and the Caffe deep learning library (Jia et al., 2014).”
Distributed GraphLab: A Framework for Machine Learning in the Cloud (2012, methodology): “(a) [38] describes and iterative extension of MapReduce.”
Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication (2017, methodology): “As a result, distributed computing frameworks such as Hadoop MapReduce [1] and Spark [2] have gained signiﬁcant traction, as they enable processing of data sizes at the order of tens of terabytes and more.”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, methodology): “To take advantage of parallelization across machines, a variety of approaches have been developed starting from the popular MapReduce paradigm [9, 51, 19, 37].”
Superneurons: dynamic GPU memory management for training deep neural networks (2018, methodology): “MXNet also implements a per-layer-based re-computation strategy that is similar to Resilient Distributed Datasets (RDD) in Spark [28].”
BigDansing: A System for Big Data Cleansing (2015, methodology): “As shown in the experiment section, scalable data processing platform, such as MapReduce [8] or Spark [41], can implement the violation detection process.”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, methodology): “Spark [193] has been established as a popular open source framework for implementing distributed iterative algorithms, and includes several of the algorithms mentioned in this section.”
On Model Parallelization and Scheduling Strategies for Distributed Machine Learning (2014, background): “Crucially, these frameworks allow the user to specify a coarse order to parameter updates, but automatically decide on the precise execution order — for example, MapReduce and Spark allow users to specify that parallel jobs should be executed in some topological order; e.g. mappers are guaranteed to”
The rise of "big data" on cloud computing: Review and open research issues (2015, methodology): “…registers (z-nodes) similar to a file system; ZooKeeper is a distributed service with master and slave nodes and stores configuration information [82] Spark TM A fast and general computation engine for Hadoop data [83] Chukwa Chukwa has just passed its development stage; it is a data collection…”
GPS: a graph processing system (2013, background): “• Other systems: Spark [38] is a general cluster computing system, whose API is designed to express generic iterative computations.”
Coded Computation Over Heterogeneous Clusters (2019, methodology): “G ENERAL distributed computing frameworks, such as MapReduce [2] and Spark [3], along with the availability of large-scale commodity servers, such as Amazon EC2, have made it possible to carry out large-scale data analytics at the production level.”
A Survey of Traffic Prediction: from Spatio-Temporal Data to Intelligent Transportation (2021, methodology): “One popular parallel framework is called Map-Reduce, based on which the distributed system Hadoop [80] is created.”