SparkNet: Training Deep Networks in Spark

By Philipp Moritz, Robert Nishihara, Ion Stoica, Michael I. Jordan. 2015

178 citations recorded by the source corpus.

Training deep networks is a time-consuming process, with networks for object recognition often requiring multiple days to train. For this reason, leveraging the resources of a cluster to speed up training is an important area of work. However, widely-popular batch-processing computational frameworks like MapReduce and Spark were not designed to support the asynchronous and communication-intensive workloads of existing distributed deep learning systems. We introduce SparkNet, a framework for training deep networks in Spark. Our implementation includes a convenient interface for reading data from Spark RDDs, a Scala interface to the Caffe deep learning framework, and a lightweight multi-dimensional tensor library. Using a simple parallelization scheme for stochastic gradient descent, SparkNet scales well with the cluster size and tolerates very high-latency communication. Furthermore, it is easy to deploy and use with no parameter tuning, and it is compatible with existing Caffe models. We quantify the dependence of the speedup obtained by SparkNet on the number of machines, the communication frequency, and the cluster's communication overhead, and we benchmark our system's performance on the ImageNet dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In ML Systems & Hardware

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, neutral): “For example, the SparkNet system for training deep neural networks on Spark takes 20 seconds to broadcast weights and collect updates from five workers [52].”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “A natural idea is to allow workers to perform τ local updates to the model instead of just computing gradients, and then periodically averaging the local models (Moritz et al., 2015; Zhang et al., 2016; Povey et al., 2014; Su & Chen, 2015; Chaudhari et al., 2017).”
Exploiting Unintended Feature Leakage in Collaborative Learning (2018, background): “Several architectures have been proposed for distributed, collaborative, and federated learning [9, 11, 33, 38, 62, 68]: with and without a central server, with different ways of aggregating models, etc.”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Large-scale distributed training improves the productivity of training deeper and larger models (Chilimbi et al., 2014; Xing et al., 2015; Moritz et al., 2015; Zinkevich et al., 2010).”
Distributed Statistical Machine Learning in Adversarial Settings (2017, background): “Many efficient distributed machine learning algorithms [6, 13] and system implementations [24, 28, 10, 19] have been proposed and studied.”
Bayesian Nonparametric Federated Learning of Neural Networks (2019, background): “There is also significant work on distributed deep learning (Lian et al., 2015; 2017; Moritz et al., 2015; Li et al., 2014; Dean et al., 2012).”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “The training of large-scale models with huge amounts of data are often carried on distributed systems [1][2][3][4][5][6][7][8][9], where data parallelism is adopted to exploit the compute capability empowered by multiple workers [10].”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, background): “Periodic averaging has been shown to offer signiﬁcant speedup in deep neural network training (Moritz et al., 2015; Zhang et al., 2016; Su & Chen, 2015; Zhou & Cong, 2017; Lin et al., 2018).”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “ently, and thus the overall algorithm has to be Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:23 Category Method Model Consistency Synchronization Synchronous[45,112,173,210,224,246] Stale-Synchronous[88,98,120,156,261] Asynchronous[55,56,127,182,190,205,260”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, methodology): “Despite their benefits, communication heavy training of deep nets is often based on custom implementations [8, 6, 36, 20] relying on the parameter server architecture [28, 27, 15], where the centralized server aggregates the gradients from workers and distributes the updated weights, either in a syn”
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters (2017, background): “Also of note are several efforts to port Caffe onto other distributed platforms, such as SparkNet [22], YahooCaffe [33] and FireCaffe [13], the former reports a 4-5 times speedup with 10 machines (and hence less scalability than our results herein).”
Distributed stochastic optimization for deep learning (thesis) (2016, unknown): “The mini-batch SGD and the asynchronous SGD methods that we have discussed so far can be implemented in a distributed computing environment [34, 26].”
Strategies and Principles of Distributed Machine Learning on Big Data (2015, background): “[25]), while deep learning on MapReduce-like systems such as Spark has yet to achieve 5× speedup with 10× machines [29].”
TF-Replicator: Distributed Machine Learning for Researchers (2019, methodology): “For example, the SparkNet system (based on Spark) requires 20 seconds to broadcast weights and collect updates from just 5 workers, despite extending earlier frameworks to store intermediate results in memory (Moritz et al., 2015).”
OD-SGD (2020, background): “Under this circumstance, distributed training becomes the prevailing practice and it greatly improves the productivity of training deeper and larger neural networks [7, 8, 9, 10].”