More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server

By Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B. Gibbons, Garth A. Gibson, G. R. Ganger, Eric P. Xing. 2013

813 citations recorded by the source corpus.

We propose a parameter server system for distributed ML, which follows a Stale Synchronous Parallel (SSP) model of computation that maximizes the time computational workers spend doing useful work on ML algorithms, while still providing correctness guarantees. The parameter server provides an easy-to-use shared interface for read/write access to an ML model’s values (parameters and variables), and the SSP model allows distributed workers to read older, stale versions of these values from a local cache, instead of waiting to get them from a central storage. This significantly increases the proportion of time workers spend computing, as opposed to waiting. Furthermore, the SSP model ensures ML algorithm correctness by limiting the maximum age of the stale values. We provide a proof of correctness under SSP, as well as empirical results demonstrating that the SSP model achieves faster algorithm convergence on several different ML problems, compared to fully-synchronous and asynchronous schemes.

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “In this paper, we consider a centralized server-worker architecture, also known as the Parameter Server (PS) architecture (Li et al., 2014a,b; Ho et al., 2013).”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “There are many large-scale ML systems that aim to enable efficient ML training over centralized datasets using communicationefficient designs, such as relaxing synchronization requirements (Recht et al., 2011; Ho et al., 2013; Goyal et al., 2017) or sending fewer updates to parameter servers (Li et ”
Learning Privately over Distributed Features: An ADMM Sharing Approach (2019, background): “…a wide range of distributed and collaborative machine learning schemes, including gradient-based meth-ods Li et al. (2014a,b); Hsieh et al. (2017); Ho et al. (2013) and ADMM-based methods Zhang et al. (2018); Shi et al. (2014); Zhang and Zhu (2016a); Huang et al. (2018), have been proposed to…”
Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, methodology): “Stochastic Gradient Descent (SGD) with PS architecture, is widely used in typical distributed machine learning (Li et al., 2014a; Ho et al., 2013; Li et al., 2014b).”
Federated Machine Learning (2019, methodology): “ibuted machine learning covers many aspects, including distributed storage of training data, distributed operation of computing tasks, distributed distribution of model results, etc. Parameter Server [30] is a typical element in distributed machine learning. As a tool to accelerate the training proc”
Deep learning with Elastic Averaging SGD (2014, neutral): “Also in order to minimize the staleness [16] of the difference xt − x̃t between the center and the local variable, the update for the master in Equation 4 involves xt instead of x i t+1.”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “usceptible to stragglers in the face of device variability. Asynchronous schemes are an attractive approach to mitigate stragglers in heterogeneous environments, particularly in shared-memory systems [27, 30, 48, 92, 141]. However, they typically rely on bounded-delay assumptions to control the degr”
Generalized Byzantine-tolerant SGD (2018, methodology): “It is very popular to partition the parameters into disjoint subsets, and use multiple server nodes to storage and aggregate them (Li et al., 2014a;b; Ho et al., 2013).”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “A common approach to overcome such a network bottleneck is asynchronous SGD [1][4][7][12][13][14], which continues computation by using stale values without waiting for the completeness of synchronization.”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “Modern large-scale machine learning applications require scaling stochastic optimization algorithms [30, 28, 15, 11] to distributed computational architectures [10, 39, 12, 20, 19, 38] or multicore systems [26, 9, 22, 25].”
SparkNet: Training Deep Networks in Spark (2015, methodology): “Chilimbi et al. (2014) build Project Adam, a system for training deep networks on hundreds of machines using asynchronous SGD. Li et al. (2014); Ho et al. (2013) build parameter servers to exploit model and data parallelism, and though their systems are better suited to sparse gradient updates,…”
How to scale distributed deep learning? (2016, background): “Large scale parameter server systems in the non-deep learning setting have also been demonstrated in [14] and [15].”
Demystifying Parallel and Distributed Deep Learning (2018, background): “In SSP (Figure 20(d)), the gradient staleness is enforced to be bounded by performing a global synchronization step after a maximal staleness may have been reached by one of the nodes.”
Petuum: A New Platform for Distributed Machine Learning on Big Data (2013, methodology): “implements the Eager Stale Synchronous Parallel (ESSP) consistency model [14], [23], which reduces network synchronization and communication costs, while maintaining bounded-staleness convergence guarantees implied by ESSP.”
Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study (2015, background): “In any parameter server based deep learning system [12], staleness will negatively impact model accuracy.”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, methodology): “Finally, we prove the convergence of Pipe-SGD for convex and strongly convex objectives by adjusting the proof of [23, 15].”
Asynchronous Federated Optimization (2019, methodology): “Typically, a federated learning system is composed of servers and workers, whose architecture is similar to parameter servers (Li et al., 2014a;b; Ho et al., 2013).”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “As it is important that the results generated through our proposed system configuration are correct [16, 21, 54], we show the convergence of our methodology in Section 6.”