Scaling Distributed Machine Learning with the Parameter Server

By Mu Li, D. Andersen, Jun Woo Park, Alex Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, Bor-Yiing Su. 2014

1,926 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “Some approaches (Harlap et al., 2018; Chen et al., 2018) use a parameter server (Li et al., 2014) in conjunction with pipeline parallelism.”
TensorFlow: A system for large-scale machine learning (2016, neutral): “We refer to the computation at vertices as operations , and the values that ﬂow along edges as tensors , because TensorFlow is designed for mathematical computation, and uses tensors (or multi-dimensional arrays) to represent all data in those computations.”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “Each GPU maintains a full copy of the model and trains on its own partition of data while periodically synchronizing weights with other GPUs, using either collective communication primitives [14] or Parameter Servers [28, 9].”
Revisiting Distributed Synchronous SGD (2016, background): “algorithms include Recht et al. (2011); Duchi et al. (2013); Zhang et al. (2015a); Reddi et al. (2015); Leblond et al. (2016). Implementations of asynchronous optimization include Xing et al. (2015); Li et al. (2014); Chilimbi et al. (2014). Attempts have also been made in Zinkevich et al. (2010) an”
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems (2015, methodology): “Finally, model divergence is controlled via consistency model [8].”
IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, background): “FL stands in contrast to established distributed training systems [17, 21], where all data is transmitted to a central data center and is subsequently distributed among cluster nodes to train in parallel.”
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, background): “Unlike traditional distributed optimization [6, 7] where consensus (either through a central server or peer-to-peer communication) is performed after every local gradient computation, in federated learning, the subset of clients selected in each communication round perform multiple local updates bef”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Related work In this paper, we consider a centralized server-worker architecture, also known as the Parameter Server (PS) [9, 15, 16, 21].”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Such a problem is traditionally solved under centralized optimization paradigms such as parameter servers [17].”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “Some of them propose communication-efﬁcient designs, such as relaxing synchronization requirements (Recht et al., 2011; Ho et al., 2013; Goyal et al., 2017) or sending fewer updates to parameter servers (Li et al., 2014a;b).”
Learning Privately over Distributed Features: An ADMM Sharing Approach (2019, background): “Recently, a wide range of distributed and collaborative machine learning schemes, including gradient-based meth-ods Li et al. (2014a,b); Hsieh et al. (2017); Ho et al. (2013) and ADMM-based methods Zhang et al. (2018); Shi et al. (2014); Zhang and Zhu (2016a); Huang et al. (2018), have been…”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, methodology): “For example, the parameter server [Abadi et al., 2016, Li et al., 2014], AllReduce [Renggli et al., 2018, Seide and Agarwal, 2016], adaptive distributed learning, diﬀerentially private distributed learning, distributed proximal primal-dual algorithm, non-smooth distributed optimization,…”
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling (2019, background): “There have been massive amount of work in the context of algorithms [37, 33, 36, 5, 26] and systems [16, 43, 9] that improve the communication efficiency of synchronous distributed SGD in a fully-connected network.”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “For example, the parameter server (Abadi et al., 2016b; Li et al., 2014) and AllReduce that averages models over a ring topology (Renggli et al.”
Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, methodology): “Stochastic Gradient Descent (SGD) with PS architecture, is widely used in typical distributed machine learning (Li et al., 2014a; Ho et al., 2013; Li et al., 2014b).”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, methodology): “A popular class of distributed learning algorithms is Synchronous Stochastic Gradient Descent (sync-SGD), using a single server (called Parameter Server - PS) and n workers, also called nodes (Li et al., 2014a;b).”
Towards Federated Learning at Scale: System Design (2019, background): “A particularly common approach in the datacenter is the parameter server , e.g., Li et al. (2014); Dean et al. (2012); Abadi et al. (2016), which allows a large number of workers to collaborate on a shared global model, the parameter vector.”
Distributed Learning over Unreliable Networks (2018, methodology): “Our modeling covers two standard distributed settings: the Parameter Server model (Li et al., 2014; Abadi et al., 2016)1, as well as standard implementations of the AllReduce averaging operation in a decentralized setting (Seide and Agarwal, 2016; Renggli et al., 2018).”