Communication Efficient Distributed Machine Learning with the Parameter Server

By Mu Li, D. Andersen, Alex Smola, Kai Yu. 2014

704 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “Some approaches (Harlap et al., 2018; Chen et al., 2018) use a parameter server (Li et al., 2014) in conjunction with pipeline parallelism.”
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems (2015, methodology): “We implemented KVStore based on the parameter server [8, 9, 4](Figure 5).”
Federated Optimization in Heterogeneous Networks (2018, methodology): “…proximal term we introduce has also been explored in previous optimization literature with different purposes, such as Allen-Zhu (2018), to speed up (mini-batch) SGD training on a single machine, and in Li et al. (2014b) for efficient SGD training both in a single machine and distributed settings.”
rTop-k: A Statistical Estimation Approach to Distributed SGD (2020, neutral): “SGD methods [1], [3]–[15], as well as other fundamental contributions to distributed first-order [16]–[20] and secondorder [21]–[25] optimization procedures.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “[16] Mu Li, David G Andersen, Alexander J Smola, and Kai Yu.”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “Some of them propose communication-efﬁcient designs, such as relaxing synchronization requirements (Recht et al., 2011; Ho et al., 2013; Goyal et al., 2017) or sending fewer updates to parameter servers (Li et al., 2014a;b).”
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, background): “Distributed optimization plays a pivotal role in distributed machine learning applications (Ahmed et al., 2013; Dean et al., 2012; Li et al., 2013, 2014) that commonly aims to minimize 1 N =1 f n ( θ ) with N workers.”
On the Convergence of FedAvg on Non-IID Data (2019, methodology): “If the data are iid, distributed optimization can be efficiently solved by the second-order algorithms Mahajan et al. (2018); Reddi et al.”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, background): “For large-scale distributed machine learning problems, such as training deep neural networks, a parallel version of SGD, known as parallel minibatch SGD, is widely adopted (Dean et al., 2012; Dekel et al., 2012; Li et al., 2014).”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the notable references in a chronological order are Langford et al. (2009); Niu et al. (2011); Agarwal and Duchi (2011); Paine et al. (2013); Li et al. (2014b); Zhang et al. (2014); Keuper and Pfreundt (2015); De and Goldstein (2015); Feyzmahdavian et al. (2015); Lian et al. (2015); Mania…”
Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, methodology): “Stochastic Gradient Descent (SGD) with PS architecture, is widely used in typical distributed machine learning (Li et al., 2014a; Ho et al., 2013; Li et al., 2014b).”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, methodology): “A popular class of distributed learning algorithms is Synchronous Stochastic Gradient Descent (sync-SGD), using a single server (called Parameter Server - PS) and n workers, also called nodes (Li et al., 2014a;b).”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “…parallel method to solve problem (1) is to sample each worker’s local stochastic gradient in parallel, aggregate all gradients in a single server to obtain the average, and update each worker’s local solution using the averaged gradient in its SGD step1 (Dekel et al. 2012) (Li et al. 2014).”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Often, the communication cost of sending the gradient becomes the bottleneck [30, 23, 22].”
The Hidden Vulnerability of Distributed Learning in Byzantium (2018, methodology): “We follow the usual parameter server model used in distributed implementations of machine learning (Dean et al., 2012; Li et al., 2014a;b).”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Asynchronous SGD accelerates the training by removing gradient synchronization and updating parameters immediately once a node has completed back-propagation [5, 10, 11].”
AIDE: Fast and Communication Efficient Distributed Optimization (2016, methodology): “[9] Mu Li, David G Andersen, Alex J Smola, and Kai Yu.”
Generalized Byzantine-tolerant SGD (2018, methodology): “It is very popular to partition the parameters into disjoint subsets, and use multiple server nodes to storage and aggregate them (Li et al., 2014a;b; Ho et al., 2013).”