Parameter Server for Distributed Machine Learning

By Mu Li, Li Zhou, Zichao Yang, Aaron Q. Li, D. Andersen, Alex Smola. 2013

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization (2015, methodology): “…2011, Zhang et al., 2014, Feyzmahdavian et al., 2015, Paine et al., 2013, Mania et al., 2015], stochastic coordinate descent [Avron et al., 2014, Liu et al., 2014a, Sridhar et al., 2013], dual stochastic coordinate ascent [Tran et al., 2015], and randomized Kaczmarz algorithm [Liu et al.,…”
Practical Network Blocks Design with Q-Learning (2017, background): “It’s like a simpliﬁed version of parameter-server (Dean et al. 2012; Li et al. 2013).”
A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order (2016, background): “One of core theoretical questions is how much speedup (or benefit) the asynchronous parallelization can bring us.”
TF-Replicator: Distributed Machine Learning for Researchers (2019, methodology): “Subsequent systems have extended the parameter server model, introduced by (Smola & Narayanamurthy, 2010) and later popularized by (Li et al., 2013; 2014) for machine learning.”
Asynchronous Online Federated Learning for Edge Devices with Non-IID Data (2019, background): “Parameter servers, where server nodes maintain globally shared parameters with data distributed on local nodes, however, often suffer from problems such as high network bandwidth or communication overhead [1], [12], [13].”
Practical Block-Wise Neural Network Architecture Generation (2017, background): “It can be regarded as a simplified parameter-server [5, 18].”
Online Machine Learning in Big Data Streams (2018, methodology): “The parameter server introduced in [209] is a popular way to implement model parallel training, with several variants developed [115, 159, 157, 158], including an”
IncBricks: Toward In-Network Computation with an In-Network Cache (2017, methodology): “Modern applications such as big data analytics and distributed machine learning workloads [7, 11, 18, 31, 49] generate a tremendous amount of network trafﬁc.”
Communication Efficient Distributed Machine Learning with the Parameter Server (2014, methodology): “The second generation parameter servers were application specific, exemplified by Distbelief [10] and the synchronization mechanism in [20].”
Federated Learning for 6G: Applications, Challenges, and Opportunities (2021, methodology): “FRL can also be used to study the user association and beamforming of a multi-antenna NOMA network [70].”
Parallel and Distributed Block-Coordinate Frank-Wolfe Algorithms (2014, methodology): “On a distributed system, the parameter server (Dai et al., 2013; Li et al., 2013) broadcasts the most recent parameter vector periodically to each worker and workers keep sending updates to the parameter vector after solving the subroutines corresponding to a randomly chosen parameter.”
Scaling Distributed Machine Learning with the Parameter Server (2014, unknown): “This second generation of application specific parameter servers can also be found in Distbelief [13] and the synchronization mechanism of [33].”
Asynchronous Byzantine Machine Learning (2018, methodology): “We consider the general distributed model for machine learning, namely the parameter server (Dean et al., 2012; Abadi et al., 2016; Li et al., 2013; 2014b;a; Ho et al., 2013; Cui et al., 2016)1.”
On Convergence of Model Parallel Proximal Gradient Algorithm for Stale Synchronous Parallel System (2016, methodology): “Asymptotic convergence is established for proximal gradient algorithm for data parallelism [23] and for”
Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, unknown): “Parameter Server is the prototype of FL [14] and implements local computation and global parameter sharing through a set of worker nodes and a set of master nodes.”
A Survey on Distributed Machine Learning (2019, background): “E � ectively, it combines Parameter Servers (PS) [156] with Su � cient Factor Broadcasting (SFB) [160], choosing the best communication method depending on the sparsity of the parameter tensor.”