Parallelized Stochastic Gradient Descent

By Martin A. Zinkevich, Markus Weimer, Alex Smola, Lihong Li. 2010

1,440 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “In the DL community, data parallelism is generally used as the de facto standard to scale DL training to multiple GPUs [5, 26, 35].”
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (2011, methodology): “This recursion underlies many convergence proofs for SGD where ak denotes the distance to the optimal solution after k iterations.”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “Stochastic gradient methods have, however, a major drawback: owing to the sequential nature of the iteration and small batch sizes, there is limited avenue for parallelization.”
Revisiting Distributed Synchronous SGD (2016, background): “(2015a); Reddi et al. (2015); Leblond et al. (2016). Implementations of asynchronous optimization include Xing et al. (2015); Li et al. (2014); Chilimbi et al. (2014). Attempts have also been made in Zinkevich et al. (2010) and Zhang & Jordan (2015) to algorithmically improve synchronous SGD. An”
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, neutral): “Distributed learning is also considered in [16, 49, 57, 78, 87, 89].”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “[45] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola.”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Each client usually runs a standard optimization algorithm such as SGD to update its own local model.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, neutral): “• In the homogeneous setting, if Θ = Θ1:K , and γ = η, then LocalUpdate is equivalent to Local SGD with K local steps.”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, positive): “Re-markably, for instance for local SGD (Zinkevich et al., 2010; Stich, 2019b; Patel & Dieuleveut, 2019) we show improved rates for the convex and strongly-convex case and recover the best known rates for the non-convex case under weaker assumptions than assumed in prior work (highlighted in Table…”
Adaptive Federated Optimization (2020, methodology): “When the client are homogeneous, FEDAVG reduces to local SGD (Zinkevich et al., 2010), which has been analyzed theoretically by many works (Stich, 2019; Yu et al.”
Is Local SGD Better than Minibatch SGD? (2020, background): “(Zinkevich et al., 2010; Zhang et al., 2012; Li et al., 2014; Rosenblatt and Nadler, 2016; Godichon-Baggioni and Saadane, 2017; Jain et al., 2017).”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, methodology): “This motivates us to study the following key question: Can we reduce the number of communication rounds even more, and yet achieve linear speedup? In this paper, we give an affirmative answer to this question by providing a tighter analysis of local SGD via model averaging [43, 38, 51, 23, 45, 49].”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, methodology): “Beyond them, a simple but powerful method is to reduce the communication frequency by allowing more local updates [58, 39, 22, 47, 44], which we focus on in this paper.”
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, neutral): “For identical clients, FedAvg coincides with parallel SGD analyzed by Zinkevich et al. (2010) who proved identical asymptotic convergence.”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, methodology): “…algorithms to reduce the dependency on intensive parameter updates to enable more efﬁcient parallel training (e.g., (Jaggi et al., 2014; Zhang et al., 2013; Zinkevich et al., 2010; Takác et al., 2013; Neiswanger et al., 2014; Shamir et al., 2014; Zhang & Lin, 2015; Shalev-Shwartz & Zhang, 2013)).”
Communication trade-offs for synchronized distributed SGD with large step size (2019, methodology): “Over the past decade, the amount of available data has steadily increased: to adapt SGD to such situations, it has become necessary to distribute the workload between several machines, also referred to as workers (Delalleau and Bengio, 2007; Zinkevich et al., 2010; Recht et al., 2011).”