Gradient Diversity: a Key Ingredient for Scalable Distributed Learning

By Dong Yin, Ashwin Pananjady, Maximilian Lam, Dimitris Papailiopoulos, Kannan Ramchandran, Peter L. Bartlett. 2017

151 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “Under diﬀerent assumptions, Yin et al. (2018) establish a critical batch size and a pathological loss function that together exhibit a transition from a b -fold beneﬁt to no beneﬁt.”
Federated Optimization in Heterogeneous Networks (2018, methodology): “Interestingly, similar assumptions (e.g., Schmidt & Roux, 2013; Vaswani et al., 2019; Yin et al., 2018) have been explored elsewhere but for differing purposes; we provide a discussion of these works in Appendix B. Definition 3 (B-local dissimilarity).”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Furthermore certain real-world data sets, it is well known that larger minibatches also lead to poor generalization and gradient saturation that lead to significant performance gaps between the ideal and practical speed up [12, 22, 41, 20].”
Local SGD Converges Fast and Communicates Little (2018, background): “Recent work in [11, 43] explores various limitations of this approach, as in general it is reported that performance degrades for too large mini-batch sizes [14, 19, 42].”
Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates (2018, background): “Communication eﬃciency has been studied extensively in non-Byzantine distributed settings (McMa-han et al., 2016, Yin et al., 2017).”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Heterogeneity diagnostics Recent works have aimed to quantify statistical heterogeneity through metrics such as local dissimilarity (as defined in the context of federated learning in [25] and used for other purposes in works such as [51]).”
Age-Based Scheduling Policy for Federated Learning in Mobile Edge Networks (2019, background): “This gain can be essentially attributed to that while the MaxPack policy exploits the multi-user diversity [23] by allocating the best subchannels and power to maximize the total number of receivable updates, the ABS strikes a balance between the aggressive channel use and short term fairness and he”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “This is a well-known fact in optimization, and most convergence bounds for stochastic-gradient based methods, including minibatch, involve upper bounds on E[‖ĝ(wk)‖](2)2, in a multiplicative form, for both convex and nonconvex setups [39, 40, 41, 42, 42, 43, 44, 45, 46, 47].”
On the Convergence of Local Descent Methods in Federated Learning (2019, background): “Unlike [34], our modiﬁed notion of gradient diversity is introduced to quantify heterogeneity among local objectives to establish conditions on convergence in non-i.i.d. stetting.”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “Gradient diversity: Yin et al. (2017) construct a scalar measure of the gradient diversity given by ∑ k (cid:107) ∇ f k ( x ) (cid:107) / (cid:107) ∇ f ( x ) (cid:107) , and analyze its effect on the maximum allowed batch-size in the context of distributed optimization.”
On Large-Cohort Training for Federated Learning (2021, background): “…where larger batches can stagnate convergence improvements (Dean et al., 2012; You et al., 2017; Golmant et al., 2018; McCandlish et al., 2018; Yin et al., 2018), and even lead to generalization issues with deep neural networks (Shallue et al., 2019; Ma et al., 2018; Keskar et al., 2017;…”
Robust Federated Learning in a Heterogeneous Environment (2019, background): “Distributed and Federated Learning: Learning with a distributed computing framework has been studied extensively in various settings [12, 13, 14, 15, 16].”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “Definition 1 (Gradient Diversity [19]).”
Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning (2018, background): “Byzantine-robust distributed learning Solving large scale learning problems in distributed systems has received much attention in recent years, where communication eﬃciency and Byzantine robustness are two important topics [58, 41, 70, 10, 15, 20].”