Trading Redundancy for Communication: Speeding up Distributed SGD for Non-convex Optimization

By Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, Viveck R. Cadambe. 2019

72 citations recorded by the source corpus.

Subfields: Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, background): “[14] Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe.”
Is Local SGD Better than Minibatch SGD? (2020, background): “…additional assumption of a bounded third derivative, but even with this assumption do not improve over mini-batch SGD. Numerous works study local SGD in the non-convex setting (see e.g. Haddadpour et al., 2019b; Stich and Karimireddy, 2019; Wang et al., 2017; Yu et al., 2019; Zhou and Cong, 2018).”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Additionally, [14] shows an interesting trade-off between the amount of data redundancy and the accuracy of local SGD for general non-convex optimization.”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Haddadpour et al. [4] also achieves the linear dependence on I but only requires each node draws samples from its local partitions.”
Adaptive Personalized Federated Learning (2020, positive): “As we theoretically prove and empirically verify in this paper, when the data is distributed IID, we cannot beneﬁt from personalization, and it is similar to the local SGD scenario (Stich, 2018; Haddadpour et al., 2019a,b).”
On the Convergence of Local Descent Methods in Federated Learning (2019, background): “…of previous studies of local SGD with periodic averaging such as [31,33] is based on i.i.d. data distribution at each machine/device, the reference [9] shows a trade-oﬀ between the amount of data redundancy and the accuracy of local SGD for general non-convex optimization for non-i.i.d. data…”
Variance Reduced Local SGD with Lower Communication Complexity (2019, background): “Haddadpour et al. [2019] verify that the use of redundant data can lead to lower communication complexity and hence faster convergence.”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Considering such memory constraints, Haddadpour et al. [91] proposed an approach of distributing shards of data across FL clients to attain the target model swiftly.”
FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data (2020, methodology): “In [14], the data on the local agents are separated into blocks and shared with other agents.”
Communication-Efficient Local Decentralized SGD Methods (2019, background): “3 [4] Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe.”
Federated Learning with Compression: Unified Analysis and Sharp Guarantees (2020, result): “This result is further improved in follow up studies [10,11,20,46,49,56].”