Decentralized Deep Learning with Arbitrary Communication Compression

By Anastasia Koloskova, Tao Lin, Sebastian U. Stich, Martin Jaggi. 2019

272 citations recorded by the source corpus.

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD $-$ recently introduced and analyzed for strongly-convex objectives only $-$ converges under arbitrary high compression ratio on general non-convex functions at the rate $O\bigl(1/\sqrt{nT}\bigr)$ where $T$ denotes the number of iterations and $n$ the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over distributed user devices, connected by a social network and (ii) in a datacenter (outperforming all-reduce time-wise).

Subfields: Artificial Intelligence, Computer Networks and Communications, Efficiency & Compression, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Efficiency & Compression

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Translating and generalizing some of the existing compressed communication schemes from the centralized orchestratorfacilitated setting to the fully decentralized setting, without negatively impacting the convergence is an active research direction [243, 335, 380, 242].”
Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs (2020, background): “There has been a signiﬁcant recent progress in this topic (see [AGL + 17,KRSJ19,WXY + 17,SCJ18,AHJ + 18,KLSJ19,SCJ18,BDKD19,SDGD19,SDGD20] and references therein).”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, background): “In the machine learning context, decentralized implementations of stochastic gradient descent have gained a lot of attention recently (Lian et al., 2017; Tang et al., 2018b; Assran et al., 2019; Koloskova et al., 2020), especially for the particular (but not fully decentralized) case of a star-shape”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, background): “Note also that all of our baselines (or base optimizers) leverage a local momentum scheme, following previous works (Assran et al., 2019; Koloskova et al., 2019a).”
Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints (2019, background): “Other distributed training frameworks [18][19][20][21] are facing the same issues.”
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization (2019, methodology): “We compare our algorithm against CHOCO-SGD [KLSJ19], which is the state-of-the-art in compressed decentralized training and provide theoretical justiﬁcation for communication eﬃ-ciency of SPARQ-SGD over CHOCO-SGD to achieve the same target accuracy.”
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning (2019, unknown): “…Q-GADMM that allows each worker to communicate with only two neighbors, these methods pre-assume sufﬁcient network connectivity characterized by the symmetric and doubly stochastic connectivity matrix [49]–[51], column stochastic connectivity matrix [52], and an extended star network topology [54].”
Federated Learning with Compression: Unified Analysis and Sharp Guarantees (2020, background): “Recently a few attempts are made to explore the compression schema in a decentralized setting where each device shares compressed messages with direct neighbors over the underlying communication network [21, 38, 39, 42].”
Decentralized Federated Learning: A Survey and Perspective (2023, background): “In recent studies, DFL frameworks have been developed speciﬁcally for mobile IoT devices, aiming to leverage their computational power and sensor capabilities [192], [193].”
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, background): “…under centralized (Bernstein et al., 2018; Suresh et al., 2017; Sun et al., 2019; Vogels et al., 2019; Alistarh et al., 2017; Horv´ath et al., 2019) and decentralized network topologies (Sriranga et al., 2019; Zhu et al., 2016; Koloskova et al., 2019; Gao and Huang, 2020; Elgabli et al., 2020b).”