COLA: Decentralized Linear Learning

By Lie He, An Bian, Martin Jaggi. 2018

140 citations recorded by the source corpus.

Decentralized machine learning is a promising emerging paradigm in view of global challenges of data ownership and privacy. We consider learning of linear classification and regression models, in the setting where the training data is decentralized over many user devices, and the learning algorithm must run on-device, on an arbitrary communication network, without a central coordinator. We propose COLA, a new decentralized training algorithm with strong theoretical guarantees and superior practical performance. Our framework overcomes many limitations of existing methods, and achieves communication efficiency, scalability, elasticity as well as resilience to changes in data and allows for unreliable and heterogeneous participating devices.

Subfields: Artificial Intelligence, Computer Networks and Communications, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “While for the special case of generalized linear models, schemes using the duality structure could enable some of these desired robustness properties [201], for the case of deep learning and SGD this remains an open question.”
Robust Aggregation for Federated Learning (2019, background): “Recent work includes primal-dual frameworks [62, 80] and variants suited to decentralized [37], and asynchronous [52] settings.”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “entralized learning when the worker number is comparably large under a poor network condition (Lian et al., 2017). There are two main types of decentralized learning algorithms: xed network topology (He et al., 2018), and time-varying (Nedic and Olshevsky, 2015; Lian et al., 2018) during training. W”
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, background): “bles at each iteration. Note that, in GADMM, each worker is required to share the primal parameters only per iteration. Finally, it is worth mentioning that a decentralized algorithm was proposed in (He et al., 2018), but that algorithm was studied only for linear learning tasks. For time-varying to”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, background): “There are two main types of decentralized learning algorithms: ﬁxed network topology [He et al., 2018], or time-varying [Lian et al., 2018, Nedić and Olshevsky, 2015] during training.”
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (2019, background): “He et al. (2018) address the more speciﬁc problem class of generalized linear models.”
Distributed Learning over Unreliable Networks (2018, background): “If the underlying network is unreliable (He et al., 2018), the two aggregation steps change as follows.”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Some recent works have investigated decentralized training over heterogeneous data with local updating schemes [18].”
Federated Learning With Quantized Global Model Updates (2020, background): “Several studies address this issue by considering communications with rate limitations, and propose different compression and quantization techniques [2–9], as well as performing local updates to reduce the frequency of communications from the devices to the PS [10, 11].”
Optimal Convergence Rates for Convex Distributed Optimization in Networks (2019, unknown): “Mokhtari and Ribeiro, 2016) to more recent improvements (Qu and Li, 2016; Jiang et al., 2017; He et al., 2018; Pu et al., 2018; Uribe et al., 2018; Koloskova et al., 2019; Xin et al., 2019; Jakoveti´c, 2019).”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, background): “, decentralised FL) [56].”
Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges (2022, background): “DFL,also known as Serverless or Distributed FL, emerged in 2018 to distribute the aggregation of model parameters between the neighboring participants [5].”