Robust and Communication-Efficient Collaborative Learning
By Amirhossein Reisizadeh, Hossein Taheri, Aryan Mokhtari, Hamed Hassani, Ramtin Pedarsani. 2019
98 citations recorded by the source corpus.
We consider a decentralized learning problem, where a set of computing nodes aim at solving a non-convex optimization problem collaboratively. It is well-known that decentralized optimization schemes face two major system bottlenecks: stragglers' delay and communication overhead. In this paper, we tackle these bottlenecks by proposing a novel decentralized and gradient-based optimization algorithm named as QuanTimed-DSGD. Our algorithm stands on two main ideas: (i) we impose a deadline on the local gradient computations of each node at each iteration of the algorithm, and (ii) the nodes exchange quantized versions of their local models. The first idea robustifies to straggling nodes and the second alleviates communication efficiency. The key technical contribution of our work is to prove that with non-vanishing noises for quantization and stochastic gradients, the proposed method exactly converges to the global optimal for convex loss functions, and finds a first-order stationary point in non-convex scenarios. Our numerical evaluations of the QuanTimed-DSGD on training benchmark datasets, MNIST and CIFAR-10, demonstrate speedups of up to 3x in run-time, compared to state-of-the-art decentralized optimization methods.
Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Aryan Mokhtari
- Peter Richtárik
- Virginia Smith
- Martin Jaggi
- Sebastian U. Stich
- Mehrdad Mahdavi
- Jakub Konecný
- Tian Li
In Computer Networks and Communications
- Aryan Mokhtari
- Sebastian U. Stich
- Martin Jaggi
- Hamed Hassani
- Amirhossein Reisizadeh
- Anastasia Koloskova
- Tian Li
- Virginia Smith
In Electrical and Electronic Engineering
- Virginia Smith
- Martin Jaggi
- Tian Li
- Mehrdad Mahdavi
- Farzin Haddadpour
- Aryan Mokhtari
- Mohammad Mahdi Kamani
- Anastasia Koloskova
In Federated & Distributed Learning
- Aryan Mokhtari
- Virginia Smith
- Mehrdad Mahdavi
- Tian Li
- Farzin Haddadpour
- Martin Jaggi
- Ameet Talwalkar
- Anit Kumar Sahu
Citations in the corpus
- FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, methodology): “…and compression in distributed learning has recently gained considerable attention for both master-worker [Alistarh et al., 2017, Bernstein et al., 2018, Seide et al., 2014, Smith et al., 2016] and masterless topologies [Koloskova et al., 2019, Reisizadeh et al., 2019a, Zhang et al., 2018].”
- Decentralized Deep Learning with Arbitrary Communication Compression (2019, background): “(Tang et al., 2018; Wang et al., 2019; Tang et al., 2019; Reisizadeh et al., 2019) that mostly considered training performance (on the train-set).”
- Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Some works propose deadlinebased approaches where all workers compute the local gradients using a variable number of samples within a fixed global cycle, which helps mitigate the impact of stragglers [16], [39].”
- Federated Learning with Compression: Unified Analysis and Sharp Guarantees (2020, background): “Recently a few attempts are made to explore the compression schema in a decentralized setting where each device shares compressed messages with direct neighbors over the underlying communication network [24, 42, 43, 46].”