Nested Distributed Gradient Methods with Adaptive Quantized Communication
By Albert S. Berahas, Charikleia Iakovidou, Ermin Wei. 2019
14 citations recorded by the source corpus.
In this paper, we consider minimizing a sum of local convex objective functions in a distributed setting, where communication can be costly. We propose and analyze a class of nested distributed gradient methods with adaptive quantized communication (NEAR-DGD+Q). We show the effect of performing multiple quantized communication steps on the rate of convergence and on the size of the neighborhood of convergence, and prove R-Linear convergence to the exact solution with increasing number of consensus steps and adaptive quantization. We test the performance of the method, as well as some practical variants, on quadratic functions, and show the effects of multiple quantized communication steps in terms of iterations/gradient evaluations, communication and cost.
Subfields: Computer Networks and Communications
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Sebastian U. Stich
- Aryan Mokhtari
- Peter Richtárik
- Martin Jaggi
- Ahmed Khaled
- Amirhossein Reisizadeh
- Hamed Hassani
- Anastasia Koloskova
Citations in the corpus
- Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “Exact methods usually exchange compressed gradients or compressed iterates [6, 14,19,21,26,33].”
- Robust and Communication-Efficient Collaborative Learning (2019, methodology): “…decentralized optimization method with quantized messages was given in (Reisizadeh et al., 2018; Zhang et al., 2018), and more recently, new techniques have been developed in this context for convex problems (Doan et al., 2018; Koloskova et al., 2019; Berahas et al., 2019; Lee et al., 2018a,b).”
- Decentralized Deep Learning with Arbitrary Communication Compression (2019, background): “Another line of work proposed adaptive schemes (with increasing compression accuracy) that converge at the expense of higher communication cost (Carli et al., 2010a; Doan et al., 2018; Berahas et al., 2019).”