Distributed learning with compressed gradient differences

By Konstantin Mishchenko, Eduard A. Gorbunov, Martin Takác, Peter Richtárik. 2019

208 citations recorded by the source corpus.

Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck. For this reason, several methods based on the compression of updates were recently proposed using sparsification or quantization. However, none of the prior methods are able to learn the gradients, which renders them incapable of converging to the true optimum in the batch mode. In this work we propose a new method – DIANA – which resolves this issue via compression of gradient differences. We provide theory in the strongly convex and nonconvex settings that shows improved convergence rates, and use it to obtain the first convergence rate for the previously proposed method TernGrad. Finally, we provide theory to support non-smooth regularizers.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Networks and Communications

Citations in the corpus

Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “We note that in the setting of gradient compression, various methods have also been developed to reduce the noise from gradient compression and the method we develop is similar in spirit to some of them, as [17,22].”
Gradient Descent with Compressed Iterates (2019, methodology): “Further, while there is ample work on non-local methods which communicate compressed gradients [1, 3, 11, 34, 35], including methods which perform variance-reduction to remove the variance introduced by compression [10, 23, 24], to the best of our knowledge there is little work on methods performing”
Natural Compression for Distributed Deep Learning (2019, methodology): “Typically, problem (1) is solved by distributed stochastic gradient descent (SGD) [38], which works as follows: i) given model xk maintained on each node, machine i computes a random vector gi(xk) whose mean is∇fi(xk) (i.e., a stochastic gradient), ii) all stochastic gradients gi(x k) are sent to a ”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, methodology): “The proofs of the claimed bounds can be found in (Mishchenko et al., 2018).”
Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization (2020, methodology): “Several examples of quantization operators can be found in [26].”