Gradient Descent with Compressed Iterates
By Ahmed Khaled, Peter Richtárik. 2019
26 citations recorded by the source corpus.
We propose and analyze a new type of stochastic first order method: gradient descent with compressed iterates (GDCI). GDCI in each iteration first compresses the current iterate using a lossy randomized compression technique, and subsequently takes a gradient step. This method is a distillation of a key ingredient in the current practice of federated learning, where a model needs to be compressed by a mobile device before it is sent back to a server for aggregation. Our analysis provides a step towards closing the gap between the theory and practice of federated learning, and opens the possibility for many extensions.
Subfields: Artificial Intelligence, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Richtárik
- Martin Jaggi
- Jakub Konecný
- Qiang Yang
- Sebastian U. Stich
- Peter Kairouz
- Ananda Theertha Suresh
- Jianyu Wang
In Optimization & Training
- Peter Richtárik
- Jakub Konecný
- Martin Jaggi
- Ahmed Khaled
- Sebastian U. Stich
- Jianyu Wang
- Oluwasanmi Koyejo
- Gauri Joshi
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “The impact of (b) on convergence in general has not been studied until very recently; a limited analysis is presented in [231].”
- Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “In this paper we improve the results of [18] and generalize them in two ways.”
- From Local SGD to Local Fixed Point Methods for Federated Learning (2020, methodology): “We will also investigate compression [10, 11] of the communicated variables, with or without variance reduction, in combination with locality.”