Distributed Learning with Sparse Communications by Identification
By Dmitry Grishchenko, Franck Iutzeler, Jérôme Malick, Massih-Reza Amini. 2018
8 citations recorded by the source corpus.
In distributed optimization for large-scale learning, a major performance limitation comes from the communications between the different entities. When computations are performed by workers on local data while a coordinator machine coordinates their updates to minimize a global loss, we present an asynchronous optimization algorithm that efficiently reduces the communications between the coordinator and workers. This reduction comes from a random sparsification of the local updates. We show that this algorithm converges linearly in the strongly convex case and also identifies optimal strongly sparse solutions. We further exploit this identification to propose an automatic dimension reduction, aptly sparsifying all exchanges between coordinator and workers.
Subfields: Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Richtárik
- Konstantin Mishchenko
- Dmitry Kovalev
- Samuel Horváth
- Sebastian U. Stich
- Nicolas Loizou
- Ahmed Khaled
- Filip Hanzely
In Computational Mechanics
- Peter Richtárik
- Samuel Horváth
- Dmitry Kovalev
- Sebastian U. Stich
- Konstantin Mishchenko
- Ahmed Khaled
- Nicolas Loizou
- Martin Jaggi
Citations in the corpus
- Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “A direction orthogonal to these approaches tries to reduce the size of the messages—typically gradient vectors—that are exchanged between the nodes (Seide et al., 2014; Strom, 2015; Alistarh et al., 2017; Wen et al., 2017; Grishchenko et al., 2018).”