SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization
By Navjot Singh, Deepesh Data, Jemin George, Suhas N. Diggavi. 2019
24 citations recorded by the source corpus.
In this paper, we propose and analyze SPARQ-SGD, which is an event-triggered and compressed algorithm for decentralized training of large-scale machine learning models. Each node can locally compute a condition (event) which triggers a communication where quantized and sparsified local model parameters are sent. In SPARQ-SGD each node takes at least a fixed number ($H$) of local gradient steps and then checks if the model parameters have significantly changed compared to its last update; it communicates further compressed model parameters only when there is a significant change, as specified by a (design) criterion. We prove that the SPARQ-SGD converges as $O(\frac{1}{nT})$ and $O(\frac{1}{\sqrt{nT}})$ in the strongly-convex and non-convex settings, respectively, demonstrating that such aggressive compression, including event-triggered communication, model sparsification and quantization does not affect the overall convergence rate as compared to uncompressed decentralized training; thereby theoretically yielding communication efficiency for "free". We evaluate SPARQ-SGD over real datasets to demonstrate significant amount of savings in communication over the state-of-the-art.
Subfields: Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Medhi Bennis
- Anis Elgabli
- Jihong Park
- Chaouki Ben Issaid
- Merouane Debbah
- Sumudu Samarakoon
- Seong-Lyun Kim
- H. Vincent Poor
In Computational Mechanics
Citations in the corpus
- Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning (2019, unknown): “The number of communication links can be reduced by censoring the links of the workers with small model updates [53].”
- Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, methodology): “To this end, link censoring for negligible model updates is applied under centralized (Chen et al., 2018; Sun et al., 2019) and decentralized network topologies (Singh et al., 2019b; Elgabli et al., 2020c).”