Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

By Anis Elgabli, Jihong Park, Amrit Singh Bedi, Medhi Bennis, Vaneet Aggarwal. 2019

74 citations recorded by the source corpus.

In this article, we propose a communication-efficient decentralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). To reduce the number of communication links, every worker in Q-GADMM communicates only with two neighbors, while updating its model via the group alternating direction method of multipliers (GADMM). Moreover, each worker transmits the quantized difference between its current model and its previously quantized model, thereby decreasing the communication payload size. However, due to the lack of centralized entity in decentralized ML, the spatial sparsity and payload compression may incur error propagation, hindering model training convergence. To overcome this, we develop a novel stochastic quantization method to adaptively adjust model quantization levels and their probabilities, while proving the convergence of Q-GADMM for convex objective functions. Furthermore, to demonstrate the feasibility of Q-GADMM for non-convex and stochastic problems, we propose quantized stochastic GADMM (Q-SGADMM) that incorporates deep neural network architectures and stochastic sampling. Simulation results corroborate that Q-GADMM significantly outperforms GADMM in terms of communication efficiency while achieving the same accuracy and convergence speed for a linear regression task. Similarly, for an image classification task using DNN, Q-SGADMM achieves significantly less total communication cost with identical accuracy and convergence speed compared to its counterpart without quantization, i.e., stochastic GADMM (SGADMM).

Subfields: Artificial Intelligence, Computational Mechanics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Electrical and Electronic Engineering

Citations in the corpus

Secure Aggregation with Heterogeneous Quantization in Federated Learning (2020, unknown): “Our work is different from [21], since our objective is to provide schemes that not only allow for heterogeneous quantization, but also guarantee the privacy of the users’ models by doing secure model aggregation.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, methodology): “Utilizing GADMM that exploits sparse connectivity (see Section IV-A), Q-GADMM allows each worker to share a quantized version of its model with neighbors [97].”