GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning

By Anis Elgabli, Jihong Park, Amrit Singh Bedi, Medhi Bennis, Vaneet Aggarwal. 2019

91 citations recorded by the source corpus.

When the data is distributed across multiple servers, lowering the communication cost between the servers (or workers) while solving the distributed learning problem is an important problem and is the focus of this paper. In particular, we propose a fast, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm, Group Alternating Direction Method of Multipliers (GADMM) is based on the Alternating Direction Method of Multipliers (ADMM) framework. The key novelty in GADMM is that it solves the problem in a decentralized topology where at most half of the workers are competing for the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges to the optimal solution for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets. Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove its convergence under the time-varying network topology of the workers.

Subfields: Computational Mechanics, Computer Networks and Communications, Electrical and Electronic Engineering, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “While multi-agent optimization has a long history in the control community, fully decentralized variants of SGD and other optimization algorithms have recently been considered in machine learning both for improved scalability in datacenters [30] as well as for decentralized networks of devices [110,”
Wireless Network Intelligence at the Edge (2018, methodology): “b) Group ADMM: Without the aid of any central entity, group alternating direction method of multiplier (GADMM) exchanges model weights with neighboring devices and achieves fast training convergence with much less communication rounds [106], [183].”
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning (2019, unknown): “The n -th worker has its model vector θ n ∈ d , and aims to solve the following decentralized learning problem: (1) In [23], GADMM is proposed to solve (1) where the workers are divided into two groups, heads and tails , such that each worker in the head (or tail) group is communicating with two…”
Distilling On-Device Intelligence at the Network Edge (2019, methodology): “GADMM is scalable in a way that it requires each device to communicate only with two neighboring devices, regardless of the total number of federating devices.”
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, methodology): “In (Elgabli et al., 2020b), a stochastic quantization is applied for reducing the communication payload size.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, methodology): “Alternatively, to achieve the spatial link sparsity, one can enforce a sparse network topology by making each worker communicate only with very few neighbors, as exemplified by decentralized gradient descent (GD), dual averaging [87], and GADMM algorithms [64].”
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “Moreover, a communication network topology design is also critical to improve the communication efficiency [151], for which a group alternating direction method of multipliers [152] was proposed to form a connectivity chain by dividing the workers into head and tail workers.”