Federated Learning With Quantized Global Model Updates

By Mohammad Mohammadi Amiri, Deniz Gündüz, Sanjeev R. Kulkarni, H. Vincent Poor. 2020

160 citations recorded by the source corpus.

We study federated learning (FL), which enables mobile devices to utilize their local datasets to collaboratively train a global model with the help of a central server, while keeping data localized. At each iteration, the server broadcasts the current global model to the devices for local training, and aggregates the local model updates from the devices to update the global model. Previous work on the communication efficiency of FL has mainly focused on the aggregation of model updates from the devices, assuming perfect broadcasting of the global model. In this paper, we instead consider broadcasting a compressed version of the global model. This is to further reduce the communication cost of FL, which can be particularly limited when the global model is to be transmitted over a wireless medium. We introduce a lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted. We analyze the convergence behavior of the proposed LFL algorithm assuming the availability of accurate local model updates at the server. Numerical experiments show that the quantization of the global model can actually improve the performance for non-iid data distributions. This observation is corroborated with analytical convergence results.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “To tackle the system constraints challenge, researchers apply compression, sparsiﬁcation, or quantization techniques to reduce the communication overheads and computation costs during the training process [6, 7, 8, 9, 10, 11, 12].”
Server Averaging for Federated Learning (2021, methodology): “Then, periodic averaging for both server and client models followed up quickly [1].”
Soft-Label Dataset Distillation and Text Dataset Distillation (2019, background): “There has also been an increasing amount of literature discussing using the original DD algorithm in the context of federated learning [30]–[33].”
FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (2021, methodology): “Building upon gradient quantisation methods [47, 26, 56, 28], [2] proposed using quantisation in the model sharing and aggregation steps.”