Distilled One-Shot Federated Learning

By Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, Dapeng Oliver Wu. 2020

192 citations recorded by the source corpus.

Current federated learning algorithms take tens of communication rounds transmitting unwieldy model weights under ideal circumstances and hundreds when data is poorly distributed. Inspired by recent work on dataset distillation and distributed one-shot learning, we propose Distilled One-Shot Federated Learning, which reduces the number of communication rounds required to train a performant model to only one. Each client distills their private dataset and sends the synthetic data (e.g. images or sentences) to the server. The distilled data look like noise and become useless after model fitting. We empirically show that, in only one round of communication, our method can achieve 96% test accuracy on federated MNIST with LeNet (centralized 99%), 81% on federated IMDB with a customized CNN (centralized 86%), and 84% on federated TREC-6 with a Bi-LSTM (centralized 89%). Using only a few rounds, DOSFL can match the centralized baseline on all three tasks. By evading the need for model-wise updates (i.e., weights, gradients, loss, etc.), the total communication cost of DOSFL is reduced by over an order of magnitude. We believe that DOSFL represents a new direction orthogonal to previous work, towards weight-less and gradient-less federated learning.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Federated & Distributed Learning

Citations in the corpus

Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, methodology): “We use KD [19] in a different manner from existing and concurrent works [38, 39, 40, 41, 42, 43, 44, 45].”
Federated Learning: Opportunities and Challenges (2021, background): “In order to overcome communication overheads of sending bulky gradients, [41] proposes a distilled oneshot federated learning, where each device distills their data and send the fabricated data to the central server.”
Dataset Distillation by Matching Training Trajectories (2022, background): “Dataset distillation has enabled various applications including continual learning [44, 45, 47], efﬁcient neural architecture search [45, 47], federated learning [11, 37, 50], and privacy-preserving ML [22, 37] for images, text, and medical imaging data.”