Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data

By Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, Koji Yamamoto. IEEE Transactions on Mobile Computing, 2020

379 citations recorded by the source corpus.

This study develops a federated learning (FL) framework overcoming largely incremental communication costs due to model sizes in typical frameworks without compromising model performance. To this end, based on the idea of leveraging an unlabeled open dataset, we propose a distillation-based semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models among mobile devices, instead of model parameter exchange employed by the typical frameworks. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size. The exchanged model outputs are used to label each sample of the open dataset, which creates an additionally labeled dataset. Based on the new dataset, local models are further trained, and model performance is enhanced owing to the data augmentation effect. We further highlight that in DS-FL, the heterogeneity of the devices’ dataset leads to ambiguous of each data sample and lowing of the training convergence. To prevent this, we propose entropy reduction averaging, where the aggregated model outputs are intentionally sharpened. Moreover, extensive experiments show that DS-FL reduces communication costs up to 99 percent relative to those of the FL benchmark while achieving similar or higher classification accuracy.

Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/10475b4d2467cfa15bb0fd149b34b295928002e4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

Ming Ding

Citations in the corpus

Federated Learning for Internet of Things: A Comprehensive Survey (2021, positive): “In terms of communication cost in ondevice FL training, the authors in [207] propose a light-weight model training strategy, by using a concept of model output exchange instead of model parameter exchange.”
Continual Federated Learning Based on Knowledge Distillation (2022, background): “There are also several works leveraging additional datasets constructed from public dataset [Li and Wang, 2019; Lin et al. , 2020] or original local data [Itahara et al. , 2020] to aid distillation.”