Ensemble Distillation for Robust Model Fusion in Federated Learning

By Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi. Neural Information Processing Systems, 2020

1,521 citations recorded by the source corpus.

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

Subfields: Artificial Intelligence, Building and Construction, Federated & Distributed Learning

View Paper: https://arxiv.org/abs/2006.07242

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dacheng Tao
  2. Qiang Yang
  3. Bo Du
  4. Guodong Long
  5. Mang Ye
  6. Jing Jiang
  7. H. Vincent Poor
  8. Chengqi Zhang

In Building and Construction

  1. Dacheng Tao
  2. Bingsheng He
  3. Young Geun Kim
  4. Walid Saad
  5. Murali Annavaram
  6. Bo Du
  7. Qiang Yang
  8. Medhi Bennis

In Federated & Distributed Learning

  1. Qiang Yang
  2. Dacheng Tao
  3. Bingsheng He
  4. Qinbin Li
  5. Murali Annavaram
  6. Zeyi Wen
  7. H. Vincent Poor
  8. Medhi Bennis

Citations in the corpus