Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

By Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Medhi Bennis, Seong-Lyun Kim. 2018

759 citations recorded by the source corpus.

On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Efficiency & Compression, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Medhi Bennis
  2. Qiang Yang
  3. Dacheng Tao
  4. Jihong Park
  5. H. Vincent Poor
  6. Philip S. Yu
  7. Virginia Smith
  8. Lichao Sun

In Electrical and Electronic Engineering

  1. Medhi Bennis
  2. Merouane Debbah
  3. Jihong Park
  4. H. Vincent Poor
  5. Sumudu Samarakoon
  6. Walid Saad
  7. Mingzhe Chen
  8. Dusist Niyato

In Efficiency & Compression

  1. Medhi Bennis
  2. Sebastian U. Stich
  3. Martin Jaggi
  4. Jihong Park
  5. Tao Lin
  6. Deniz Gündüz
  7. H. Vincent Poor
  8. Shusen Wang

In Federated & Distributed Learning

  1. Medhi Bennis
  2. Virginia Smith
  3. Tian Li
  4. H. Vincent Poor
  5. Qiang Yang
  6. Ameet Talwalkar
  7. Jihong Park
  8. Bingsheng He

Citations in the corpus