Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
By Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Medhi Bennis, Seong-Lyun Kim. 2018
759 citations recorded by the source corpus.
On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Efficiency & Compression, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Medhi Bennis
- Qiang Yang
- Dacheng Tao
- Jihong Park
- H. Vincent Poor
- Philip S. Yu
- Virginia Smith
- Lichao Sun
In Electrical and Electronic Engineering
- Medhi Bennis
- Merouane Debbah
- Jihong Park
- H. Vincent Poor
- Sumudu Samarakoon
- Walid Saad
- Mingzhe Chen
- Dusist Niyato
In Efficiency & Compression
- Medhi Bennis
- Sebastian U. Stich
- Martin Jaggi
- Jihong Park
- Tao Lin
- Deniz Gündüz
- H. Vincent Poor
- Shusen Wang
In Federated & Distributed Learning
- Medhi Bennis
- Virginia Smith
- Tian Li
- H. Vincent Poor
- Qiang Yang
- Ameet Talwalkar
- Jihong Park
- Bingsheng He
Citations in the corpus
- Federated Optimization in Heterogeneous Networks (2018, methodology): “However, these methods may be unrealistic: in addition to imposing burdens on network bandwidth, sending local data to the server (Jeong et al., 2018) violates the key privacy assumption of federated learning, and sending globally-shared proxy data to all devices (Zhao et al.”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, positive): “FD (Jeong et al., 2018) utilizes distillation to reduce FL communication costs.”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Besides, many empirical studies are investigating the effect of system heterogeneity [31, 55], the power of scalability [22], strategies of preserving privacy [6, 27], and saving communication [9].”
- GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, background): “…decreasing communication payload sizes is one popular solution, which is enabled by gradient quantization (Suresh et al., 2017), model parameter quantization (Zhu et al., 2016; Sriranga et al., 2019), and model output exchange for large-sized models via knowledge distillation (Jeong et al., 2018).”
- Wireless Network Intelligence at the Edge (2018, methodology): “This method and its scalability are demonstrated by federated augmentation (FAug) in our previous study [166].”
- Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [20, 21, 22, 23], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
- Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “Co-distillation approach is a method to use distillation on private training data with other parties [7], [28], however, without defending against data poisoning and inference attacks [48], [56], [22].”
- Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records (2019, background): “Moreover, unlike other optimization algorithms for federated learning on non-IID data [26-28] that required a fraction of all data to be shared across the clients, CBFL obviated any degrees of patient data transmission, thereby keeping privacy intact.”
- Federated Learning for Open Banking (2021, background): “In [18], a new federated distillation (FD) is proposed to improve communication efficiency for an on-device machine learning framework.”
- No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “The common practices are to share a public dataset [6], synthesized data [16, 17] or a condensed version of the training samples [18] to supplement training on the clients or on the server.”
- Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data (2020, methodology): “Being different from [29], as in the benchmark frameworks [4], [6], this study considers horizontal partitioned data and addresses the communication costs for uploading model parameters or gradients.”
- Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, methodology): “Periodic and dynamic aggregation approaches [22], [25], [58] have been proposed in an attempt to decrease the communication overhead and avoid these bottlenecks.”
- Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning (2021, background): “…do Broadcasting x t +1 to workers. end for For non-i.i.d. datasets, many works (Sattler et al., 2019; Zhao et al., 2018; Li et al., 2018; Wang et al., 2019a; Karimireddy et al., 2019; Huang et al., 2018; Jeong et al., 2018) heuristically demonstrated the performance of FedAvg and its variants.”
- Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, methodology): “Using an online version of KD can reduce such kind of communication cost [194].”
- Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, methodology): “FEDDISTILL (Federated Distillation) is proposed by (Seo et al., 2020) which extracts from user models the statistics of the logit-vector outputs, and shares this meta-data to users for KD.”
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “While FD was proposed in [124], its effectiveness is not limited to simple classification tasks under a perfectly controlled”