Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data

By Jinhyun Ahn, Osvaldo Simeone, Joonhyuk Kang. 2019

129 citations recorded by the source corpus.

Cooperative training methods for distributed machine learning typically assume noiseless and ideal communication channels. This work studies some of the opportunities and challenges arising from the presence of wireless communication links. We specifically consider wireless implementations of Federated Learning (FL) and Federated Distillation (FD), as well as of a novel Hybrid Federated Distillation (HFD) scheme. Both digital implementations based on separate source-channel coding and over-the-air computing implementations based on joint source-channel coding are proposed and evaluated over Gaussian multiple-access channels.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Electrical and Electronic Engineering

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “In terms of training procedures, the training procedures in federated GAN [59, 60] and transfer learning-based FL [61, 62, 63, 64, 65] are significantly different from the vanilla FedAvg algorithm [66].”
Federated Learning for Vehicular Internet of Things: Recent Advances and Open Issues (2020, background): “The implementation issue of FL over wireless channels is discussed in [61].”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “In [126] and [127], FD is implemented in a wireless fading channel, demonstrating comparable accuracy under channel fluctuations and outages with much less payload sizes compared to FL.”
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning (2019, unknown): “Additionally, for large models such as large deep neural networks (DNNs) consisting of millions of model parameters, model outputs can be exchanged via knowledge distillation [14], [15].”
Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence (2019, background): “A work that puts this principle into practice is [47], which proposes a hybrid-federated distillation (HFD) scheme based on separate source-channel coding and over-the-air computing.”
Hierarchical Federated Learning ACROSS Heterogeneous Cellular Networks (2019, background): “Recently, FL with a particular focus on wireless communications has been studied in several papers [1–3, 7–12].”
Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data (2020, methodology): “in [6], [7], [8], [9] and the experiments presented in Section 4.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “Leveraging this to enable distributed learning, the knowledge in FD is constructed the ensemble of different workers’ prediction, each of which is locally averaged per label in a classification task [141], [145] or across neighboring states in RL [146].”
On Analog Gradient Descent Learning Over Multiple Access Fading Channels (2019, background): “Other recent studies considered federated learning with heterogeneous [18], [19] or redundant [20] data, including overthe-air aggregation techniques via MAC with known channel states [19], [20], digital gradient transmissions [21], and privacy over MAC [22].”
Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge Based Framework (2020, background): “Moreover, federated distillation can significantly reduce the communication cost as it exchanges not the model parameters but the model outputs [25].”