Federated Knowledge Distillation

By Hyowoon Seo, Jihong Park, Seungeun Oh, Medhi Bennis, Seong-Lyun Kim. 2020

109 citations recorded by the source corpus.

Distributed learning frameworks often rely on exchanging model parameters across workers, instead of revealing their raw data. A prime example is federated learning that exchanges the gradients or weights of each neural network model. Under limited communication resources, however, such a method becomes extremely costly particularly for modern deep neural networks having a huge number of model parameters. In this regard, federated distillation (FD) is a compelling distributed learning solution that only exchanges the model outputs whose dimensions are commonly much smaller than the model sizes (e.g., 10 labels in the MNIST dataset). The goal of this chapter is to provide a deep understanding of FD while demonstrating its communication efficiency and applicability to a variety of tasks. To this end, towards demystifying the operational principle of FD, the first part of this chapter provides a novel asymptotic analysis for two foundational algorithms of FD, namely knowledge distillation (KD) and co-distillation (CD), by exploiting the theory of neural tangent kernel (NTK). Next, the second part elaborates on a baseline implementation of FD for a classification task, and illustrates its performance in terms of accuracy and communication efficiency compared to FL. Lastly, to demonstrate the applicability of FD to various distributed learning tasks and environments, the third part presents two selected applications, namely FD over asymmetric uplink-and-downlink wireless channels and FD for reinforcement learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “…are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al. 2019; Bashivanetal.2019),interpretabilityofdeepneuralnetworks (Liu et al. 2018b), and federated learning (Bistritz et al. 2020; Lin et al. 2020; Seo et al. 2020; He et al. 2020a).”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “For more details on the fundamentals of FD, readers are encouraged to check [70].”
Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, methodology): “FEDDISTILL (Federated Distillation) is proposed by (Seo et al., 2020) which extracts from user models the statistics of the logit-vector outputs, and shares this meta-data to users for KD.”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “For a comprehensive survey on the topic, please refer to [107], [108]”
Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges (2022, background): “A decentralized version of this approach has been proposed in [114], where instead of having a pre-trained model to imitate it, each model considers the aggregated knowledge of the remaining participants’ models.”