Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge

By Chaoyang He, Murali Annavaram, S. Avestimehr. 2020

580 citations recorded by the source corpus.

Subfields: Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Federated & Distributed Learning

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “Federated Averaging (FedAvg) [47], FedOpt (server Adam) [2], FedNova (client optimizer) [3], FedProx [4], FedMA [5], as well as some novel algorithms that have diverse training paradigms and network typologies, including FedGKT (efficient edge training) [73], Decentralized FL [59], Vertical Federate”
Distilled One-Shot Federated Learning (2020, background): “Recently, knowledge distillation is introduced to the field of FL to reduce the computation costs for edge devices [11].”
Knowledge Distillation: A Survey (2020, methodology): “…are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al. 2019; Bashivanetal.2019),interpretabilityofdeepneuralnetworks (Liu et al. 2018b), and federated learning (Bistritz et al. 2020; Lin et al. 2020; Seo et al. 2020; He et al. 2020a).”
Federated Adversarial Debiasing for Fair and Transferable Representations (2021, background): “For example, in [10, 14, 21, 26], the consensus on task knowledge is achieved by distillation.”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “[67] also consider the limited computing resources in the edge devices.”
Towards Personalized Federated Learning (2021, background): “Communication efficiency is evaluated by the number of communication rounds [12,32], the number of parameters [23, 57, 65] and message sizes [58, 90].”
Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning (2022, background): “There exist extensive works on improving the global performance of FL via client selection [4, 7, 15], split learning [11, 38], domain adaptation [26, 33], etc.”
Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, background): “For practical FL applications such as healthcare or finance, sharing entire model parameters may be associated with considerable privacy risks, as discussed in prior work (He et al., 2020).”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “FedGKT [71] periodically transfers the knowledge of small CNNs on edges to the large server-side CNN through knowledge distillation, thereby decreasing the burden of edge training.”
YOLOv1 to v8: Unveiling Each Variant–A Comprehensive Review of YOLO (2024, unknown): “Federated learning introduces a collaborative training paradigm where YOLO models are trained across multiple edge devices [98], enhancing privacy, adaptability, and generalization [99].”
Federated Learning on Non-IID Data Silos: An Experimental Study (2021, background): “It has attracted many research interests [7, 22, 31, 36, 39, 45, 64] and been widely used in practice [3, 20, 30].”
Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “In order to solve this problem, a main stream of subsequent effort leverages knowledge transfer through labeled data [38, 74], shared model [48, 72, 92] or group operation [21, 50].”