Personalized Federated Learning using Hypernetworks

By Aviv Shamsian, Aviv Navon, Ethan Fetaya, Gal Chechik. International Conference on Machine Learning, 2021

473 citations recorded by the source corpus.

Personalized federated learning is tasked with training machine learning models for multiple clients, each with its own data distribution. The goal is to train personalized models in a collaborative way while accounting for data disparities across clients and reducing communication costs. We propose a novel approach to this problem using hypernetworks, termed pFedHN for personalized Federated HyperNetworks. In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients, while maintaining the capacity to generate unique and diverse personal models. Furthermore, since hypernetwork parameters are never transmitted, this approach decouples the communication cost from the trainable model size. We test pFedHN empirically in several personalized federated learning challenges and find that it outperforms previous methods. Finally, since hypernetworks share information across clients we show that pFedHN can generalize better to new clients whose distributions differ from any client observed during training.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/148de9bf06862825290d6801728758bc5c6aa72b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, methodology): “Experiments For our experiments, we use CIFAR100 [32], a relatively large-scale dataset compared to those commonly used to benchmark federated learning methods [49, 50].”
Meta-learning without data via Wasserstein distributionally-robust model fusion (2022, background): “Many works are in the context of federated learning McMahan et al. [2017], Shamsian et al. [2021].”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “Many researchers have recently focused on multiple personalized models by leveraging data heterogeneity across devices as a blessing in disguise (Chen et al., 2018; Dinh et al., 2021; Zhang et al., 2020; Fallah et al., 2020; Shamsian et al., 2021; Smith et al., 2017).”
FedALA: Adaptive Local Aggregation for Personalized Federated Learning (2022, methodology): “The first one is the pathological heterogeneous setting (McMahan et al. 2017; Shamsian et al. 2021), where we sample 2/2/10 classes for MNIST/Cifar10/Cifar100 from a total of 10/10/100 classes for each client, with disjoint data samples.”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “Personalized models are typically found in the realms of recommendation systems (Ben-hamdi et al., 2017; Amat et al., 2018; Martinez et al., 2009; Cho et al., 2002) or in federated learning (Mansour et al., 2020; Jiang et al., 2019; Fallah et al., 2020; Shamsian et al., 2021).”
Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, unknown): “[30] proposes to use a central hypernetwork to generate personalized models for clients.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “Successful results of HyperNetwork are also reported in image generation using generative adversarial networks [1, 10] and other machine learning tasks [51].”