Personalized Federated Learning with First Order Model Optimization

By Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, José Manuel Álvarez. International Conference on Learning Representations, 2020

434 citations recorded by the source corpus.

While federated learning traditionally aims to train a single global model across decentralized local datasets, one model may not always be ideal for all participating clients. Here we propose an alternative, where each client only federates with other relevant clients to obtain a stronger model per client-specific objectives. To achieve this personalization, rather than computing a single model average with constant weights for the entire federation as in traditional FL, we efficiently calculate optimal weighted model combinations for each client, based on figuring out how much a client can benefit from another's model. We do not assume knowledge of any underlying data distributions or client similarities, and allow each client to optimize for arbitrary target distributions of interest, enabling greater flexibility for personalization. We evaluate and characterize our method on a variety of federated settings, datasets, and degrees of local data heterogeneity. Our method outperforms existing alternatives, while also enabling new features for personalized FL such as transfer outside of local data distributions.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/30dba214afa23aa38340d9035ebfdbd77a135411

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Du
2. Mang Ye
3. Dacheng Tao
4. Bingsheng He
5. Qinbin Li
6. Tianyi Zhou
7. Guodong Long
8. Jing Jiang

In Management Science and Operations Research

1. Tian Li
2. Dawn Song
3. Dacheng Tao
4. Bingsheng He
5. Virginia Smith
6. Haibing Guan
7. Bo Du
8. Aurélien Bellet

Citations in the corpus

• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, background): “[31] enables a more flexible personalization by adaptively weighted aggregation.”
• FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “Many researchers have recently focused on multiple personalized models by leveraging data heterogeneity across devices as a blessing in disguise (Chen et al., 2018; Dinh et al., 2021; Zhang et al., 2020; Fallah et al., 2020; Shamsian et al., 2021; Smith et al., 2017).”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “[246] highlight that the clients should consider the suitability of other models to their goals when downloading personalized weighted combinations, and thus devised a personalized federated learning framework named FedFomo.”
• Model-Contrastive Federated Learning (2021, background): “Another research direction is personalized federated learning [8, 7, 10, 47, 17], which tries to learn personalized local models for each party.”
• Ditto: Fair and Robust Federated Learning Through Personalization (2020, methodology): “…meta-learning (Jiang et al., 2019; Chen et al., 2018; Khodak et al., 2019; Fallah et al., 2020; Li et al., 2020a; Singhal et al., 2021), and other forms of MTL, such as hard model parameter sharing (Agarwal et al., 2020; Liang et al., 2020) or the weighted combination method in Zhang et al. (2021).”
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction (2022, background): “[29] dynamic calculates the optimally weighted combination of clients’ local model by figuring out how much a client can benefit from the global model.”
• Federated Multi-Task Learning under a Mixture of Distributions (2021, methodology): “More similar to our approach, FedFOMO [68] lets each client interpolate other clients’ local models with opportune weights learned during training.”
• Personalized Federated Learning using Hypernetworks (2021, methodology): “The two simple base-lines, local and FedAvg, that do not use personalized federated learning perform quite poorly on most tasks 2 , showing the importance of personalized federated learning. pFedHN achieves large improvements of 2%-10% over all competing approaches.”
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning (2022, methodology): “FedFomo (Zhang et al. 2020) focuses on aggregating other client models locally for local initialization in each iteration and approximates the client-specific weights for aggregation using the local models from other clients.”