Personalized Federated Learning with Moreau Envelopes
By Canh T. Dinh, Nguyen H. Tran, Tuan Dung Nguyen. 2020
1,438 citations recorded by the source corpus.
Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data. One challenge associated with FL is statistical diversity among clients, which restricts the global model from delivering good performance on each client's task. To address this, we propose an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalized FL. Theoretically, we show that pFedMe's convergence rate is state-of-the-art: achieving quadratic speedup for strongly convex and sublinear speedup of order 2/3 for smooth nonconvex objectives. Experimentally, we verify that pFedMe excels at empirical performance compared with the vanilla FedAvg and Per-FedAvg, a meta-learning based personalized FL algorithm.
Subfields: Artificial Intelligence, Optimization & Training, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Qiang Yang
- Percy Liang
- Mohit Bansal
- Yoshua Bengio
- Bo Du
- Ruslan Salakhutdinov
- Jianfeng Gao
In Optimization & Training
- Qiang Yang
- Dacheng Tao
- Tian Li
- H. Vincent Poor
- Bingsheng He
- Qinbin Li
- Virginia Smith
- Khaled Ben Letaief
In Federated & Distributed Learning
Citations in the corpus
- Salvaging Federated Learning by Local Adaptation (2020, background): “Recent papers on personalizing federated models (Wang et al., 2019; Jiang et al., 2019; Fallah et al., 2020; Dinh et al., 2020) propose various methods to improve the accuracy for individual participants; (Jiang et al., 2019) also connects meta learning with personalization.”
- Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [7, 29, 33, 47, 50, 57, 69, 75, 79]; see also the recent surveys [46, 56].”
- Multi-center federated learning: clients clustering for better personalization (2020, background): “, 2020], or deploys a regularisation term in the objective [Deng et al., 2020; Dinh et al., 2020; Hanzely and Richtárik, 2020].”
- Federated Learning on Non-IID Data Silos: An Experimental Study (2021, background): “There are also other FL studies related to nonIID data setting, such as personalizing the local models for each party [10, 12, 19] and designing robust algorithms against different combinations of local distributions [8, 49, 53], which are out of the scope of this paper.”
- Personalized Federated Learning using Hypernetworks (2021, methodology): “We set K = 3 ,i.e., 60 local steps, for the pFedMe algorithm, as it was reported to work well in the original paper (Dinh et al., 2020).”
- Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, methodology): “Distinguished from prior work that applies weight regularization to local models (Li et al., 2020b; Dinh et al., 2020), FEDGEN can serve as an alternative and compatible solution to address user heterogeneity, which inherently bridges the gap among user models w.”
- Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “The most closely related works are those that regularize personalized models towards their average (Hanzely & Richtárik, 2020; Hanzely et al., 2020; Dinh et al., 2020), which can be seen as a form of classical mean-regularized MTL (Evgeniou & Pontil, 2004).”
- Federated Learning for Open Banking (2021, background): “[10,16] enables participants to pursue their personalized models with different directions, but use a regularization term to limit each personalized model to not far away from the “initial point”, the global model.”
- Personalized Federated Learning with First Order Model Optimization (2020, methodology): “Our method strongly outperforms other methods in non-IID federated learning settings.”
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, methodology): “In [78], the authors designed a personalized FL algorithm using Moreau envelopes as each device’s regularized loss function which can decouple personalized ML model optimization from the global model”
- From distributed machine learning to federated learning: a survey (2021, neutral): “In order to address this problem, the model can be split into local layers and global layers, which has been proposed in adaptive personalized federated learning (APFL) [37], FedPer [6], and pFedMe [38].”
- Towards Personalized Federated Learning (2021, background): “The idea of Per-FedAvg has been extended in [39] to propose a federated meta-learning formulation using Moreau envelopes (pFedMe).”
- FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, methodology): “MOCHA (Smith et al., 2017) is a representative personalized FL paper using regularization based on relationships between tasks. pFedMe (T Dinh et al., 2020) and Ditto (Li et al., 2021) train local models with a regularization based on the divergence between a global model and local models.”
- Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “pFedME [197] utilizes the Moreau envelope function as a regularized loss function, which decouples the personalized model optimization process from global model learning.”