Adaptive Personalized Federated Learning

By Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi. 2020

710 citations recorded by the source corpus.

Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. Information theoretically, we prove that the mixture of local and global models can reduce the generalization error. We also propose a communication-reduced bilevel optimization method, which reduces the communication rounds to $O(\sqrt{T})$ and show that under strong convexity and smoothness assumptions, the proposed algorithm can achieve a convergence rate of $O(1/T)$ with some residual error. The residual error is related to the gradient diversity among local models, and the gap between optimal local and global models. The extensive experiments demonstrate the effectiveness of our personalization, as well as the correctness of our theory.

Subfields: Artificial Intelligence, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

Citations in the corpus

Personalized Federated Learning with Moreau Envelopes (2020, methodology): “, ‖∇fi(w)‖ ≤ G,∀i, which was used in several related works [6,8].”
Federated Continual Learning with Weighted Inter-client Transfer (2020, background): “This is an opposite direction from personalized federated learning methods (Fallah et al., 2020; Lange et al., 2020; Deng et al., 2020) which put more emphasis on the performance of local models.”
Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “In particular, we consider objectives that (i) regularize with the average (L2SGD (Hanzely & Richtárik, 2020)) or the learnt device relationship matrix (MOCHA (Smith et al., 2017)), (ii) encourage closeness to the global model in terms of some specific function behavior (EWC (Kirkpatrick et al., 20”
From distributed machine learning to federated learning: a survey (2021, neutral): “However, it is difficult to choose a dataset and its partition among clients to measure the personalization brought by APFL or FedPer, so as to prove the improvement compared with FedAvg.”
Personalized Cross-Silo Federated Learning on Non-IID Data (2020, methodology): “In this section, we evaluate the performance of FedAMP and HeurFedAMP and compare them with the state-of-the-art personalized federated learning algorithms, including SCAFFOLD (Karimireddy et al. 2019), APFL (Deng, Kamani, and Mahdavi 2020), FedAvg-FT and FedProx-FT (Wang et al. 2019).”
Personalized Federated Learning: An Attentive Collaboration Approach (2020, methodology): “2, most existing methods [9, 10, 11, 12, 13, 14, 15, 16] that produce personalized models by locally customizing a global model cannot effectively identify the underlying collaboration relationships between clients.”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “, 2021), showing that FedAvg beats recent personalized algorithms (Smith et al., 2017; Fallah et al., 2020; Liang et al., 2020; Hanzely & Richtárik, 2020; Deng et al., 2020; Li et al., 2021; Arivazhagan et al., 2019) in many cases.”
Federated Learning for Open Banking (2021, background): “[9] proposes a mixture model of global and local models in a federated setting.”
Lower Bounds and Optimal Algorithms for Personalized Federated Learning (2020, background): “However, the requirements of several FL applications [53, 25, 10] question the appropriateness of the objective (1) as a modeling paradigm for FL.”
Federated Learning on Non-IID Data: A Survey (2021, background): “In general, there are several major types of personalization methods, including conducting local fine-tuning (personalization via regularization & interpolation, or meta learning), including a personalization layer, multi-task learning, and knowledge distillation [25].”
Personalized Federated Learning using Hypernetworks (2021, background): “Another approach to PFL is model mixing where the clients learn a mixture of the global and local models (Deng et al., 2020; Arivazhagan et al., 2019).”
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach (2020, background): “A similar idea has been studied in [40], where the authors propose an adaptive federated learning algorithm that learns a mixture of local and global models as the personalized model.”
FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, background): “Some recent studies (Arivazhagan et al. 2019; Liang et al. 2020; Deng, Kamani, and Mahdavi 2020) train personalized models to leverage both globally shared information and the personalized part (Tan et al. 2021; Jiang, Ji, and Long 2020).”