Personalized Federated Learning: A Meta-Learning Approach

By Alireza Fallah, Aryan Mokhtari, Asuman E. Ozdaglar. 2020

720 citations recorded by the source corpus.

In Federated Learning, we aim to train models across multiple computing units (users), while users can only communicate with a common central server, without exchanging their data samples. This mechanism exploits the computational power of all users and allows users to obtain a richer model as their models are trained over a larger set of data points. However, this scheme only develops a common output for all the users, and, therefore, it does not adapt the model to each user. This is an important missing feature, especially given the heterogeneity of the underlying data distribution for various users. In this paper, we study a personalized variant of the federated learning in which our goal is to find an initial shared model that current or new users can easily adapt to their local dataset by performing one or a few steps of gradient descent with respect to their own data. This approach keeps all the benefits of the federated learning architecture, and, by structure, leads to a more personalized model for each user. We show this problem can be studied within the Model-Agnostic Meta-Learning (MAML) framework. Inspired by this connection, we study a personalized variant of the well-known Federated Averaging algorithm and evaluate its performance in terms of gradient norm for non-convex loss functions. Further, we characterize how this performance is affected by the closeness of underlying distributions of user data, measured in terms of distribution distances such as Total Variation and 1-Wasserstein metric.

Subfields: Artificial Intelligence, Meta & Continual Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

In Federated & Distributed Learning

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “In terms of network topology, a variety of network typologies such as vertical FL [34, 35, 36, 37, 38, 39, 40], split learning [41, 42], decentralized FL [43, 44, 45, 46], hierarchical FL [47, 48, 49, 50, 51, 52], and meta FL [53, 54, 55] have been proposed.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “Connections between these two areas were noted by Jiang et al. (2019) and have since been explored in many other works (Khodak et al., 2019; Fallah et al., 2020).”
Salvaging Federated Learning by Local Adaptation (2020, background): “Recent papers on personalizing federated models (Wang et al., 2019; Jiang et al., 2019; Fallah et al., 2020; Dinh et al., 2020) propose various methods to improve the accuracy for individual participants; (Jiang et al., 2019) also connects meta learning with personalization.”
Adaptive Personalized Federated Learning (2020, background): “Fallah et al. (2020) borrow ideas from MAML to learn personalized models for each client with convergence guarantees.”
Federated Continual Learning with Weighted Inter-client Transfer (2020, background): “This is an opposite direction from personalized federated learning methods (Fallah et al., 2020; Lange et al., 2020; Deng et al., 2020) which put more emphasis on the performance of local models.”
Federated Adversarial Debiasing for Fair and Transferable Representations (2021, background): “One major challenge of FL comes from the user heterogeneity where users provide statistically different data for training local models [5, 9].”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “Personalized models are typically found in the realms of recommendation systems (Ben-hamdi et al., 2017; Amat et al., 2018; Martinez et al., 2009; Cho et al., 2002) or in federated learning (Mansour et al., 2020; Jiang et al., 2019; Fallah et al., 2020; Shamsian et al., 2021).”
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “Several works aim to make the global model customized to suit the need of individual users, either by treating each client as a task in meta-learning [19, 28, 20, 29] or multi-task learning [30], or by learning both global parameters for all clients and local private parameters for individual client”
Personalized Federated Learning using Hypernetworks (2021, methodology): “While PFL methods like pFedME (Dinh et al., 2020) and Per-FedAvg (Fallah et al., 2020a) To evaluate pFedHN in this setting, we use the CIFAR10 dataset, with a total of 100 clients, of which 90 are used for training and 10 are held out novel clients.”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, methodology): “In Cheng et al. (2021), they compared local-only (zero collaboration), FedAvg (zero personalization) (McMahan et al., 2017), FedAvg+Fine-tuning, Per-FedAvg (Fallah et al., 2020), and pFedMe (T Dinh et al., 2020) in the high-dimensional asymptotic limit by analyzing bias-variance.”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “Also of particular interest are methods for robustness or adaptation to different users [29, 10, 34, 48, 38, 17, 40], a setting commonly referred to as robust or fair federated learning [45, 48, 38].”
Personalized Federated Learning with First Order Model Optimization (2020, methodology): “[7] instead optimize the global model for fast personalization through meta-learning, while [26] train global and local models under regularization with Moreau envelopes.”
Survey of Personalization Techniques for Federated Learning (2020, methodology): “A new formulation of the standard federated learning problem proposed by Fallah et al. [27] incorporates MAML and seeks to ﬁnd a global model which performs well after each user updates it with respect to its own loss function.”