Improving Federated Learning Personalization via Model Agnostic Meta Learning

By Yihan Jiang, Jakub Konecný, Keith Rush, Sreeram Kannan. 2019

688 citations recorded by the source corpus.

Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL, MAML, and broader ML research.

Subfields: Artificial Intelligence, Meta & Continual Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “• The observed gap between the global and personalized acccuracy [217] creates a good argument that personalization should be of central importance to FL.”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “In terms of network topology, a variety of network typologies such as vertical FL [34, 35, 36, 37, 38, 39, 40], split learning [41, 42], decentralized FL [43, 44, 45, 46], hierarchical FL [47, 48, 49, 50, 51, 52], and meta FL [53, 54, 55] have been proposed.”
Personalized Federated Learning with Moreau Envelopes (2020, background): “[36] discovered that FedAvg can be interpreted as meta-learning and proposed combining FedAvg with Reptile [33] for FL personalization.”
Three Approaches for Personalization with Applications to Federated Learning (2020, methodology): “One can use different optimizers as proposed by Jiang et al. (2019).”
Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “Jiang et al. (2019) proposed a personalized FedAvg algorithm in which the classic FedAvg is ﬁrst deployed, and then they switch to Reptile, a meta-learning al-gorithm proposed in (Nichol et al., 2018), and ﬁnally run local updates to achieve personalization.”
LEAF: A Benchmark for Federated Settings (2018, background): “These particular pipelines shed light on how different modeling approaches may be more or less appropriate for different federated datasets [10, 12].”
Adaptive Personalized Federated Learning (2020, background): “Therefore, in the presence of statistical device heterogeneity, the global model might not generalize well on the local data of each client individually (Jiang et al., 2019).”
LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets (2020, background): “Existing works achieve personalization via meta-learning [4–6], multi-task learning [7, 8], transfer learning [9, 10], etc.”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, methodology): “We first investigate personalization of a single global model using the FedAvg algorithm, following [42, 23], to connect a single global model to multiple personalized models.”
An Efficient Framework for Clustered Federated Learning (2020, background): “This problem has recently received much attention [4]–[6].”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “[81] interpret FedAvg in the light of existing MAML algorithms.”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “[94] point out that the MAML setting is consistent with the personalized objectives of heterogeneous federated learning.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “Personalization In recent years, personalization has become a prominent factor in various fields within Machine Learning such as recommendation systems [2], language models [11], and Federated Learning [28].”
Towards Personalized Federated Learning (2021, methodology): “Method Quantity Skew Feature Distribution Skew Label Distribution Skew Label Preference Skew Data Augmentation [23] [24] [21]–[23] Client Selection [28] [27]–[28] [25]–[28] Regularization [29] [29] [12], [29]–[32] Meta-Learning [39] [36], [40] [37], [39] Transfer Learning [42]–[44] [42] Parameter De”
Survey of Personalization Techniques for Federated Learning (2020, methodology): “…approaches for federated learning treat development of the global model and its personalization as two distinct activities, Jiang et al. [15] propose a modiﬁcation to the Federated Averaging algorithm that allows both to be addressed simultaneously resulting in better personalized…”
HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients (2020, methodology): “…for non-IID data (Liang et al., 2020), e.g., by integrating FL with other frameworks such as assisted learning (Xian et al., 2020), metalearning (Jiang et al., 2019; Khodak et al., 2019), multi-task learning (Smith et al., 2017), transfer learning (Wang et al., 2019; Mansour et al., 2020),…”