Three Approaches for Personalization with Applications to Federated Learning

By Yishay Mansour, Mehryar Mohri, Jae Ro, Ananda Theertha Suresh. 2020

666 citations recorded by the source corpus.

The standard objective in machine learning is to train a single model for all users. However, in many learning scenarios, such as cloud computing and federated learning, it is possible to learn a personalized model per user. In this work, we present a systematic learning-theoretic study of personalization. We propose and analyze three approaches: user clustering, data interpolation, and model interpolation. For all three approaches, we provide learning-theoretic guarantees and efficient algorithms for which we also demonstrate the performance empirically. All of our algorithms are model-agnostic and work for any hypothesis class.

Subfields: Artificial Intelligence, Information Systems, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Federated & Distributed Learning

Citations in the corpus

Personalized Federated Learning with Moreau Envelopes (2020, neutral): “[27] introduced three personalization approaches to: user clustering, data interpolation, and model interpolation.”
Adaptive Personalized Federated Learning (2020, background): “Perhaps, the closest approach for personalization to our proposal is introduced by Mansour et al. (2020).”
Federated Learning with Compression: Unified Analysis and Sharp Guarantees (2020, background): “Also, the personalization of local models for a better generalization in a heterogeneous setting is of great importance from both theoretical and practical point of view [7,8, 27,32,43].”
Towards Personalized Federated Learning (2021, methodology): “Multi-Task Learning [59]–[62] [59], [61]–[62] [61]–[62] [59, 61] - [59, 61] - Model Interpolation [17], [64]–[65] [17], [63]–[65] [63]–[65] [65] [65] [65] - - Clustering [66]–[70], [72] [66]–[70], [72] [66]–[67], [69]–[70], [72] - - -”
Multi-center federated learning: clients clustering for better personalization (2020, methodology): “To solve the problem caused by non-IID data in a federated setting [5, 50] proposed clustered FL (FedCluster) by integrating FL and bi-partitioning-based clustering into an overall framework, and [14, 42] proposed a hypothesis-based federated clustering that assigns the cluster by considering the lo”
Federated Learning for Open Banking (2021, methodology): “[25] discusses two approaches, namely Data Interpolation and Model Interpolation, to learn a personalized federated learning model by weighting two components between local and global servers in terms of data distributions or models respectively.”
HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients (2020, background): “, 2017), transfer learning (Wang et al., 2019; Mansour et al., 2020), knowledge distillation (Li & Wang, 2019) and lottery ticket hypothesis (Li et al.”
Federated Learning: A signal processing perspective (2021, background): “, a mixture of models (also referred to as model interpolation [61]).”
Personalized Federated Learning with First Order Model Optimization (2020, methodology): “Alternatively, Clustered FL (Sattler et al., 2020; Ghosh et al., 2020; Briggs et al., 2020; Mansour et al., 2020) assumes that inherent partitions or data distributions exist behind clients’ local data, and aim to cluster these partitions to federate within each cluster.”
Federated Learning on Non-IID Data: A Survey (2021, methodology): “The first similarity evaluation approach is reported in [93, 68, 39, 38] by comparing the loss values of different cluster models.”
Personalized Federated Learning using Hypernetworks (2021, background): “Alternatively, clustering methods for federated learning assume that the local data of each client is partitioned by nature (Mansour et al., 2020).”
Lower Bounds and Optimal Algorithms for Personalized Federated Learning (2020, background): “There are multiple strategies in the literature for incorporating e personalization into FL: multi-task learning [49, 48, 14], transfer learning [54, 23], variational inference [8], mixing of local and global models [41, 19, 31, 10] and others [13].”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “Two studies (Briggs et al., 2020; Mansour et al., 2020) clustered similar clients to match the data distribution within a cluster and learned separate models for each cluster without inter-cluster federation.”
Personalized Federated Learning: An Attentive Collaboration Approach (2020, methodology): “The most practical way to build a personalized model for a client is to customize a global model by local ﬁne-tuning [10, 11, 12, 20, 21, 22], where the parameters of the global model are used as the initialization to train a personalized model on the local data of every client.”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “Personalized models are typically found in the realms of recommendation systems (Ben-hamdi et al., 2017; Amat et al., 2018; Martinez et al., 2009; Cho et al., 2002) or in federated learning (Mansour et al., 2020; Jiang et al., 2019; Fallah et al., 2020; Shamsian et al., 2021).”