Federated Learning of a Mixture of Global and Local Models

By Filip Hanzely, Peter Richtárik. 2020

447 citations recorded by the source corpus.

We propose a new optimization formulation for training federated learning models. The standard formulation has the form of an empirical risk minimization problem constructed to find a single global model trained from the private data stored across all participating devices. In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication. Further, we develop several efficient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees. Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the essence of the elusive method. In particular, our methods do not perform full averaging steps and instead merely take steps towards averaging. We argue for the benefits of this new paradigm for federated learning.

Subfields: Artificial Intelligence, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

Citations in the corpus

Adaptive Personalized Federated Learning (2020, background): “Hanzely and Richt´arik (2020) try to introduce a general approach for federated learning by combining the optimization of the local and global models.”
Federated Learning for Open Banking (2021, background): “[10,16] enables participants to pursue their personalized models with diﬀerent directions, but use a regularization term to limit each personalized model to not far away from the “initial point”, the global model.”
Exploiting Shared Representations for Personalized Federated Learning (2021, background): “APFL [Deng et al., 2020] interpolates between local and global models, and L2GD [Hanzely and Richt´arik, 2020] and Ditto [Li et al., 2020] learn local models that are encouraged to be close together by global regularization.”
Personalized Federated Learning using Hypernetworks (2021, background): “…Lin et al., 2018), others tackle the more challenging heterogeneous setup in which each client is equipped with its own data distribution (Zhou & Cong, 2017; Hanzely & Richt´arik, 2020; Zhao et al., 2018; Sahu et al., 2018; Karim-ireddy et al., 2019; Haddadpour & Mahdavi, 2019; Hsu et al., 2019).”
Federated Multi-Task Learning under a Mixture of Distributions (2021, methodology): “Other MTL-based approaches [24, 23, 16, 26, 36] are able to train more general models at the cost of considering simpler penalization terms (e.”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “Moreover, they apply accelerated proximal gradient descent (APGD) and accelerated L2SGD+ [68], which can achieve optimal complexity bound.”
Survey of Personalization Techniques for Federated Learning (2020, background): “Hanzely et al. [8] question the utility of a global model that is too far removed from the typical usage of a user.”
Federated Learning on Non-IID Data: A Survey (2021, background): “design a new form of the cost function by adding a regularization term to investigate a trade-off between local and global models [45].”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, unknown): “[69] find a trade-off between the global and local models by adding a regularization term, and learn an implicit mixture model of the global and local models.”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, result): “8Similar results were reported in the recent paper (Collins et al., 2021), showing that FedAvg beats recent personalized algorithms (Smith et al., 2017; Fallah et al., 2020; Liang et al., 2020; Hanzely & Richtárik, 2020; Deng et al., 2020; Li et al., 2021; Arivazhagan et al., 2019) in many cases.”
Personalized Federated Learning: An Attentive Collaboration Approach (2020, methodology): “2, most existing methods [9, 10, 11, 12, 13, 14, 15, 16] that produce personalized models by locally customizing a global model cannot effectively identify the underlying collaboration relationships between clients.”
Multi-center federated learning: clients clustering for better personalization (2020, methodology): “The detailed method and optimization procedure is provided in Section 4.”
Lower Bounds and Optimal Algorithms for Personalized Federated Learning (2020, methodology): “, x ? n] ∈ R of (2) can be expressed as xi = x̄ − 1 λ∇fi(x ? i ), where x̄ ? = 1 n ∑n i=1 x ? i [19], which strongly resembles the influential modelagnostic meta learning (MAML) framework from [15].”
Towards Personalized Federated Learning (2021, background): “Multi-Task Learning [59]–[62] [59], [61]–[62] [61]–[62] [59, 61] - [59, 61] - Model Interpolation [17], [64]–[65] [17], [63]–[65] [63]–[65] [65] [65] [65] - - Clustering [66]–[70], [72] [66]–[70], [72] [66]–[67], [69]–[70], [72] - - -”
Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “The most closely related works are those that regularize personalized models towards their average (Hanzely & Richtárik, 2020; Hanzely et al., 2020; Dinh et al., 2020), which can be seen as a form of classical mean-regularized MTL (Evgeniou & Pontil, 2004).”
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach (2020, background): “Also, in [39], the authors consider a framework for training a mixture of a single global model and local models, leading to a personalized solution for each user.”