Federated Optimization in Heterogeneous Networks
By Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, Virginia Smith. Conference on Machine Learning and Systems, 2018
8,339 citations recorded by the source corpus.
Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the network (statistical heterogeneity). In this work, we introduce a framework, FedProx, to tackle heterogeneity in federated networks. FedProx can be viewed as a generalization and re-parametrization of FedAvg, the current state-of-the-art method for federated learning. While this re-parameterization makes only minor modifications to the method itself, these modifications have important ramifications both in theory and in practice. Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity). Practically, we demonstrate that FedProx allows for more robust convergence than FedAvg across a suite of realistic federated datasets. In particular, in highly heterogeneous settings, FedProx demonstrates significantly more stable and accurate convergence behavior relative to FedAvg---improving absolute test accuracy by 22% on average.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training, Federated & Distributed Learning
View Paper: https://arxiv.org/abs/1812.06127
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Peter Richtárik
- Percy Liang
- Qiang Yang
- Martin Jaggi
- Yoshua Bengio
- Mohit Bansal
- Ruslan Salakhutdinov
In Electrical and Electronic Engineering
- H. Vincent Poor
- Khaled Ben Letaief
- Mingzhe Chen
- Walid Saad
- Yuanming Shi
- Jianyu Wang
- Shuguang Cui
- Jianmin Lu
In Optimization & Training
- Peter Richtárik
- Sebastian U. Stich
- Jakub Konecný
- Martin Jaggi
- Jianyu Wang
- Nicolas Loizou
- Gauri Joshi
- Aryan Mokhtari
In Federated & Distributed Learning
- Jianyu Wang
- Peter Richtárik
- Gauri Joshi
- Virginia Smith
- Mehrdad Mahdavi
- Jakub Konecný
- Aryan Mokhtari
- Zachary B. Charles
Citations in the corpus
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “For example, FEDPROX (Li et al., 2018) incorporates a proximal term for the local training.”
- Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, methodology): “In order to analyze the convergence, the federated proximal algorithm (FedProx) was proposed [19] by adding regularization on each local loss function.”
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The first is Federated Learning [303], where the server and clients are not allowed to communicate with data.”
- Flower: A Friendly Federated Learning Research Framework (2020, methodology): “We therefore apply the concept of partial work (Li et al., 2018) in which a client submits their locally updated θ k upon reaching ∆ along with c k , the number of examples actually involved in computing θ k , even if c k E m k B B .”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “In this setting, we consider three data sets: MNIST [132], Federated EMINST [4], and Synthetic ( α , β ) [3], with the logistic regression model as the benchmark.”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “More sophisticated approaches such as FedProx [38], VRLSGD [21] and SCAFFOLD [20] − x ( t, 0) ) /τ i .”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “…of federated learning with topics including differential privacy (McMahan et al., 2018; Augenstein et al., 2020), fairness (Mohri et al., 2019; Li et al., 2020b), robustness (Ghosh et al., 2019; Bagdasaryan et al., 2018; Sun et al., 2019), and communication-efficiency (Konečný et al., 2016;…”
- Personalized Federated Learning with Moreau Envelopes (2020, methodology): “For synthetic data, we adopt the data generation and distribution procedure from [15], using two parameters ᾱ and β̄ to control how much the local model and the dataset of each client differ, respectively.”
- A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, background): “…with decentralized training: notably, high communication cost (Tang et al., 2018a; Wang et al., 2019; Koloskova et al., 2019), a need for time-varying topologies (Nedi´c & Olshevsky, 2014; Assran et al., 2019) and data-heterogeneity (Li et al., 2018; Karimireddy et al., 2019; Li et al., 2020a;b).”
- Adaptive Federated Optimization (2020, methodology): “Examples include methods that add a regularization term in the client objectives towards the broadcast model (Li et al., 2018), and methods that use momentum on the server (Hsu et al., 2019).”
- Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [7, 29, 33, 47, 50, 57, 69, 75, 79]; see also the recent surveys [46, 56].”
- A Communication Efficient Vertical Federated Learning Framework (2019, background): “Proximal Gradient Descent We add a proximal term [21] when calculating gradients to alleviate potential divergence when local iteration is large.”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Federated Learning (FL), which has a central parameter server, enables massive edge computing devices to jointly learn a centralized model while keeping all local data localized [13, 34, 26, 12, 18, 30, 57].”