Federated Optimization in Heterogeneous Networks

By Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, Virginia Smith. Conference on Machine Learning and Systems, 2018

8,339 citations recorded by the source corpus.

Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the network (statistical heterogeneity). In this work, we introduce a framework, FedProx, to tackle heterogeneity in federated networks. FedProx can be viewed as a generalization and re-parametrization of FedAvg, the current state-of-the-art method for federated learning. While this re-parameterization makes only minor modifications to the method itself, these modifications have important ramifications both in theory and in practice. Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity). Practically, we demonstrate that FedProx allows for more robust convergence than FedAvg across a suite of realistic federated datasets. In particular, in highly heterogeneous settings, FedProx demonstrates significantly more stable and accurate convergence behavior relative to FedAvg---improving absolute test accuracy by 22% on average.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training, Federated & Distributed Learning

View Paper: https://arxiv.org/abs/1812.06127

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dacheng Tao
  2. Peter Richtárik
  3. Percy Liang
  4. Qiang Yang
  5. Martin Jaggi
  6. Yoshua Bengio
  7. Mohit Bansal
  8. Ruslan Salakhutdinov

In Electrical and Electronic Engineering

  1. H. Vincent Poor
  2. Khaled Ben Letaief
  3. Mingzhe Chen
  4. Walid Saad
  5. Yuanming Shi
  6. Jianyu Wang
  7. Shuguang Cui
  8. Jianmin Lu

In Optimization & Training

  1. Peter Richtárik
  2. Sebastian U. Stich
  3. Jakub Konecný
  4. Martin Jaggi
  5. Jianyu Wang
  6. Nicolas Loizou
  7. Gauri Joshi
  8. Aryan Mokhtari

In Federated & Distributed Learning

  1. Jianyu Wang
  2. Peter Richtárik
  3. Gauri Joshi
  4. Virginia Smith
  5. Mehrdad Mahdavi
  6. Jakub Konecný
  7. Aryan Mokhtari
  8. Zachary B. Charles

Citations in the corpus