Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

By Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H. Vincent Poor. 2020

2,059 citations recorded by the source corpus.

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.

Subfields: Artificial Intelligence, Computer Networks and Communications, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Qiang Yang
  2. Jiashi Feng
  3. Dacheng Tao
  4. Yang Liu
  5. Quoc V. Le
  6. Eric P. Xing
  7. Percy Liang
  8. Mohit Bansal

In Computer Networks and Communications

  1. Young Geun Kim
  2. Carole-Jean Wu
  3. Hamed Hassani
  4. Yuanming Shi
  5. Yang Liu
  6. Bingsheng He
  7. Zeyi Wen
  8. Qinbin Li

In Optimization & Training

  1. Jianyu Wang
  2. Aryan Mokhtari
  3. Gauri Joshi
  4. Yae Jee Cho
  5. Giovanni Neglia
  6. Othmane Marfoq
  7. Richard Vidal
  8. Yuanming Shi

In Federated & Distributed Learning

  1. Qiang Yang
  2. Murali Annavaram
  3. Chaoyang He
  4. Yang Liu
  5. Bingsheng He
  6. Qinbin Li
  7. Praneeth Vepakomma
  8. S. Avestimehr

Citations in the corpus