FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

By Xinwei Zhang, Mingyi Hong, Sairaj V. Dhople, Wotao Yin, Yang Liu. 2020

286 citations recorded by the source corpus.

Federated Learning (FL) is popular for communication-efficient learning from distributed data. To utilize data at different clients without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a computation then aggregation model, in which multiple local updates are performed using local data before aggregation. These algorithms fail to work when faced with practical challenges, e.g., the local data being non-identically independently distributed. In this paper, we first characterize the behavior of the FedAvg algorithm, and show that without strong and unrealistic assumptions on the problem structure, it can behave erratically. Aiming at designing FL algorithms that are provably fast and require as few assumptions as possible, we propose a new algorithm design strategy from the primal-dual optimization perspective. Our strategy yields algorithms that can deal with non-convex objective functions, achieves the best possible optimization and communication complexity (in a well-defined sense), and accommodates full-batch and mini-batch local computation models. Importantly, the proposed algorithms are communication efficient, in that the communication effort can be reduced when the level of heterogeneity among the local data also reduces. In the extreme case where the local data becomes homogeneous, only $\mathcal {O}(1)$ communication is required among the agents. To the best of our knowledge, this is the first algorithmic framework for FL that achieves all the above properties.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies (2020, methodology): “The effect of data heterogeneity on the convergence of local-update SGD is analyzed in several recent works [9–19] and methods to overcome the adverse effects of data and computational heterogeneity are proposed in [19–21], among others.”
Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “Substantial efforts have been made to address client heterogeneity in FL (Liang et al., 2019; Li et al., 2020b,a; Wang et al., 2020a; Zhang et al., 2021; Haddadpour et al., 2021; Yuan and Li, 2022; Alghunaim, 2023; Cheng et al., 2024), and develop other FL protocols involving variance reduction…”
Federated Learning with Compression: Unified Analysis and Sharp Guarantees (2020, background): “cally evolving models [14,27], which are not improvable in general [70].”
Personalized Cross-Silo Federated Learning on Non-IID Data (2020, background): “The performance of FedAvg and FedProx degenerates a lot on FMNIST and EMNIST, because taking the global aggregation of all personalized models trained on the non-IID data of different clients introduces signiﬁcant unstableness to the gradient-based optimization process (Zhang et al. 2020).”