Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
By Tzu-Ming Harry Hsu, Hang Qi, Matthew Brown. 2019
1,651 citations recorded by the source corpus.
Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.
Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Martin Jaggi
- Jakub Konecný
- Qiang Yang
- Peter Richtárik
- Sebastian U. Stich
- Dacheng Tao
- Virginia Smith
- Ananda Theertha Suresh
In Computer Science Applications
- Zachary B. Charles
- H. Brendan McMahan
- Bingsheng He
- Qinbin Li
- Sashank J. Reddi
- Sebastian U. Stich
- Galen Andrew
- Dawn Song
In Computer Vision and Pattern Recognition
- Jiashi Feng
- Dacheng Tao
- Luc Van Gool
- Bernt Schiele
- Trevor Darrell
- Sebastian U. Stich
- Jian Liang
- Yi Yang
In Federated & Distributed Learning
- Jianyu Wang
- Gauri Joshi
- Bingsheng He
- Qinbin Li
- Virginia Smith
- Zachary B. Charles
- Martin Jaggi
- Sebastian U. Stich
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, neutral): “[206] proposed a momentum scheme similar to SlowMo.”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, methodology): “This is referred to as server-momentum since it is computed and applied only at the server level [11].”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “…step at the t -round will be corrected by Besides, on the server side, one can also implement server momentum or adaptive server optimizers [25, 42, 40], in which the aggregated normalized gradient − τ eff =1 ηp i d i is used to update the server momentum buffer instead of directly updating…”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “Modelling assumptions and relevance As FL has matured, it has become more evident that there are two varieties, with distinct system-imposed constraints, recently termed by Kairouz et al. (2019) as cross-device federated learning and cross-silo federated learning.”
- Adaptive Federated Optimization (2020, methodology): “), as well as techniques such as server-side momentum (leading to FedAvgM, proposed by Hsu et al. (2019)).”
- Salvaging Federated Learning by Local Adaptation (2020, background): “9 , similar to (Hsu et al., 2019).”
- Differentially Private Learning with Adaptive Clipping (2019, positive): “We augment basic federated averaging with server momentum, which improves convergence (Hsu et al., 2019; Reddi et al., 2020).”
- Federated Visual Classification with Real-World Data Distribution (2020, methodology): “We follow the protocol described in [11] such that the class distribution of every client is sampled from a Dirichlet distribution with varying concentration parameter α.”
- Federated Learning Based on Dynamic Regularization (2021, methodology): “, 2020a); or modifying server side updates (Hsu et al., 2019; Reddi et al., 2020).”
- Model-Contrastive Federated Learning (2021, methodology): “FedAvgM [14] applies momentum when updating the global model on the server.”
- Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning (2021, background): “, 2018; Hsu et al., 2019; Karimireddy et al., 2019; Wang et al., 2019a) have emerged as a prevailing approach for FL. Similar to the traditional distributed learning, FedAvg leverages local computation at each worker and employs a centralized parameter server to aggregate and update the model parame”
- Towards Personalized Federated Learning (2021, background): “An α of 100 is equivalent to the IID setting, while a smaller α value means that each client is more likely to hold data from only one class resulting in high data heterogeneity [55], [56], [86].”
- Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data (2020, background): “(3) There can be different levels of non-IID data across clients, as discussed in (Hsu et al. (2019)), which provides a simulation process.”
- Federated Learning with Buffered Asynchronous Aggregation (2021, methodology): “Figure 1: (left) Number of communication rounds and number of client updates required by FedAvgM [11], a synchronous FL algorithm, to reach a target accuracy on the Sent140 [4].”
- SCAFFOLD: Stochastic Controlled Averaging for Federated Learning (2019, methodology): “We divide this dataset among N = 100 clients as follows: for s % similar data we allocate to each client s % i.i.d. data and the remaining (100 − s )% by sorting according to label (cf. Hsu et al. (2019)).”
- FairFed: Enabling Group Fairness in Federated Learning (2021, positive): “In our context, we use a generic non-IID synthesis method based on the Dirichlet distribution proposed in (Hsu, Qi, and Brown 2019) but apply it in a novel way for configurable sensitive attribute distribution : for each sensitive attribute value a , we sample p a ∼ Dir( α ) and allocate a portion p…”