Federated Learning with Non-IID Data
By Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra. 2018
3,343 citations recorded by the source corpus.
Federated learning enables resource-constrained edge compute devices, such as mobile phones and IoT devices, to learn a shared model for prediction, while keeping the training data local. This decentralized approach to train models provides privacy, security, regulatory and economic benefits. In this work, we focus on the statistical challenge of federated learning when local data is non-IID. We first show that the accuracy of federated learning reduces significantly, by up to 55% for neural networks trained for highly skewed non-IID data, where each client device trains only on a single class of data. We further show that this accuracy reduction can be explained by the weight divergence, which can be quantified by the earth mover's distance (EMD) between the distribution over classes on each device and the population distribution. As a solution, we propose a strategy to improve training on non-IID data by creating a small subset of data which is globally shared between all the edge devices. Experiments show that accuracy can be increased by 30% for the CIFAR-10 dataset with only 5% globally shared data.
Subfields: Artificial Intelligence, Computer Science Applications, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Dacheng Tao
- Yoshua Bengio
- Martin Jaggi
- Peter Richtárik
- Sergey Levine
- Percy Liang
- Mohit Bansal
In Computer Science Applications
In Federated & Distributed Learning
- Jianyu Wang
- Gauri Joshi
- Virginia Smith
- Peter Richtárik
- Bingsheng He
- Qinbin Li
- Aryan Mokhtari
- Tian Li
Citations in the corpus
- Federated Optimization in Heterogeneous Networks (2018, background): “…on network bandwidth, sending local data to the server (Jeong et al., 2018) violates the key privacy assumption of federated learning, and sending globally-shared proxy data to all devices (Zhao et al., 2018; Huang et al., 2018) requires effort to carefully generate or collect such auxiliary data.”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “[44] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, background): “[32] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.”
- Personalized Federated Learning: A Meta-Learning Approach (2020, background): “More related to our paper, there are several works that study statistical heterogeneity of users’ data points in FL (Zhao et al., 2018; Sahu et al., 2018; Karimireddy et al., 2019; Haddadpour and Mahdavi, 2019; Khaled et al., 2019; Li et al., 2019), but they do not attempt to find a personalized…”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Besides, many empirical studies are investigating the effect of system heterogeneity [31, 55], the power of scalability [22], strategies of preserving privacy [6, 27], and saving communication [9].”
- SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “Finally, Zhao et al. (2018) propose global sharing of the clients’ data.”
- The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “The closest to our work is Zhao et al.’s study (Zhao et al., 2018) on FederatedAveraging over non-IID data, which shows FederatedAveraging suffers significant accuracy loss in the Non-IID setting.”
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification (2019, methodology): “In a similar sort-and-partition manner, Zhao et al. [2018] and Sattler et al. [2019] generate extreme partitions on the CIFAR-10 dataset, forming a population consisting of 10 clients in total.”
- On the Convergence of FedAvg on Non-IID Data (2019, background): “Sattler et al. [22], Zhao et al. [39] studied the non-iid setting, however, they do not have convergence rate.”
- Decentralized Bayesian Learning over Graphs (2019, neutral): “This has been also shown to lead to poor accuracy empirically in the federated learning literature [28].”
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, methodology): “them from diverging too far from one another, and hence, top-k sparsification does not suffer from weight divergence [32] as it is the case for federated averaging.”
- Federated Machine Learning (2019, background): “The authors of [77] showed that, with non-IID local data, performance can be greatly reduced for federated learning.”
- Wireless Network Intelligence at the Edge (2018, methodology): “The non-IID training data set of on-device ML can be corrected by obtaining the missing local data samples at each device from the other devices [179].”
- Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data (2018, methodology): “Preserving data privacy facilitates such access, in a way that a global model is collectively trained not by directly sharing private data but by exchanging the local model parameters of devices, as exemplified by federated learning (FL) [3–9].”
- Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Zhao et al. [138] explore transfer learning for personalization by running FedAvg after training a global model centrally on some shared proxy data.”
- LoAdaBoost: Loss-based AdaBoost federated machine learning with reduced computational complexity on IID and non-IID intensive care data (2018, methodology): “[11] addressed the non-IID data challenges in federated learning and presented an improved version of FedAvg with a data-sharing strategy whereby the test accuracy could be enhanced significantly with only a small portion of globally shared data among clients.”