The Non-IID Data Quagmire of Decentralized Machine Learning

By Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip B. Gibbons. 2019

692 citations recorded by the source corpus.

Many large-scale machine learning (ML) applications need to perform decentralized learning over datasets generated at different devices and locations. Such datasets pose a significant challenge to decentralized learning because their different contexts result in significant data distribution skew across devices/locations. In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. Our study shows that: (i) skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms; (ii) the problem is particularly challenging for DNN models with batch normalization; and (iii) the degree of data skew is a key determinant of the difficulty of the problem. Based on these findings, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the accuracy loss of batch normalization.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Science Applications, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Science Applications

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “[205], we first survey some common ways in which data tend to deviate from being identically distributed, that is Pi 6= Pj for different clients i and j.”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, methodology): “[10] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B Gibbons.”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, positive): “I.D. setting is lower than that of the I.I.D. setting, which is consistent with ﬁndings reported in [76].”
Adaptive Federated Optimization (2020, methodology): “We use the same approach as Hsu et al. (2019), where we apply latent Dirichlet allocation (LDA) over the labels of CIFAR-10 to create a federated dataset.”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “This is confirmed by both this study and other CNN-based FL works [26, 27]; 2.”
Federated Visual Classification with Real-World Data Distribution (2020, methodology): “We replaced BatchNorm with GroupNorm [32] due to its superior stability for FL tasks [10].”
Federated Learning in Edge Computing: A Systematic Survey (2022, background): “For neural networks trained on highly biased non-IID data, the FL model accuracy drops by around 55% when each client device trains on only a single class of data [183].”
Federated Learning With Quantized Global Model Updates (2020, background): “Diﬀerent approaches have been studied to mitigate the eﬀect of non-iid data in FL [3, 13–17].”
FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning (2020, methodology): “See subsection C.5 for a discussion on using group normalization (GN) (Wu & He, 2018; Hsieh et al., 2020), which converges much slower.”
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (2021, methodology): “(2019); Hsieh et al. (2019) can be described by the joint probability between features x and labels y on each client. We can rewrite Pi(x, y) as Pi(y|x)Pi(x) and Pi(x|y)Pi(y). We define feature shift as the case that covers: 1) covariate shift: the marginal distributions Pi(x) varies across clients,”
On Large-Cohort Training for Federated Learning (2021, methodology): “The use of group norm over batch norm in federated learning was ﬁrst advocated by Hsieh et al. (2020), who showed that this helped improve classiﬁcation accuracy in the presence of heterogeneous clients.”
A first look into the carbon footprint of federated learning (2020, background): “Due to these regulations, moving data across national borders becomes subject to data sovereignty law, making centralized training infeasible in some scenarios (Hsieh et al., 2020).”
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, neutral): “In this work, we focus on addressing the non-IID quagmire [6, 24].”
Personalized Federated Learning with First Order Model Optimization (2020, background): “…a natural strategy to personalize (Mansour et al., 2020; Wang et al., 2019), prior work has shown that non-IID decentralized data can introduce challenges such as parameter divergence (Zhao et al., 2018), data distribution biases (Hsieh et al., 2019), and unguaranteed convergence Li et al. (2020).”