Federated Learning on Non-IID Data Silos: An Experimental Study

By Qinbin Li, Yiqun Diao, Quan Chen, Bingsheng He. IEEE International Conference on Data Engineering, 2021

1,460 citations recorded by the source corpus.

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple “data silos” (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in “data silos”.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/dd30a98f5274541f018a762fba46a0730519606a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Du
2. Mang Ye
3. Dacheng Tao
4. Bingsheng He
5. Qinbin Li
6. Hangyu Zhu
7. Zeyi Wen
8. Zhaomin Wu

Citations in the corpus

• Federated Learning on Non-IID Data: A Survey (2021, methodology): “Generate synthetic Non-IID data by arbitrary partitioning datasets cannot effectively evaluate the performance of a method proposed for handling Non-IID data [79].”
• NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (2022, methodology): “[23, 15, 43, 42] Specifically, we simulate a non-i.i.d partition for T tokens on C classes.”
• Model-Contrastive Federated Learning (2021, background): “The data can be non-identically distributed among the parties in many real-world applications, which can degrade the performance of federated learning [22, 29, 24].”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “To explore the difficulties of the statistical heterogeneity with the Non-IID phenomenon, we classify statistical heterogeneity from a distribution perspective [96, 117].”
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “, non-IID data) [97], which can degrade the performance of federated learning a lot [82], [100], [103].”
• Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “…the literature typically (i) require stringent assumptions, e.g. , bounded gradients (Seide et al., 2014; Koloskova et al., 2019; Basu et al., 2019; Li et al., 2022b) or bounded gradient dissimilarity (Huang et al., 2022; Li and Li, 2023), (ii) rely on impractical algorithmic structure, e.g. ,…”
• Federated Learning for Smart Healthcare: A Survey (2021, background): “Quantitative metrics are needed to assess non-iid data in the FL-based smart healthcare sector, such as standard deviation, precision, and accuracy with respect to label/feature distribution skew and homogeneous partitions [128].”
• Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “d (identically and independently distributed), leading to data heterogeneity [30,39,95].”