No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data

By Romy Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, Jiashi Feng. Neural Information Processing Systems, 2021

483 citations recorded by the source corpus.

A central challenge in training classification models in the real-world federated system is learning with non-IID data. To cope with this, most of the existing works involve enforcing regularization in local optimization or improving the model aggregation scheme at the server. Other works also share public datasets or synthesized samples to supplement the training of under-represented classes or introduce a certain level of personalization. Though effective, they lack a deep understanding of how the data heterogeneity affects each layer of a deep classification model. In this paper, we bridge this gap by performing an experimental analysis of the representations learned by different layers. Our observations are surprising: (1) there exists a greater bias in the classifier than other layers, and (2) the classification performance can be significantly improved by post-calibrating the classifier after federated training. Motivated by the above findings, we propose a novel and simple algorithm called Classifier Calibration with Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated gaussian mixture model. Experimental results demonstrate that CCVR achieves state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10. We hope that our simple yet effective method can shed some light on the future research of federated learning with non-IID data.

Subfields: Artificial Intelligence, Health Information Management

View Paper: https://www.semanticscholar.org/paper/9563e9578073447b68261eda55e7a6c4fb2af7c8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Du
2. Mang Ye
3. Guodong Long
4. Jing Jiang
5. Tianyi Zhou
6. Dacheng Tao
7. Yue Tan
8. Lu Liu

Citations in the corpus

• AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning (2021, background): “5Note, while per-device data class distribution is also used in a number of prior works [19, 42, 80, 87, 132, 140], it reveals additional information.”
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, background): “[23] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng.”
• FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “Zhuang et al. (2021) proposed a divergence-aware predictor update module, and Luo et al. (2021) and Achituve et al. (2021) proposed a head calibration method.”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “Classifier Calibration with Virtual Representations (CCVR) [139] generates approximate Gaussian mixture model (GMM) based virtual representations in feature space via learned feature extractors.”
• Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, methodology): “To improve over FedAvg, previous works have either focused on local client training [18, 19, 20, 21, 22, 23, 24] or global server aggregation [25, 26, 27, 28, 29, 30].”