Model-Contrastive Federated Learning

By Qinbin Li, Bingsheng He, Dawn Song. Computer Vision and Pattern Recognition, 2021

1,746 citations recorded by the source corpus.

Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data. A key challenge in federated learning is to handle the heterogeneity of local data distribution across parties. Although many studies have been proposed to address this challenge, we find that they fail to achieve high performance in image datasets with deep learning models. In this paper, we propose MOON: model-contrastive federated learning. MOON is a simple and effective federated learning framework. The key idea of MOON is to utilize the similarity between model representations to correct the local training of individual parties, i.e., conducting contrastive learning in model-level. Our extensive experiments show that MOON significantly outperforms the other state-of-the-art federated learning algorithms on various image classification tasks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/748c4f1945b0d994ecef38c8aac01db1c6dc7029

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Bo Du
3. Jian Liang
4. Guodong Long
5. Jing Jiang
6. Mang Ye
7. Chengqi Zhang
8. Jiashi Feng

Citations in the corpus

• Federated Learning on Non-IID Data Silos: An Experimental Study (2021, background): “When preparing this paper, there are other contemporary works [2], [40], [48], [73] on federated learning under non-IID setting.”
• FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, background): “On-client intelligent agents in the FL system can share knowledge by exchanging information in terms of representations, despite statistical and model heterogeneity (Cai et al. 2020; Li et al. 2021).”
• Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, methodology): “Shortly after, a large body methods [12,41,43,73,77] research on non-i.i.d data.”
• Towards Personalized Federated Learning (2021, methodology): “The goal of MOON is to reduce the distance between the representations learned by the local models and the global model (i.e., to alleviate weight divergence), and increase the distance between the representations learned between a given local model and its previous local model (i.e., to speed up co”
• Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, neutral): “To improve over FedAvg, previous works have either focused on local client training [18, 19, 20, 21, 22, 23, 24] or global server aggregation [25, 26, 27, 28, 29, 30].”
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, background): “There are also some works incorporate contrastive learning into federated learning to assist local training to achieve higher model performance [43, 44, 45].”
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “Another approach is to record the time or round to achieve the same target performance [82], [99].”
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning (2022, methodology): “…and FedRep (Collins et al. 2021), (2) methods that learn additional personalized models, including pFedMe (T Dinh, Tran, and Nguyen 2020) and Ditto (Li et al. 2021a), and (3) methods that learn local models with personalized (local) aggregation, including FedAMP (Huang et al. 2021), Fed-PHP (Li et…”
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, methodology): “MOON [8] adopts the contrastive loss to maximize the agreement of the representation learned by the local model and that by the global model.”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “Unlike these methods, MOON [118] considers not only the regularization between the global model and the local model but also the regularization between the current local model and the previous local model.”