Salvaging Federated Learning by Local Adaptation

By Tao Yu, Eugene Bagdasarian, Vitaly Shmatikov. 2020

315 citations recorded by the source corpus.

Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the federated model, latest FL approaches use differential privacy or robust aggregation to limit the influence of "outlier" participants. First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants. Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each technique is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model and who have no incentive to participate in FL today improve less, but sufficiently to make the adapted federated model better than their local models.

Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Sociology and Political Science

In Federated & Distributed Learning

Citations in the corpus

Differentially Private Learning Needs Better Features (or Much More Data) (2020, background): “Yet, training deep neural networks with strong DP guarantees comes at a significant cost in utility (Abadi et al., 2016; Yu et al., 2020; Bagdasaryan et al., 2019; Feldman, 2020).”
Adaptive Personalized Federated Learning (2020, background): “Yu et al. (2020) argue that the global model learned by federated learning, especially with having diﬀerential privacy and robust learning objectives, can hurt the performance of many clients.”
Federated Learning on Non-IID Data: A Survey (2021, background): “However, the local models trained by FL may be harmed by the shared model in terms of performance [158], and may fail to generalize due to the drift of heterogeneous data shards [27].”
Federated Mutual Learning (2020, background): “However, in the personalization situation, Yu et al. [15] shows that some participants may not beneﬁt from the global model when the global model is less accurate than the local model.”
Federated Learning for Open Banking (2021, methodology): “In [37], the pre-trained global model is treated as the teacher, while the adapted model is treated as a student.”
Exploiting Shared Representations for Personalized Federated Learning (2021, background): “A variety of recent works have studied personalization in federated learning using, for example, local fine-tuning (Wang et al., 2019; Yu et al., 2020), metalearning (Chen et al.”
Ditto: Fair and Robust Federated Learning Through Personalization (2020, methodology): “, 2020), finetuning/transfer learning (Zhao et al., 2018; Yu et al., 2020), meta-learning (Jiang et al.”
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, background): “Personalization Ditto [37] EWC [71] 86.”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “[236] attempt to mitigate overfitting in personalized updates by enhancing the logits similarity between the global model and the local models.”
Survey of Personalization Techniques for Federated Learning (2020, background): “In some cases, however, personalized models fail to reach the same performance as local models, especially when differential privacy and robust aggregation is implemented [7].”