Exploiting Shared Representations for Personalized Federated Learning

By Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai. International Conference on Machine Learning, 2021

1,173 citations recorded by the source corpus.

Deep neural networks have shown the ability to extract universal feature representations from data such as images and text that have been useful for a variety of learning tasks. However, the fruits of representation learning have yet to be fully-realized in federated settings. Although data in federated settings is often non-i.i.d. across clients, the success of centralized deep learning suggests that data often shares a global feature representation, while the statistical heterogeneity across clients or tasks is concentrated in the labels. Based on this intuition, we propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. We prove that this method obtains linear convergence to the ground-truth representation with near-optimal sample complexity in a linear setting, demonstrating that it can efficiently reduce the problem dimension for each client. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions, for example in meta-learning and multi-task learning. Further, extensive experimental results show the empirical improvement of our method over alternative personalized federated learning approaches in federated environments with heterogeneous data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ffd393dacee23e476bd8eb0802dec86f2296b36c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Bo Du
3. Richard E. Turner
4. Mang Ye
5. Guodong Long
6. Jing Jiang
7. Tianyi Zhou
8. Sebastian Nowozin

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Bo Du
3. Mang Ye
4. Luc Van Gool
5. Ling Shao
6. Ming-Hsuan Yang
7. Trevor Darrell
8. Xiaogang Wang

Citations in the corpus

• FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, background): “• The partial model personalization literature (Collins et al., 2021; Arivazhagan et al., 2019; Liang et al., 2020; Pillutla et al., 2022) is more concerned with proposing stable federated training algorithms that would work within heterogeneous settings, where clients have diverse data and…”
• FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, methodology): “In addition, the efficiency of personalized FL algorithms increases when all clients participate; therefore, FedPer (Arivazhagan et al., 2019) assumes the activation of all clients during every communication round.”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “, sharing partial structures [35, 139], model optimization [46, 118, 124], knowledge transfer [48, 85].”
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning (2022, methodology): “Recent pFL studies on model aggregation on the server can be classified into three categories: (1) methods that learn a single global model and fine-tune it, including Per-FedAvg (Fallah, Mokhtari, and Ozdaglar 2020) and FedRep (Collins et al. 2021), (2) methods that learn additional personalized mo”
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, methodology): “[3, 7] share part of the model and keep personalized layers private to achieve personalization.”