Federated User Representation Learning

By Duc Viet Bui, Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, Kang G. Shin. 2019

73 citations recorded by the source corpus.

Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propose Federated User Representation Learning (FURL), a simple, scalable, privacy-preserving and resource-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting. FURL divides model parameters into federated and private parameters. Private parameters, such as private user embeddings, are trained locally, but unlike federated parameters, they are not transferred to or averaged on the server. We show theoretically that this parameter split does not affect training for most model personalization approaches. Storing user embeddings locally not only preserves user privacy, but also improves memory locality of personalization compared to on-server training. We evaluate FURL on two datasets, demonstrating a significant improvement in model quality with 8% and 51% performance increases, and approximately the same level of performance as centralized training with only 0% and 4% reductions. Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that FURL can learn collaboratively through the shared parameters while preserving user privacy.

Subfields: Artificial Intelligence, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jian Liang
2. Jiashi Feng
3. Yifan Zhang
4. Dapeng Hu
5. Han Yu
6. Qiang Yang
7. Li-zhen Cui
8. Alysa Ziying Tan

In Sociology and Political Science

1. Li-zhen Cui
2. Qiang Yang
3. Alysa Ziying Tan
4. Han Yu
5. Jiashi Feng

Citations in the corpus

• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “Several works aim to make the global model customized to suit the need of individual users, either by treating each client as a task in meta-learning [19, 28, 20, 29] or multi-task learning [30], or by learning both global parameters for all clients and local private parameters for individual client”
• Towards Personalized Federated Learning (2021, methodology): “Method Quantity Skew Feature Distribution Skew Label Distribution Skew Label Preference Skew Data Augmentation [23] [24] [21]–[23] Client Selection [28] [27]–[28] [25]–[28] Regularization [29] [29] [12], [29]–[32] Meta-Learning [39] [36], [40] [37], [39] Transfer Learning [42]–[44] [42] Parameter De”