Think Locally, Act Globally: Federated Learning with Local and Global Representations

By Percy Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, Louis-philippe Morency. 2020

750 citations recorded by the source corpus.

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.

Subfields: Artificial Intelligence, Sociology and Political Science, Transportation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Guodong Long
3. Jing Jiang
4. Tianyi Zhou
5. Bo Du
6. Chengqi Zhang
7. Aryan Mokhtari
8. Mang Ye

In Sociology and Political Science

1. Dacheng Tao
2. Azzam Mourad
3. Nicholas Donald Lane
4. Virginia Smith
5. Tian Li
6. Bo Du
7. Tarik Taleb
8. Hadi Otrok

In Transportation

1. Bo Du
2. Mang Ye
3. Aryan Mokhtari
4. Chengqi Zhang
5. Guodong Long
6. Jing Jiang
7. Hangyu Zhu
8. Virginia Smith

Citations in the corpus

• Federated Mutual Learning (2020, background): “Each client has different communication and computing capabilities, different representation of local data[4, 5] , or the need of fast deployment to devices by neural architecture search (NAS)[6], for which they may require to design their own models.”
• Personalized Federated Learning using Hypernetworks (2021, methodology): “To align with previous works (Dinh et al., 2020; Liang et al., 2020; Fallah et al., 2020a), we use a LeNet-based (target) network with two convolution layers, where the second layer has twice the number of filters in comparison to the first.”
• Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, background): “Specifically, we need communication techniques that can prove improvements at the Pareto frontier, thus giving an accuracy that is larger than any other approach under the same communication budget and across a large domain of communication/accuracy profiles [13], [117].”
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “Several works aim to make the global model customized to suit the need of individual users, either by treating each client as a task in meta-learning [19, 28, 20, 29] or multi-task learning [30], or by learning both global parameters for all clients and local private parameters for individual client”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “[128] LG-FedAvg uses personalized layers to extract high-level features and server-shared base layers for classification.”
• Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “Another direction is introducing shared extra model such as FML [72] and LGFEDAVG [48].”
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients (2020, methodology): “A research trend is to adapt the global model to accommodate personalized local models for non-IID data (Liang et al., 2020), e.”
• Multi-center federated learning: clients clustering for better personalization (2020, methodology): “The personalizing strategy either applies fine-tuning of the global model [56] for each node, or only updates a subset of personalized layers for each node [44, 53], or deploys a regularization term in the objective [57–59].”
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets (2020, methodology): “However, very few efforts have been spent on addressing these two critical challenges simultaneously, The only possible exception is LG-FedAvg, which is proposed by Liang et al. [13].”
• Exploiting Shared Representations for Personalized Federated Learning (2021, methodology): “Unlike in the paper introducing LG-FedAvg [Liang et al., 2020], we did not initialize the models for all methods with the solution of many rounds of FedAvg (instead, we initialized randomly) and we computed the local test accuracy as the average local test accuracy over the final ten communication…”
• FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, background): “Some recent studies (Arivazhagan et al. 2019; Liang et al. 2020; Deng, Kamani, and Mahdavi 2020) train personalized models to leverage both globally shared information and the personalized part (Tan et al. 2021; Jiang, Ji, and Long 2020).”
• FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, methodology): “LG-FedAvg (Liang et al., 2020) learns the entire network jointly during local updates and only aggregates the top layers based on the pre-trained global network via FedAvg.”
• Personalized Federated Learning with First Order Model Optimization (2020, methodology): “Based on prior work (McMahan et al., 2016; Liang et al., 2020), we compare methods with the MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009), and CIFAR-100 datasets.”
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (2021, background): “Until now, a lot of research effort has been invested with the sole goal of maximising the accuracy of the global model [49, 45, 42, 34, 66], while complementary mechanisms have been proposed to ensure privacy and robustness [7, 17, 50, 51, 30, 5].”
• Federated Learning for Open Banking (2021, background): “[24] thinks representation layers should be the personalized part, and then the decision layers could be shared across participants.”
• Ditto: Fair and Robust Federated Learning Through Personalization (2020, methodology): “…meta-learning (Jiang et al., 2019; Chen et al., 2018; Khodak et al., 2019; Fallah et al., 2020; Li et al., 2020a; Singhal et al., 2021), and other forms of MTL, such as hard model parameter sharing (Agarwal et al., 2020; Liang et al., 2020) or the weighted combination method in Zhang et al. (2021).”