Federated Learning with Personalization Layers

By Manoj Ghuhan Arivazhagan, V. Aggarwal, Sunav Choudhary. 2019

1,302 citations recorded by the source corpus.

The emerging paradigm of federated learning strives to enable collaborative training of machine learning models on the network edge without centrally aggregating raw data and hence, improving data privacy. This sharply deviates from traditional machine learning and necessitates the design of algorithms robust to various sources of heterogeneity. Specifically, statistical heterogeneity of data across user devices can severely degrade the performance of standard federated averaging for traditional machine learning applications like personalization with deep learning. This paper pro-posesFedPer, a base + personalization layer approach for federated training of deep feedforward neural networks, which can combat the ill-effects of statistical heterogeneity. We demonstrate effectiveness ofFedPerfor non-identical data partitions ofCIFARdatasetsand on a personalized image aesthetics dataset from Flickr.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Personalized Federated Learning with Moreau Envelopes (2020, methodology): “One personalization method used in neural networks is FedPer [28], in which a network is divided into base and personalized layers, and while the base layers are trained by the server, both types of layers will be trained by users to create a personalized model.”
Adaptive Personalized Federated Learning (2020, background): “Another approach adopted for deep neural networks is introduced by Arivazhagan et al. (2019), where they freeze the base layers and only change the last “personalized” layer for each client locally.”
FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, background): “The local average test accuracy of FedProto is greater than for the FeSEM , FedProx , FedPer and FedAvg algorithms in all the settings.”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “[6] FedPer combines base layers and personalized layers for federated training.”
Personalized Federated Learning using Hypernetworks (2021, background): “Another approach to PFL is model mixing where the clients learn a mixture of the global and local models (Deng et al., 2020; Arivazhagan et al., 2019).”
Multi-center federated learning: clients clustering for better personalization (2020, background): “[53] added a personalized layer for each local model, i.e., FedPer, to tackle heterogeneous data.”
Exploiting Shared Representations for Personalized Federated Learning (2021, methodology): “Among the personalized methods, FedPer [Arivazhagan et al., 2019] is most similar to ours, as it also learns a global representation and personalized heads, but makes simultaneous local updates for both sets of parameters, therefore makes the same number of local updates for the head and the…”
From distributed machine learning to federated learning: a survey (2021, neutral): “However, it is difficult to choose a dataset and its partition among clients to measure the personalization brought by APFL or FedPer, so as to prove the improvement compared with FedAvg.”
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “Several works aim to make the global model customized to suit the need of individual users, either by treating each client as a task in meta-learning [19, 28, 20, 29] or multi-task learning [30], or by learning both global parameters for all clients and local private parameters for individual client”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “In addition, the efficiency of personalized FL algorithms increases when all clients participate; therefore, FedPer (Arivazhagan et al., 2019) assumes the activation of all clients during every communication round.”
Federated Learning for Open Banking (2021, background): “[2] proposes to share representation layers across participants, and then keep decision layers as a personalized part.”
Survey of Personalization Techniques for Federated Learning (2020, background): “To temper the adverse effects of this statistical heterogeneity, Arivazhagan et al. [32] propose FedPer, a neural network architecture where the base layers are trained centrally by Federated Averaging, and the top layers (also called personalization layers) are trained locally with a variant of…”
Federated Learning on Non-IID Data: A Survey (2021, background): “A typical paradigm called FedPer is introduced in [3], where the base layers are the shallow layers of the neural network that extracts high-level representations and the personalization layers are the deep layers for classifications.”
Towards Personalized Federated Learning (2021, background): “The ﬁrst is a “base layers + personalized layers” design proposed by [46].”
FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, background): “• The partial model personalization literature (Collins et al., 2021; Arivazhagan et al., 2019; Liang et al., 2020; Pillutla et al., 2022) is more concerned with proposing stable federated training algorithms that would work within heterogeneous settings, where clients have diverse data and…”
A Full Dive Into Realizing the Edge-Enabled Metaverse: Visions, Enabling Technologies, and Challenges (2022, methodology): “Fourth, since personal data is used to train the ML model, personalization can be built into FL to deliver tailored services to users [242].”
Federated Learning in Mobile Edge Networks: A Comprehensive Survey (2019, background): “Similarly, the authors in [73] borrow concepts from multitask learning to deal with the statistical heterogeneity in FL.”