Variational Federated Multi-Task Learning

By Luca Corinzia, Joachim M. Buhmann. 2019

186 citations recorded by the source corpus.

In classical federated learning a central server coordinates the training of a single model on a massively distributed network of devices. This setting can be naturally extended to a multi-task learning framework, to handle real-world federated datasets that typically show strong non-IID data distributions among devices. Even though federated multi-task learning has been shown to be an effective paradigm for real world datasets, it has been applied only to convex models. In this work we introduce VIRTUAL, an algorithm for federated multi-task learning with non-convex models. In VIRTUAL the federated network of the server and the clients is treated as a star-shaped Bayesian network, and learning is performed on the network using approximated variational inference. We show that this method is effective on real-world federated datasets, outperforming the current state-of-the-art for federated learning.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Another approach [26] models the star topology as a Bayesian network and performs variational inference during learning.”
FedMD: Heterogenous Federated Learning via Model Distillation (2019, methodology): “Approaches based on frameworks such as Bayesian [11], meta-learning [12] and transfer learning [14] also achieve good performance on non-i.i.d. data while allowing a certain amount of model customization.”
Federated Learning of a Mixture of Global and Local Models (2020, methodology): “Approaches based on variational inference (Corinzia & Buhmann, 2019), cyclic patterns in practical FL data sampling (Eichner et al., 2019) and transfer learning (Zhao et al., 2018) have been proposed.”
Towards Personalized Federated Learning (2021, methodology): “This motivated [60] to propose the VIRTUAL federated MTL algorithm that performs variational inference using a Bayesian approach.”
Federated Learning for Healthcare Informatics (2019, methodology): “[22] introduced VIRTUAL, an algorithm for federated multi-task learning with non-convex models.”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “Corinzia and Buhmann [43] propose a federated MTL method with non-convex models.”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “To ensure that the FMTL algorithm can be applied to general non-convex models, VIRTUAL [36] considers the federation of a central server and clients as a Bayesian network and employs approximated variational inference for training.”
Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge Based Framework (2020, background): “Along a different line, federated multi-task learning aims at learning distinct tasks for different devices simultaneously and tries to capture the model relationships amongst them without privacy risk [22].”
Federated Learning on Non-IID Data: A Survey (2021, methodology): “Their method achieves satisfactory results on several Non-IID datasets, but has difficulties in converging when there is a large number of clients, due to the sequential fine-tuning [21].”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Corinzia and Buhmann [171] modeled a Bayesian network by performing variational inference during learning.”
Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints (2019, background): “[26] model the connectivity structure between clients and server as a Bayesian network and perform variational inference during learning.”
Lower Bounds and Optimal Algorithms for Personalized Federated Learning (2020, background): “There are multiple strategies in the literature for incorporating e personalization into FL: multi-task learning [49, 48, 14], transfer learning [54, 23], variational inference [8], mixing of local and global models [41, 19, 31, 10] and others [13].”
Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective (2020, background): “…of research effort has been devoted to tackling various challenges in this new distributed machine learning framework, including developing new optimization and model aggregation algorithms [5]–[7], handling non-i.i.d. and unbalanced datasets [8]–[10], and preserving model privacy [11]–[15] etc.”
Federated Multi-Task Learning under a Mixture of Distributions (2021, unknown): “approach is to jointly train a global model and one local model per client and then let each client build a personalized model by interpolating them [14, 9, 44].”