Partitioned Variational Inference: A unified framework encompassing federated and continual learning

By Thang D. Bui, Cuong V Nguyen, Siddharth Swaroop, Richard E. Turner. 2018

58 citations recorded by the source corpus.

Variational inference (VI) has become the method of choice for fitting many modern probabilistic models. However, practitioners are faced with a fragmented literature that offers a bewildering array of algorithmic options. First, the variational family. Second, the granularity of the updates e.g. whether the updates are local to each data point and employ message passing or global. Third, the method of optimization (bespoke or blackbox, closed-form or stochastic updates, etc.). This paper presents a new framework, termed Partitioned Variational Inference (PVI), that explicitly acknowledges these algorithmic dimensions of VI, unifies disparate literature, and provides guidance on usage. Crucially, the proposed PVI framework allows us to identify new ways of performing VI that are ideally suited to challenging learning scenarios including federated learning (where distributed computing is leveraged to process non-centralized data) and continual learning (where new data and tasks arrive over time and must be accommodated quickly). We showcase these new capabilities by developing communication-efficient federated training of Bayesian neural networks and continual learning for Gaussian process models with private pseudo-points. The new methods significantly outperform the state-of-the-art, whilst being almost as straightforward to implement as standard VI.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “Improved training procedures have also been developed for this type of approximate Bayesian continual learning through the use of natural gradients [10, 47], fixed-point updates [52], and local approximation [6].”
Variational Federated Multi-Task Learning (2019, methodology): “to be effective and to outperform other methods when applied in the continual learning (CL) setting [8, 32].”
TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation (2023, background): “To address these issues, recent research [36, 11, 3, 19] has introduced the concept of Continual Learning (CL) [61, 42, 51, 49, 48] within the FL framework.”
Federated Learning for Healthcare Informatics (2019, background): “[11] improved federated learning for Bayesian neural networks using partitioned variational inference, where the client can decide to upload the parameters back to the central server after multiple passes through its data, after one local epoch, or after just one mini-batch.”