Continual Multi-task Gaussian Processes

By Pablo Moreno-Muñoz, Antonio Artés-Rodríguez, Mauricio A Álvarez. 2019

14 citations recorded by the source corpus.

We address the problem of continual learning in multi-task Gaussian process (GP) models for handling sequential input-output observations. Our approach extends the existing prior-posterior recursion of online Bayesian inference, i.e.\ past posterior discoveries become future prior beliefs, to the infinite functional space setting of GP. For a reason of scalability, we introduce variational inference together with an sparse approximation based on inducing inputs. As a consequence, we obtain tractable continual lower-bounds where two novel Kullback-Leibler (KL) divergences intervene in a natural way. The key technical property of our method is the recursive reconstruction of conditional GP priors conditioned on the variational parameters learned so far. To achieve this goal, we introduce a novel factorization of past variational distributions, where the predictive GP equation propagates the posterior uncertainty forward. We then demonstrate that it is possible to derive GP models over many types of sequential observations, either discrete or continuous and amenable to stochastic optimization. The continual inference approach is also applicable to scenarios where potential multi-channel or heterogeneous observations might appear. Extensive experiments demonstrate that the method is fully scalable, shows a reliable performance and is robust to uncertainty error propagation over a plenty of synthetic and real-world datasets.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Probabilistic & Bayesian Methods, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

In Meta & Continual Learning

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, neutral): “An alternative approach that addresses this shortcoming is to regularize the predictive function directly (Benjamin et al., 2019; Bui et al., 2017; Buzzega et al., 2020; Jung et al., 2018; Kapoor et al., 2021; Kim et al., 2018; Li and Hoiem, 2018; Moreno-Muñoz et al., 2019; Pan et al., 2020; Titsias”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, background): “[19] adapt the one step recurrent approximate posterior (as in e.”
Optimal Continual Learning has Perfect Memory and is NP-hard (2020, background): “, 2019) as well as Bayesian and variationally Bayesian approaches (e.g. Nguyen et al., 2017; Tseran et al., 2018; Moreno-Munoz et al., 2019; Titsias et al., 2019).”