Streaming Sparse Gaussian Process Approximations

By Thang D. Bui, Cuong V Nguyen, Richard E. Turner. 2017

125 citations recorded by the source corpus.

Sparse pseudo-point approximations for Gaussian process (GP) models provide a suite of methods that support deployment of GPs in the large data regime and enable analytic intractabilities to be sidestepped. However, the field lacks a principled method to handle streaming data in which both the posterior distribution over function values and the hyperparameter estimates are updated in an online fashion. The small number of existing approaches either use suboptimal hand-crafted heuristics for hyperparameter learning, or suffer from catastrophic forgetting or slow updating when new data arrive. This paper develops a new principled framework for deploying Gaussian process probabilistic models in the streaming setting, providing methods for learning hyperparameters and optimising pseudo-input locations. The proposed framework is assessed using synthetic and real-world datasets.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Continual Learning (2017, background): “On the other hand, the online variational inference has been previously explored (Ghahramani and Attias, 2000; Broderick et al., 2013; Bui et al., 2017), but not for neural networks or in the context of complex tasks.”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “An alternative approach that addresses this shortcoming is to regularize the predictive function directly (Benjamin et al., 2019; Bui et al., 2017; Buzzega et al., 2020; Jung et al., 2018; Kapoor et al., 2021; Kim et al., 2018; Li and Hoiem, 2018; Moreno-Mu ˜ noz et al., 2019; Pan et al., 2020;…”
Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Sparse Gaussian processes also consider the selection of inducing points (Seeger et al., 2003; Csato & Opper, 2002; Bui et al., 2017).”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “Alternatively, the effectiveness of Gaussian Process (GP) priors is demonstrated by Csató & Opper (2002) for online regression, and by Bui et al. (2017a) for streaming data.”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, methodology): “…on sparse GP methods (Csato & Opper, 2002; Lawrence et al., 2002; Seeger et al., 2003; Quiñonero-Candela & Rasmussen, 2005; Snelson & Ghahramani, 2006; Titsias, 2009; Hensman et al., 2013; Bui et al., 2017b), which summarise posterior distributions using small numbers of inducing points.”
Continual Multi-task Gaussian Processes (2019, methodology): “Importantly, we condition every observed output y n to the inﬁnite-dimensional function space f similarly to Bui et al. (2017b), having p ( y n | f ) instead.”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, methodology): “…for various models and applications [Broderick et al., 2013, Li and Hoiem, 2016, Kirkpatrick et al., 2017, Zenke et al., 2017, Seff et al., 2017, Bui et al., 2017a, Nguyen et al., 2018, Zeno et al., 2018, Chaudhry et al., 2018], to setting up relevant metrics and benchmarks for evaluation…”
A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (2016, background): “This distinction is important as it means, for example, that we are free to add pseudo-data as more structure of the underlying function is revealed, without altering the modelling assumptions (e.g. see Bui et al. (2017) for an example in online inference).”
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation (2016, background): “ This distinction is important as it means, for example, that we are free to add pseudo-data as more structure is revealed the underlying function without altering the modelling assumptions (e.g. see Bui et al. (2017) for an example in online inference). Moreover, since the pseudo-input locations ar”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “, an evolving time-series process [216].”