Gaussian Process Regression Networks

By Andrew Gordon Wilson, David A. Knowles, Zoubin Ghahramani. 2011

209 citations recorded by the source corpus.

We introduce a new regression framework, Gaussian process regression networks (GPRN), which combines the structural properties of Bayesian neural networks with the nonparametric exibility of Gaussian processes. GPRN accommodates input (predictor) dependent signal and noise correlations between multiple output (response) variables, input dependent length-scales and amplitudes, and heavy-tailed predictive distributions. We derive both elliptical slice sampling and variational Bayes inference procedures for GPRN. We apply GPRN as a multiple output regression and multivariate volatility model, demonstrating substantially improved performance over eight popular multiple output (multi-task) Gaussian process models and three multivariate volatility models on real datasets, including a 1000 dimensional gene expression dataset.

Subfields: Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep Gaussian Processes (2012, background): “s, for example by considering sophisticated covariance functions [Durrande et al., 2011, Gonen and Alpaydin, 2011] or by embedding GPs in more¨ complex probabilistic structures [Snelson et al., 2004, Wilson et al., 2012] able to learn more powerful representations of the data. However, all GP-based ”
Neural Processes (2018, background): “Approaches that are more similar to NNs include for example neural networks whose weights are sampled using a GPs (Wilson et al., 2011) or networks where each unit represents a different kernel (Sun et al., 2018).”
Deep Kernel Learning (2015, methodology): “The Gaussian process regression network (Wilson et al., 2012) replaces all weight connections in a Bayesian neural network with Gaussian processes, allowing the authors to model input dependent correlations between multiple tasks.”
Stochastic Variational Deep Kernel Learning (2016, background): “Indeed many recent approaches have advocated building expressive kernel functions [e.g., 23, 10, 27, 26, 18, 32], and emerging research in this direction takes inspiration from deep learning models [e.g., 29, 6, 3].”
Scalable Variational Gaussian Process Classification (2014, methodology): “Furthermore, a variational approach would allow for integration of the approximation within other GP models such as GP regression networks [Wilson et al., 2012], latent variable models [Lawrence, 2004] and deep GPs [Damianou and Lawrence, 2013], as the variational objective could be optimized as…”
Avoiding pathologies in very deep networks (2014, background): “Wilson et al. [2012] introduced Gaussian process regression networks, which are defined as a matrix product of draws from GPs priors, rather than a composition.”
Remarks on multi-output Gaussian process regression (2018, methodology): “There are some other extensions of typical LMC, for example, the nonstationary MOGP by using a spatially varying correlation matrix [50, 51], the combination of the LMC and the Bayesian treed Gaussian process [52, 53], and the self-measuring MTGP that uses the information of outputs to construct inf”
Scalable Inference for Gaussian Process Models with Black-Box Likelihoods (2015, methodology): “Finally, we use SAVIGP to do inference for the Gaussian process regression network model [14] on the SARCOS dataset concerning an inverse robot dynamics problem [16].”
It is all in the noise: Efficient multi-task Gaussian process inference with structured residuals (2013, methodology): “In contrast to existing work extending Gaussian process multi task models by deﬁning more complex covariance structures [2, 9, 8], our model utilizes the gradient of the marginal likelihood for parameter estimation and does not require expected maximization, variational approximation or MCMC…”
Gaussian Process Covariance Kernels for Pattern Discovery and Extrapolation (2013, methodology): “Expressive kernels have been developed by combining Gaussian processes in a type of Bayesian neural network structure (Salakhutdinov and Hinton, 2008; Wilson et al., 2012; Damianou and Lawrence, 2012).”