Deep Gaussian Processes

By Andreas C. Damianou, Neil D. Lawrence. 2012

1,327 citations recorded by the source corpus.

In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically applied to relatively large data sets using stochastic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model selection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Time-series forecasting with deep learning: a survey (2020, background): “In addition, Gaussian processes [21] have been extensively used for time-series prediction—with recent extensions including deep Gaussian processes [22], and parallels in deep learning via neural processes [23].”
Deep Factors for Forecasting (2019, background): “Deep Gaussian Processes (Damianou & N.D., 2013) have attracted growing interests in recent years.”
Conditional Neural Processes (2018, background): “Scaling issues with GPs have been addressed by sparse GPs (Snelson & Ghahramani, 2006), while overcoming the limited expressivity resulting from functional restrictions is the motivation for Deep GPs (Damianou & Lawrence, 2013; Salimbeni & Deisenroth, 2017).”
Uncertainty in Multitask Transfer Learning (2018, background): “More recent tools such as deep Gaussian processes [7] show great potential and yet their scalability whilst learning from multiple tasks needs to be improved.”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, background): “Damianou & Lawrence (2013) stacked multiple GPs to create a hierarchy of GPs. Nguyen et al. (2019) proposed a hierarchical model using a mixture of GPs to learn global and local information.”
Neural Processes (2018, background): “Deep GPs have some conceptual similarity to NNs as they stack GPs to obtain deep models (Damianou & Lawrence, 2013).”
Deep Kernel Learning (2015, background): “Alternatively, Damianou and Lawrence (2013) replace every activation function in a Bayesian neural network with a Gaussian process transformation, in an unsupervised setting.”
Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference (2015, methodology): “The deep Gaussian process [11] is a powerful tool in statistics that allows us to model distributions over functions.”
Calibrating Deep Convolutional Gaussian Processes (2018, methodology): “Convolutional Networks with Random-Feature-Expanded Deep GP s: A DGP model represents a deep probabilistic nonparametric approach where the output of one GP at each layer is used as the input to the GP in the next layer [5].”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “People have built more expressive functions by automatically combining kernels together [14] as well as by composing GPs together [13, 11, 15].”
Stochastic Variational Deep Kernel Learning (2016, background): “Indeed many recent approaches have advocated building expressive kernel functions [e.g., 23, 10, 27, 26, 18, 32], and emerging research in this direction takes inspiration from deep learning models [e.g., 29, 6, 3].”
Manifold Gaussian Processes for regression (2014, positive): “Inspired by deep neural networks, deep GPs stack multiple layers of GP latent variable models (Damianou and Lawrence, 2013).”
Deep Neural Networks as Gaussian Processes (2017, background): “Moreover, a body of work on other constructions of GP analogs for multilayer networks can be found in [9, 3, 8, 5, 1, 17].”
On Exact Computation with an Infinitely Wide Neural Net (2019, methodology): “ure extractor, applied a GP on top of these features and then trained the resulting model end-to-end. Another line of work built probabilistic graphical models (PGMs) which uses kernel as a component[Damianou and Lawrence, 2013,Lawrence and Moore,2007,Van der Wilk et al., 2017,Kumar et al., 2018, Bl”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “(2018) requires φ to be linearly bounded in norm; Daniely et al. (2016) requires φ be twicedifferentiable with |φ|, |φ′|, |φ′′| all bounded, or that φ = ReLU; and a sufficient condition given in Novak et al.”
Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors (2016, background): “Despite the fact that the matrix variate Gaussian is a simple generalization of the multivariate case it provides us a straightforward way to separate the correlations among the rows and columns of the matrix, which implicitly affects the correlations among the input and output hidden units.”
Scalable Variational Gaussian Process Classification (2014, methodology): “The computational cost of inference in GPs is O ( N 3 ) in general, where N is the number of data.”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “An area that is less related than might be expected is that of ‘Deep Gaussian Processes’ (DGPs) (Damianou and Lawrence, 2013).”