Manifold Gaussian Processes for regression

By Roberto Calandra, Jan Peters, Carl E. Rasmussen, Marc Peter Deisenroth. 2014

297 citations recorded by the source corpus.

Off-the-shelf Gaussian Process (GP) covariance functions encode smoothness assumptions on the structure of the function to be modeled. To model complex and non-differentiable functions, these smoothness assumptions are often too restrictive. One way to alleviate this limitation is to find a different representation of the data by introducing a feature space. This feature space is often learned in an unsupervised way, which might lead to data representations that are not useful for the overall regression task. In this paper, we propose Manifold Gaussian Processes, a novel supervised method that jointly learns a transformation of the data into a feature space and a GP regression from the feature space to observed space. The Manifold GP is a full GP and allows to learn data representations, which are useful for the overall regression task. As a proof-of-concept, we evaluate our approach on complex non-smooth functions where standard GPs perform poorly, such as step functions and robotics tasks with contacts.

Subfields: Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “…and Gales, 2018a;b; Sensoy et al., 2018; Shu et al., 2017), modifying the output layer (Bendale and Boult, 2016; Tagasovska and Lopez-Paz, 2019; Calandra et al., 2016; Macêdo et al., 2020), or computing a closed-form posterior for the last dense layer only (Riquelme et al., 2018; Snoek et…”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “…and transfer optimization (Krause & Ong, 2011; Swersky et al., 2013; Bar-denet et al., 2013) and the application to diverse tasks such as sensor set selection (Garnett et al., 2010), the tuning of adaptive Monte Carlo (Mahendran et al., 2012) and robotic gait control (Calandra et al., 2014b).”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “For example, in [34] each sigmoidal layer of an L layer neural network is defined as L`(x) := σ(W`x + B`) where σ is some sigmoidal non-linearity, and W` and B` are the layer parameters.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “imators might underﬁt. While Bayesian models such as GPs perform well in low-data regimes, they do not scale favorably with dimensionality and often use kernels ill-suited for discontinuous dynamics [Calandra et al., 2016], which is typical of robots interacting through contacts. Table 1: Model unce”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, background): “This problem is commonly addressed by deep kernel learning (DKL) where GPs are combined with modern neural networks to learn a useful kernel function from the data (Calandra et al., 2016; Wilson et al., 2016a).”
Neural Processes (2018, background): “Methods like (Calandra et al., 2016; Huang et al., 2015) remain fairly close to the GPs, but incor-Neural porate NNs to pre-process the input data.”
Deep Kernel Learning (2015, background): “All covariance (kernel) matrices implicitly depend on the kernel hyperparameters γ .”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “We only model parameter beliefs θ for the output layer that predicts the mean in our Bayesian neural networks (LázaroGredilla and Figueiras-Vidal, 2010; Calandra et al., 2014) as described in Section 2.”
Calibrating Deep Convolutional Gaussian Processes (2018, background): “Most of these works are based on ideas developed in the context of manifold GP s [2], where inputs are transformed using some parametric transformation of the input.”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “[9] Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Marc Peter Deisenroth.”
Stochastic Variational Deep Kernel Learning (2016, background): “Indeed many recent approaches have advocated building expressive kernel functions [e.g., 23, 10, 27, 26, 18, 32], and emerging research in this direction takes inspiration from deep learning models [e.g., 29, 6, 3].”
Hidden physics models: Machine learning of nonlinear partial differential equations (2017, background): “Furthermore, more complex function classes can be accommodated by employing nonlinear warping of the input space to capture discontinuities [49,50].”
Machine learning of linear differential equations using Gaussian processes (2017, background): “For example, non-stationary kernels employing nonlinear warpings of the input space can be constructed to capture discontinuous response [8].”
Noise Contrastive Priors for Functional Uncertainty (2018, methodology): “We apply NCP to posit epistemic uncertainty on the output of the mean μ, and we infer the induced weight posterior for only the output layer (Lázaro-Gredilla and Figueiras-Vidal, 2010; Calandra et al., 2014) that predicts the mean.”
On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation (2020, background): “Combining neural networks with Gaussian processes has also been suggested as a way to decouple representation and uncertainty [CPRD16, IG17].”
Convolutional Gaussian Processes (2017, methodology): “This is valid since applying a deterministic transformation to kernel inputs results in a valid kernel [see e.g. 1, 10], with the (many) parameters of the transformation becoming kernel hyperparameters.”
Deep Multi-fidelity Gaussian Processes (2016, background): “In particular, we are interested in Manifold Gaussian Processes [1] that are capable of capturing discontinuities.”
Deep Kernels for Optimizing Locomotion Controllers (2017, background): “To enhance kernel flexibility [10] suggests supervised learning of a feature transform during regression.”