Deep Gaussian Processes for Regression using Approximate Expectation Propagation

By Thang D. Bui, Daniel Hernández-Lobato, José Miguel Hernández-Lobato, Yingzhen Li, Richard E. Turner. 2016

249 citations recorded by the source corpus.

Deep Gaussian processes (DGPs) are multilayer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.

Subfields: Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Continual Learning (2017, methodology): “In this paper the online VI approach is used as it typically outperforms the other methods for complex models in the static setting (Bui et al., 2016) and yet it has not been applied to continual learning of neural networks.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “While we primarily focus on image classification, we show that SWAG can significantly improve test perplexities of LSTM networks on language modeling problems, and in Appendix 7 we also compare SWAG with Probabilistic Back-propagation (PBP) [16], Deterministic Variational Inference (DVI) [47], and D”
Improving and Understanding Variational Continual Learning (2019, background): “Variational Inference (VI) is known to provide better uncertainty estimates that other approaches in this setting [2].”
Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, methodology): “As a comparison we also include test performances of a BNN with a Gaussian approximation (VI-G) (Li & Turner, 2016), a BNN with HMC, and a sparse Gaussian process model with 50 inducing points (Bui et al., 2016).”
Concrete Dropout (2017, methodology): “We compare against a two layer Bayesian NN approximated by standard dropout [9] and a Deep Gaussian Process of depth 2 [4].”
Deep Neural Networks as Gaussian Processes (2017, background): “(2014); Bui et al. (2016)), which can give rise to a richer class of probabilistic models beyond GPs. This contrasts with our work, where we study GPs which are in direct correspondence with deep, infinitely wide neural networks. Gal & Ghahramani (2016) connects dropout in deep neural networks with ”
Streaming Sparse Gaussian Process Approximations (2017, background): “Regression models that employ Gaussian processes are state of the art for many datasets [11].”
Benchmarking the Neural Linear Model for Regression (2019, methodology): “As in Bui et al. (2016), we additionally follow the procedure for the Friedman test as described in Demšar (2006), generating the plots shown in Figures 5 and 6.”
Overpruning in Variational Bayesian Neural Networks (2018, methodology): “[3] As we see in Figure 1, WN most consistently has good performance as compared to the other variational approximations tested.”
Evaluating Bayesian Deep Learning Methods for Semantic Segmentation (2018, methodology): “tationally light-weight, scalable methods of performing approximate Bayesian inference in deep neural networks has enabled these models to estimate their uncertainty in addition to making predictions [21,22,38,6,48,8]. However, as we do not have ground truth uncertainties, we cannot use the conventi”
Uncertainty in Deep Learning (2016, result): “The erroneous results are the ones used in [Bui et al., 2016].”
Physics-informed deep generative models (2018, unknown): “Hence, the ability to quantify predictive uncertainty and learn in a sample-efficient manner is a necessity [5, 6, 7, 8], especially in data-limited domains [9].”
Fixing Variational Bayes: Deterministic Variational Inference for Bayesian Neural Networks (2018, unknown): “We report test log-likelihoods at convergence and find that our method is competitive or superior to a range of state-of-the-art techniques (reproduced from Bui et al. (2016)).”
Variational Implicit Processes (2018, methodology): “Previous studies expert-designed [11, 37, 71] or learned ﬁnger-print features [25] that transforms the raw string data into ﬁxed-size feature for prediction.”
Conditional Density Estimation with Bayesian Normalising Flows (2018, background): “Additionally, they yield state of the art performance on two of these datasets, ‘energy’ and ‘yacht’ (Hern ´ andez-Lobato & Adams, 2015; Bui et al., 2016; Li & Gal, 2017; Louizos & Welling, 2016).”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, methodology): “While we could use any kernel, we choose the RBF kernel with a lengthscale for each dimension for direct comparison with Bui et al. (2016).”