Approximate Inference Turns Deep Networks into Gaussian Processes
By Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, Maciej Jan Korzepa. 2019
141 citations recorded by the source corpus.
Deep neural networks (DNN) and Gaussian processes (GP) are two powerful models with several theoretical connections relating them, but the relationship between their training methods is not well understood. In this paper, we show that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors. This enables us to relate solutions and iterations of a deep-learning algorithm to GP inference. As a result, we can obtain a GP kernel and a nonlinear feature map while training a DNN. Surprisingly, the resulting kernel is the neural tangent kernel. We show kernels obtained on real datasets and demonstrate the use of the GP marginal likelihood to tune hyperparameters of DNNs. Our work aims to facilitate further research on combining DNNs and GPs in practical settings.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Richard E. Turner
- Erik A. Daxberger
- Siddharth Swaroop
- Matthias Bauer
- Alexander Immer
- Dustin Tran
- Runa Eschenhagen
- Balaji Lakshminarayanan
In Representation & Self-Supervised Learning
- Siddharth Swaroop
- Runa Eschenhagen
- Mohammad Emtiyaz Khan
- Alexander Immer
- Richard E. Turner
- Matthias Bauer
- Richard Bamler
- Mohsin Ali
In Probabilistic & Bayesian Methods
- Siddharth Swaroop
- Richard E. Turner
- Balaji Lakshminarayanan
- Matthias Bauer
- Erik A. Daxberger
- Alexander Immer
- Dustin Tran
- Runa Eschenhagen
Citations in the corpus
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “Laplace approximation and GP inference with DNN Laplace approximation has a long history in GP and NN literature [73, 17, 61, 49, 63], and the theoretical connection between a Laplaceapproximated DNN and GP has being explored recently [40].”
- Continual Deep Learning by Functional Regularisation of Memorable Past (2020, methodology): “Khan et al. (2019) generalise this to iterations of optimisers.”
- Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “It can also be inferred entirely in function space as a Gaussian process [25, 26].”
- A survey of uncertainty in deep neural networks (2021, background): “filtering [129] etc, (ii) probabilistic programming to exploit modern Graphical Processing Units (GPUs) [130], [131], [132], [133], (iii) different types of priors [134], [135], (iv) advancements in theoretical understandings of BNNs [136], [121], [137], (iv) uncertainty propagation techniques to sp”