Deep Kernel Learning

By Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing. 2015

1,081 citations recorded by the source corpus.

We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods. Specifically, we transform the inputs of a spectral mixture base kernel with a deep architecture, using local kernel interpolation, inducing points, and structure exploiting (Kronecker and Toeplitz) algebra for a scalable kernel representation. These closed-form kernels can be used as drop-in replacements for standard kernels, with benefits in expressive power and scalability. We jointly learn the properties of these kernels through the marginal likelihood of a Gaussian process. Inference and learning cost $O(n)$ for $n$ training points, and predictions cost $O(1)$ per test point. On a large and diverse collection of applications, including a dataset with 2 million examples, we show improved performance over scalable Gaussian processes with flexible kernel learning models, and stand-alone deep architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Evidential Deep Learning to Quantify Classification Uncertainty (2018, background): “Their power in prediction has been demonstrated in different contexts such as transfer learning [15] and deep learning [32].”
A Survey on Neural Architecture Search (2019, methodology): “parameters of the initial architecture init. Cao et al. (2019) Cao et al. (2019) maximize Equation (25) by using Bayesian optimization (Section 3.3) with a Gaussian process built using a deep kernel (Wilson et al., 2016). The acquisition function is maximized by random exploration of the search spac”
Conditional Neural Processes (2018, methodology): “In order to maintain this property in a trained model, one approach is to train the model to predict a GP kernel (Wilson et al., 2016).”
Kernel Continual Learning (2021, background): “Recently, they have also been demonstrated to be strong learners in tandem with deep neural networks (Wilson et al., 2016a;b; Tossou et al., 2019), especially when learning from limited data (Zhen et al., 2020; Patacchiola et al., 2020).”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “For the SV-DKL baseline, we experimented with a grid size of {64, 128, 256}.”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, methodology): “m selection of observations. To enable our functional regularisation approach to deal with high-dimensional and complex datasets, we use a linear kernel with features parameterised by neural networks [38]. Such GPs can be understood as Bayesian neural networks, where only the weights of the last lay”
Neural Processes (2018, background): “There are two models on this spectrum that are closely re-lated to NPs: matching networks (MN, Vinyals et al. (2016)) and deep kernel learning (DKL, Wilson et al. (2016)).”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “Furthermore, our theoretical results show that, in some cases, an expensive Bayesian treatment over all layers of a network is not necessary (Theorem 2.4), thus validating the usage of last-layer Bayesian methods such as Snoek et al. (2015) and Wilson et al. (2016).”
Stochastic Variational Deep Kernel Learning (2016, background): “[30] proposed simple and scalable deep kernels for single-output regression problems, with promising performance on many experiments.”
Google Vizier: A Service for Black-Box Optimization (2017, background): “While some authors recommend using Bayesian deep learning models in lieu of Gaussian processes for scalability [27, 31], in our experience they are too sensitive to their own hyperparameters and do not reliably perform well.”
Deep Neural Networks as Gaussian Processes (2017, background): “Moreover, a body of work on other constructions of GP analogs for multilayer networks can be found in [9, 3, 8, 5, 1, 17].”
MMD GAN: Towards Deeper Understanding of Moment Matching Network (2017, background): “A more general kernel is studied in [19].”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, neutral): “…works have exploited this DNN-GP correspondence implicitly or explicitly to build new models (Cho & Saul, 2009; Lawrence & Moore, 2007; Damianou & Lawrence, 2013; Wilson et al., 2016b;a; Bradshaw et al., 2017; van der Wilk et al., 2017; Kumar et al., 2018; Blomqvist et al., 2018; Borovykh, 2018).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “There is literature on learning the representation of a standard, usually structured, network composed with a Gaussian process (Wilson et al., 2016a,b; Al-Shedivat et al., 2017).”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “[43] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing.”
Learning Deep Kernels for Non-Parametric Two-Sample Tests (2020, methodology): “To model these complex functions, we adopt a deep kernel approach (Wilson et al., 2016; Sutherland et al., 2017; Li et al., 2017; Jean et al., 2018; Li et al., 2019), building a kernel with a deep network.”
Nonparametric Variational Auto-Encoders for Hierarchical Representation Learning (2017, methodology): “Gaussian processes [15] are another line of Bayesian nonparametric approach which has been incorporated with deep neural networks for expressive kernel learning [21, 6].”