Stochastic Variational Deep Kernel Learning

By Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing. 2016

301 citations recorded by the source corpus.

Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training. Specifically, we apply additive base kernels to subsets of output features from deep neural architectures, and jointly learn the parameters of the base kernels and deep network through a Gaussian process marginal likelihood objective. Within this framework, we derive an efficient form of stochastic variational inference which leverages local kernel interpolation, inducing points, and structure exploiting algebra. We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.

Subfields: Artificial Intelligence, Environmental Engineering, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Environmental Engineering

In Representation & Self-Supervised Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “, 2016) or structured Variational Inference (VI) ( (Bradshaw et al., 2017; Wilson et al., 2016)).”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “For the SV-DKL baseline, we experimented with a grid size of {64, 128, 256}.”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “Furthermore, our theoretical results show that, in some cases, an expensive Bayesian treatment over all layers of a network is not necessary (Theorem 2.4), thus validating the usage of last-layer Bayesian methods such as Snoek et al. (2015) and Wilson et al. (2016).”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “…perform three general sets of experiments: • Classiﬁcation: here we use these models for image classiﬁcation and in agreement with the general ﬁndings of Wilson et al. [61] ﬁnd that these models give good accuracy scores even when extended to work on far deeper base networks or in low data regimes.”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, neutral): “…works have exploited this DNN-GP correspondence implicitly or explicitly to build new models (Cho & Saul, 2009; Lawrence & Moore, 2007; Damianou & Lawrence, 2013; Wilson et al., 2016b;a; Bradshaw et al., 2017; van der Wilk et al., 2017; Kumar et al., 2018; Blomqvist et al., 2018; Borovykh, 2018).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “There is literature on learning the representation of a standard, usually structured, network composed with a Gaussian process (Wilson et al., 2016a,b; Al-Shedivat et al., 2017).”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “[42] Andrew G Wilson, Zhiting Hu, Ruslan R Salakhutdinov, and Eric P Xing.”
Variational Implicit Processes (2018, background): “A very recent work [26] also minimizes the sleep phase KL divergence (3) but wrt. the BNN prior, and their goal is to regularize BNN priors and implant some smoothness properties of GPs to BNNs.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “This connection was exploited implicitly or explicitly to build new models [15, 32, 16, 61, 62, 12, 58, 31, 10, 11, 20, 39, 34].”
On Calibration of Modern Neural Networks (2017, background): “For example, deep kernel learning (Wilson et al., 2016a;b; AlShedivat et al., 2016) combines deep neural networks with Gaussian processes on classification and regression problems.”
FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing (2020, methodology): “In [53, 54, 55, 56], a DNN architecture was combined with the automatic relevance determination (ARD) kernel to approximate any kernel function (including both the stationary and non-stationary ones).”
Bayesian GAN (2017, background): “It would also be interesting to consider the Bayesian GAN in conjunction with a non-parametric Bayesian deep learning framework, such as deep kernel learning (Wilson et al., 2016a,b).”
Convolutional Gaussian Processes (2017, methodology): “A previous approach by Wilson et al. [9] transforms the inputs to a kernel using a convolutional neural network.”
Physics-informed deep generative models (2018, unknown): “In this work, we revisit these approaches from a probabilistic standpoint [17, 18, 19] and develop an adversarial inference framework [20, 21, 22, 23] to enable the posterior characterization [24, 25] of the uncertainty associated with the model predictions.”
Learning Scalable Deep Kernels with Recurrent Structure (2016, methodology): “…delay between the kernel updates becomes too long) while the learning rate is high enough, optimization does not converge; at the same time, an appropriate combination of the learning rate and the mini-batch size leads better generalization than the default batch approach of Wilson et al. (2016a).”
Random Feature Expansions for Deep Gaussian Processes (2016, methodology): “Only very recently there have been attempts to demonstrate performance of GP models on such large data sets (Wilson et al., 2016; Krauth et al., 2017), and our proposal is on par with these latest GP methods.”
GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration (2018, methodology): “Although this paper primarily focuses on the regression setting, BBMM is fully compatible with variational techniques such as [22, 53], which are also supported in GPyTorch.”