Learning Scalable Deep Kernels with Recurrent Structure

By Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, Eric P. Xing. 2016

116 citations recorded by the source corpus.

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure, and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep Neural Networks as Gaussian Processes (2017, background): “(2014); Bui et al. (2016)), which can give rise to a richer class of probabilistic models beyond GPs. This contrasts with our work, where we study GPs that are in direct correspondence with deep, infinitely wide neural networks. Krauth et al. (2016) has recently explored the performance of GP models”
On Exact Computation with an Infinitely Wide Neural Net (2019, methodology): “Wilson et al. [2016], Al-Shedivat et al. [2016] used a neural network as a feature extractor, applied a GP on top of these features and then trained the resulting model end-to-end.”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “There is relevant literature here on learning the representation of a standard, usually structured, network composed with a Gaussian process (Wilson et al., 2016a,b; Al-Shedivat et al., 2017).”
Contextual Explanation Networks (2017, background): “…networks [8, 9, 10], representing potential functions [11] and energy functions [12] with neural networks, encoding learnable structure into Gaussian process kernels with deep and recurrent networks [13, 14], or learning state-space models on top of nonlinear embeddings of observables [15, 16, 17].”
FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing (2020, methodology): “tionary kernel. In [52], the authors modiﬁed the SM kernel to a linear multiple low-rank sub-kernels with a favorable optimization structure, which enables faster and more stable numerical search. In [53, 54, 55, 56], a DNN architecture was combined with the automatic relevance determination (ARD) k”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “For Gaussian processes, there have been significant developments in libraries (Rasmussen and Nickisch, 2010; GPy, 2012; Vanhatalo et al., 2013; Matthews et al., 2017; Al-Shedivat et al., 2017; Gardner et al., 2018).”
Variational Implicit Processes (2018, background): “A very recent work [26] also minimizes the sleep phase KL divergence (3) but wrt. the BNN prior, and their goal is to regularize BNN priors and implant some smoothness properties of GPs to BNNs.”
Learnable Embedding Space for Efficient Neural Architecture Compression (2019, background): “The follow-up work (Al-Shedivat et al., 2017) extends the kernel representation to recurrent networks to model sequential data.”
Gaussian Process Regression for In Situ Capacity Estimation of Lithium-Ion Batteries (2017, methodology): “correlations between the inputs by encoding recurrent behavior into the kernel of the GP function (such as in the method presented in [40]) in order to achieve the benefits of both of these”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “20 A simple representative is to combine the structural NNs and the flexible nonparametric GP together, in which NNs map the original input space to the feature space for extracting nonstationary/recurrent features, and the last-layer sparse GP conducts standard regression over the latent space [34]”