Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

By Ruslan Salakhutdinov, Geoffrey E. Hinton. 2007

244 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, neutral): “, “transductive” models [14, 22, 21], graph-based methods [49, 4, 29], generative modeling [3, 27, 41, 9, 17, 23, 38, 34, 42], etc.”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “Recently, such basis functions have also been learned from data by training deep belief networks [71], deep neural networks [93], [144], or by factoring the empirical covariance matrix of historical data [72], [146].”
Deep Kernel Learning (2015, methodology): “We now consider the task of predicting the orientation of a face extracted from a grayscale image patch, explored in Salakhutdinov and Hinton (2008). We investigate our DKL procedure for efficiently learning meaningful representations from high-dimensional highlystructured image data.”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “Initially, Hinton and Salakhutdinov [26] trained a deep belief network in an unsupervised manner to work out a good feature space for which a GP could be placed on top of.”
Manifold Gaussian Processes for regression (2014, background): “rior mean function and a faithful representation of model uncertainty. At least two related approaches in the literature attempt joint supervised learning of features and regression/classication. In Salakhutdinov and Hinton (2007), pre-training of the input transformation makes use of computationall”
Gaussian Process Kernels for Pattern Discovery and Extrapolation (2013, background): “Detailed Gaussian process references include Rasmussen & Williams (2006), Stein (1999), and Cressie (1993).”
Avoiding pathologies in very deep networks (2014, background): “Salakhutdinov and Hinton (2008) used a deep neural network to learn feature transforms for kernels, which learn invariances in an unsupervised manner.”
Surpassing Human-Level Face Verification Performance on LFW with GaussianFace (2014, methodology): “used a deep belief net to learn a good covariance kernel for GPs [45].”
Learning Kernels with Random Features (2016, background): “In un-and semi-supervised settings, the goal is to learn an embedding of the input distribution followed by a simple classifier in the embedded space (e.g. [15]); the hope is that the input distribution carries the structure relevant to the task.”
Differentiable Compositional Kernel Learning for Gaussian Processes (2018, background): “There have been several attempts (Hinton & Salakhutdinov, 2008; Wilson et al., 2016) to combine neural networks with Gaussian processes.”
An Introduction to Deep Learning (2011, unknown): “This leads to learning very diﬀerent features (w.r.t RBM) in the intermediate layers and the network performs a trade-oﬀ between reconstruction loss and information content of the representation [21].”
Federated Semi-Supervised Learning with Inter-Client Consistency (2020, background): “While there exist numerous work on SSL, such as transductive models [18], graph-based approaches [19], and generative modeling [20], we mainly discuss consistency regularization and entropy-minimization approaches.”
Deep, Narrow Sigmoid Belief Networks Are Universal Approximators (2008, methodology): “The greedy algorithm has also been used to learn kernels that significantly improve the performance of gaussian processes on difficult tasks (compared to standard kernels) ( Salakhutdinov & Hinton, 2007b ) and was shown to”
The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training (2009, unknown): “…al., 2007; Ranzato et al., 2007; Larochelle, Erhan, Courville, Bergstra, & Bengio, 2007; Ranzato, Boureau, & LeCun, 2008), but also in regression (Salakhutdinov & Hinton, 2008), dimensionality reduction (Hinton & Salakhutdinov, 2006; Salakhutdinov & Hinton, 2007), natural language processing…”
Nonparametric Variational Auto-Encoders for Hierarchical Representation Learning (2017, methodology): “Gaussian processes [15] are another line of Bayesian nonparametric approach which has been incorporated with deep neural networks for expressive kernel learning [21, 6].”