A Spike and Slab Restricted Boltzmann Machine

By Aaron C. Courville, James Bergstra, Yoshua Bengio. 2011

101 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Pylearn2: a machine learning research library (2013, neutral): “…that were developed at LISA, including denoising auto-encoders (DAEs) [53], contractive auto-encoders (CAEs) [42] including higher-order CAEs [43], spike-and-slab RBMs (ssRBMs) [11] including ssRBMs with pooling [12], reconstruction sampling autoencoders [13], and deep sparse rectiﬁer nets [19].”
Neural Discrete Representation Learning (2017, background): “…with continuous features have been the focus of many previous work [16, 39, 6, 9] however we concentrate on discrete representations [27, 33, 8, 28] which are potentially a more natural ﬁt for many of the modalities we are interested in. Language is inherently discrete, similarly speech is…”
Representation Learning: A Review and New Perspectives (2012, methodology): “Another recently introduced RBM-based model with the objective of having the hidden units encode both the mean and covariance information is the spike-and-slab Restricted Boltzmann Machine (ssRBM) (Courville et al., 2011a,b).”
Disentangling Factors of Variation via Generative Entangling (2012, methodology): “Similar to the standard spike-and-slab restricted Boltzmann machine (Courville et al., 2011b;a), the energy function in Eq.”
Learning to Relate Images (2011, methodology): “Like an auto-encoder, this model can be trained with gradient based optimization.”
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding (2012, methodology): “Examples of shallow models applied to feature discovery include sparse coding (Raina et al., 2007), restricted Boltzmann machines (RBMs) (Hinton et al., 2006; Courville et al., 2011b), various autoencoderbased models (Bengio et al., 2007), and hybrids of autoencoders and sparse coding (Kavukcuoglu e”
Multiresolution Deep Belief Networks (2012, methodology): “They include the mean and covariance RBM [20], Relu RBM [21], Spike and Slab RBM [22], and have been shown to be superior first layer models.”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, methodology): “For instance, recently proposed variants of RBMs such as spike-and-slab RBMs [Courville et al., 2011] could directly be applied.”
Texture Modeling with Convolutional Spike-and-Slab RBMs and Deep Extensions (2012, methodology): “In this work we will concern ourselves with the ssRBM formulation referred to as the µ-ssRBM (Courville et al., 2011b) with the associated energy function: E(v, s, h) = −∑Ni=1 vTWisihi + 12vT (Λ +∑Ni=1 Φihi) v + 1 2 N∑ i=1 αis 2 i − N∑ i=1 αiµisihi − N∑ i=1 bihi + 1 2 N∑ i=1 αiµ 2 ihi, (1) where Wi…”
Hierarchical Convolutional Deep Learning in Computer Vision (2013, methodology): “Several approaches have been proposed to induce this sparsity such as sparsifying logistic activations [100], a target sparsity penalty for each activation [125], or separating out activations into spike and slab units [25].”
Connectionist multivariate density-estimation and its application to speech synthesis (2016, background): “The spike-and-slab RBM (Courville et al., 2011) introduces a set of continuous-valued latent variables associated to each binary latent variable.”
Unsupervised Models of Images by Spikeand-Slab RBMs (2011, methodology): “In the original presentation of the ssRBM in Courville et al. (2011), the possibility of the nonpositive-definiteness of the conditional covariance of v given h was dealt with by limiting the support over the domain of v (i.e., R D ) to a large but finite ball that encompasses all the training data.”
A RAD approach to deep mixture models (2019, methodology): “In order to efﬁciently parametrize the model, mixture models are often formulated as a discrete latent variable models (Hinton & Salakhutdinov, 2006; Courville et al., 2011; Mnih & Gregor, 2014; van Figure 1: Stochastic computational graphs for inference and sampling for ﬂow based models (1a, 1b)…”
Building a better probabilistic model of images by factorization (2011, methodology): “by the mcRBM has motivated the exploration of similar models [6], and for these reasons it is a particularly relevant comparison.”
Tensor Switching Networks (2016, background): “With respect to improving the nonlinearities, the idea of severing activation and analysis weights (or having multiple sets of weights) in each hidden layer has been studied in [6, 7, 8].”
Neuronal Synchrony in Complex-Valued Deep Networks (2013, background): “2), if desired, and to allow the units to switch off (rather than being constrained to the unit circle), e.g. with a ‘spike and slab’ formulation (Courville et al., 2011; Kivinen & Williams, 2011).”