Unsupervised Models of Images by Spikeand-Slab RBMs

By Aaron C. Courville, James Bergstra, Yoshua Bengio. 2011

81 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Pylearn2: a machine learning research library (2013, methodology): “Pylearn2 has implementations of several models that were de v loped at LISA, including denoising auto-encoders (DAEs) [53], contractive auto-encoders (CAEs) [42] including higher-order CAEs [43], spike-and-slab RBMs (ssRBMs) [11] including ssR BMs with pooling [12], reconstruction sampling autoencod”
Representation Learning: A Review and New Perspectives (2012, methodology): “jh ! (27) where Cov xjh= P dh i=1 1 i h W T 1 , the last equality holds only if the covariance matrix Cov xjhis positive deﬁnite. Strategies for ensuring positive deﬁnite Cov xjh are discussed in Courville et al. (2011b). Like the mcRBM and the mPoT model, the ssRBM gives rise to a fully general m”
Disentangling Factors of Variation via Generative Entangling (2012, methodology): “The bilinear ss-RBM model on the other hand generates near-perfect samples (not shown), while factoring the representation for use in deeper layers.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “Some researchers also advocate the use of different learning rates for the different types of parameters one finds in the model, such as biases and weights in the standard multi-layer network, but the issue becomes more important when parameters such as precision or variance are included in the lot ”
Discrete Variational Autoencoders (2016, methodology): “Except in Figure 6, RBMs have 128 units ( 64 units per side, with full bipartite connections between the two sides), with 4 layers of hierarchy in the approximating posterior.”
Deep Learning of Representations: Looking Forward (2013, background): “eper levels of representation. 17 e.g. they can be charted with a few coordinates 18 See examples of generated images with some of the current state-of-the-art in learned generative models of images (Courville et al., 2011; Luo et al., 2013) Deep Learning of Representations: Looking Forward 15 This ”
Learning with Hierarchical-Deep Models (2013, background): “Index Terms—Deep networks, deep Boltzmann machines, hierarchical Bayesian models, one-shot learning Ç”
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding (2012, methodology): “ i; (2) The ssRBM model family is a good starting point for S3C because it has demonstrated both reasonable performance as a feature discovery scheme and remarkable performance as a generative model (Courville et al., 2011a). Within the ssRBM family, S3C’s closest relative is a variant of the -ssRBM”
Big Neural Networks Waste Capacity (2013, neutral): “These results have since been extended to other types of auto-encoders and RBMs (Rifai et al., 2011; Courville et al., 2011).”
Training Restricted Boltzmann Machines (2015, methodology): “If the training data consists of images, this can for example be used to generate textures from these images (Le Roux et al., 2011; Courville et al., 2011; Kivinen and Williams, 2012), or to solve inpainting tasks (Kivinen and Williams, 2012; Tang et al.”
Scaling Up Spike-and-Slab Models for Unsupervised Feature Learning (2013, methodology): “Previous object recognition work is from (Coates and Ng, 2011; Courville et al., 2011b).”
Bidirectional Helmholtz Machines (2015, methodology): “If the training data consists of, for example, images, it can be used to generate textures from these images, or to solve inpainting tasks by sampling the missing or deteriorated parts of a given image from the distribution [16, 50, 62, 104].”
Texture Modeling with Convolutional Spike-and-Slab RBMs and Deep Extensions (2012, methodology): “ associated with both an element h i of the binary vector and an element s i of the real-valued variable. In this work we will concern ourselves with the ssRBM formulation referred to as the -ssRBM (Courville et al., 2011b) with the associated energy function: E(v;s;h) = P N i=1 v T W is ih i + 1 2 ”
Training restricted Boltzmann machines: An introduction (2014, methodology): “Restricted Boltzmann Machines (RBMs) are probabilistic graphical models corresponding to stochastic neural networks (Smolensky, 1986; Hinton, 2002) (see the article by Fischer and Igel (2014) for a recent review).”
Winner-Take-All Autoencoders (2014, background): “ similarity metric on the feature maps, a ∼ 3% gain in classiﬁcation rate is achievable. Accuracy 3-Way Factored RBM (3 layers) [21] 65.3% Mean-covariance RBM (3 layers) [22] 71.0% Spike-and-Slab RBM [23] 76.7% Spike-and-Slab Sparse Coding [19] 78.3% Convolutional Deep Belief Net (2 layers) [18] (ﬁn”
Disentangling Factors of Variation for Facial Expression Recognition (2012, methodology): “Whereas in the usual applications of convolutions for convolutional networks a single common receptive field (convolutional kernel size) is used, we consider multiple sizes, allowing the model to capture structure at different scales (see also [17] for a similar approach, with two scales).”