Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

By Honglak Lee, Roger Baker Grosse, Rajesh Ranganath, Andrew Y. Ng. 2009

2,705 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

How transferable are features in deep neural networks? (2014, background): “This phenomenon occurs not only for different datasets, but even with very different training objectives, including supervised image classification (Krizhevsky et al., 2012), unsupervised density learning (Lee et al., 2009), and unsupervised learning of sparse representations (Le et al., 2011).”
Deeply-Supervised Nets (2014, background): “100, and SVHN). 1 Introduction Much attention has been given to a resurgence of neural networks, deep learning (DL) in particular, which can be of unsupervised [10], supervised [12], or a hybrid form [18]. Signiﬁcant performance gain has been observed, especially in the presence of large amount of t”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “Classical generative models include Restricted Bolztmann Machines (RBMs) [12, 39, 21], and Auto-encoders [40, 20].”
Learning to generate chairs with convolutional neural networks (2014, background): “Prominent examples from this line of work are restricted Boltzmann machines (RBMs) [13] and Deep Boltz-mann Machines (DBMs) [26], as well as the plethora of models derived from them [12, 22, 20, 29, 23].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “This paper studies the problem of learning energy-based generative ConvNet models [24, 10, 12, 11, 39, 25, 35, 28, 45, 46, 15] of images.”
A Theory of Generative ConvNet (2016, methodology): “Compared to hierarchical models with explicit binary latent variables such as those based on the Boltzmann machine (Hinton et al., 2006a; Salakhutdinov & Hinton, 2009; Lee et al., 2009), the generative ConvNet is directly derived from the discriminative ConvNet.”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, methodology): “By combining convolutions and un-pooling (upsampling) layers [5, 1, 8] these models can be applied to large images.”
End-to-End Training of Deep Visuomotor Policies (2015, background): “Most applications of CNNs focus on classiﬁcation, where locational information is intentionally discarded by means of successive pooling layers [18].”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, background): “Bolztmann Machines (RBMs) [24, 40], Auto-encoders [20, 42] and generative adversarial network (GAN) [7, 10, 11] are widely studied.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, methodology): “These learning meth-ods include classical ones such as Restricted Boltzman Machines [27, 41] as well as more recent ones such as Variational Auto-Encoders [35] and Generative Adversarial Networks [15, 20].”
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, methodology): “Deep belief networks (Lee et al., 2009) have also been shown to work well in learning hierarchical representations.”
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, background): “The collection of filters F can be learnt, for example, through dictionary learning methods [18, 20, 31] or CNNs, and are generally full rank and expensive to convolve with large images.”
Squeeze-and-Excitation Networks (2017, neutral): “The two observations are consistent with findings in previous work [23, 50], namely that lower layer features are typically more general (i.”
Building high-level features using large scale unsupervised learning (2011, methodology): “In this section, we describe the algorithm that we use to learn features from the unlabeled training set.”
Learning to See by Moving (2015, methodology): “Our best performing architecture ( C96-P-C256-P ), outperforms convolutional deep belief networks [27], a previous approach based on learning features invariant to transformations [34] and features learnt by the default autoencoder network provided with Caffe[21].”
Deep Convolutional Inverse Graphics Network (2015, methodology): “ for generative modeling. 2 Related Work As mentioned before, a number of generative models have been proposed in the literature to obtain abstract visual representations. Unlike most RBM based models[9, 25, 17], our approach is trained using back-propagation from the data reconstruction term and th”