Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives

By Yoshua Bengio, Aaron C. Courville, Pascal Vincent. 2012

568 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Deep Generative Stochastic Networks Trainable by Backprop (2013, methodology): “distribution P(X). 4 Reconstruction with Noise Injected in the Reconstruction Function: Consistent Estimation of the Underlying Data Generating Distribution In the context where Theorem 1 was proven (Bengio et al., 2013b), X~ is a noisy or corrupted version of X, i.e., it lives in the same space as ”
Generalized Denoising Auto-Encoders as Generative Models (2013, background): “Different variants of auto-encoders and sparse coding have been, along with RBMs, among the most successful building blocks in recent research in deep learning (Bengio et al. , 2013b).”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “Stochastic neurons with binary outputs are also interesting because they can easily give rise to sparse representations (that have many zeros), a form of regularization that has been used in many representation learning algorithms (Bengio et al. , 2013).”
What makes ImageNet good for transfer learning? (2016, background): “The most well-known such criteria are image reconstruction [7, 38, 31, 29, 34, 23] (see [6] for a comprehensive overview) and feature slowness [48, 18].”
Challenges in representation learning: A report on three machine learning contests (2013, background): “tru Erhan, and Yoshua Bengio, was to explore the latest developments in representation learning, with a special emphasis on testing the capabilities of current representation learning algorithms (See [1] for a recent review) and pushing the eld towards new developments via these contests. Ben Hamner”
Learning to See by Moving (2015, methodology): “llions of labelled examples. Unlabelled visual data is freely available and in theory this data can be used to learn useful visual representations. However, until now unsupervised learning approaches [6, 27, 34, 37] have not yet delivered on their promise and are nowhere to be seen in current applic”
cuDNN: Efficient Primitives for Deep Learning (2014, background): “Deep neural networks have been successful at solving many kinds of tasks [4].”
Deep Gaussian Processes (2012, background): “The recent advances in the domain of deep learning [Hinton and Osindero, 2006, Bengio et al., 2012] have brought this kind of models again in popularity.”
What regularized auto-encoders learn from the data-generating distribution (2012, methodology): “n we compare the samples generated by the model with the training samples, projected into various 2-D views for visualization. 2 Contractive and Denoising Auto-Encoders Regularized auto-encoders (see Bengio et al. (2012b) for a review and a longer exposition) capture the structure of the training di”
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning (2014, methodology): “Following Bengio et al. (2013a) and Mnih and Gregor (2014), estimate a gradient using a variance-reduced variant of REINFORCE, i.e., by reinforcement learning.”
Estimating or Propagating Gradients Through Stochastic Neurons (2013, background): “also interesting because they can easily give rise to sparse representations (that have many zeros), a form of regularization that has been used in manyrepresentationlearning algorithms(Bengio, 2009; Bengio et al., 2013). Sparsity of the representation corresponds to the prior that, for a given inpu”
Deep Learning of Representations: Looking Forward (2013, background): “f multiple levels of representation. They have had important empirical successes in a number of traditional AI applications such as computer vision and natural language processing. See (Bengio, 2009; Bengio et al., 2013d) for reviews and Bengio (2013c) and the other chapters of the book by Montavon ”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “Note that this representation is the kind of representation that one could hope an unsupervised learning algorithm could discover, at best, as argued in Bengio et al. (2012).”
Is Generator Conditioning Causally Related to GAN Performance? (2018, background): “Bengio et al. (2012) discuss an interpretation of representation learning as manifold learning.”
State Representation Learning for Control: An Overview (2018, background): “This kind of a priori knowledge is called prior [Bengio et al., 2012], [Jonschkowski and Brock, 2015], [Lesort et al., 2017],[Jon-schkowski et al., 2017], and is deﬁned through cost functions.”
Regularized Auto-Encoders Estimate Local Statistics (2012, methodology): “As we further our understanding auto-encoders as estimating local properties of the data generating density, this suggests sampling algorithms as discussed in Bengio et al. (2012a); Rifai et al. (2012).”