Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition

By Marc'Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, Yann LeCun. 2007

1,191 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “The most well-known such criteria are image reconstruction [7, 38, 31, 29, 34, 23] (see [6] for a comprehensive overview) and feature slowness [48, 18].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Selfsupervised learning has a long history in computer vision with methods based on autoencoders [42, 51], clustering [1, 5, 11] or instance-level discrimination [3, 15, 21, 52].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, methodology): “…are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al.…”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “, using sparse coding [58], adversarial training [12, 13, 50], autoencoders [49, 63, 76], or probabilistic versions thereof [67].”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, background): “Bolztmann Machines (RBMs) [24, 40], Auto-encoders [20, 42] and generative adversarial network (GAN) [7, 10, 11] are widely studied.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “This can be modeled as maximizing the probability of reconstructing the input [43, 51, 67] and optionally estimating latent variables [29, 58] or using adversarial training [15, 44].”
Building high-level features using large scale unsupervised learning (2011, methodology): “Our work is inspired by recent successful algorithms in unsupervised feature learning and deep learning (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007; Lee et al., 2007).”
Learning to See by Moving (2015, methodology): “Another line of work has focused on learning features that are invariant to transformations irrelevant to the identity of objects, either from images [14, 34], or from video [41].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Typically, a parametrized mapping is learned between a predefined random noise and the images, with either an autoencoder [4, 22, 29, 40, 62], a generative adversarial network (GAN) [20] or more directly with a reconstruction loss [6].”
Unsupervised Learning by Predicting Noise (2017, methodology): “Many have also used autoencoders with a reconstruction loss (Bengio et al., 2007; Ranzato et al., 2007; Masci et al., 2011).”
Deep learning via semi-supervised embedding (2008, methodology): “They include a particular kind of generative model (a restricted Boltzmann machine) (Hinton et al., 2006), autoassociators (Bengio et al., 2007), and a method of sparse encoding (Ranzato et al., 2007).”
Stacked What-Where Auto-encoders (2015, methodology): “Among them, an architecture consists of a stack of encoder-decoder pairs [21, 22] is the most relevant approach to our SWWAE, with the main difference being that in SWWAE we train the stack of encoder-decoder pairs jointly with multiple losses as a whole, as opposed to their disjoint mechanism of bu”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, methodology): “This idea was inspired from an architecture designed for unsupervised feature learning [18].”
Deep Convolutional Inverse Graphics Network (2015, background): “[18], whose work was amongst the first to use a generic encoder-decoder architecture for feature learning.”
Unsupervised Learning of Visual Representations Using Videos (2015, background): “The most common architecture used is an auto-encoder which learns representations based on its ability to reconstruct the input images [35, 3, 49, 37].”
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition (2010, methodology): “Recent methods pre-train each hidden CNN layer one by one in an unsupervised fashion (this seems promising especially for small training sets), then use supervised learning to achieve 0.39% error rate (Ranzato et al., 2006, 2007).”
Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition (2010, background): “It has been shown that making these representations adaptive to image data can improve perform ance on vision tasks as demonstrated in [1] in a supervised learning framework and in [2, 3] using u ns pervised learning.”
Practical Black-Box Attacks against Machine Learning (2016, methodology): “alidation. The architecture of our substitute DNN is a standard one for computer vision classiﬁcation tasks: a convolutional layer of 32 kernels of size 2x2 [9, 15], a max pooling layer of kernel 2x2 [24], a convolutional layer of 64 kernels of size 2x2, a max pooling layer of kernel 2x2, two layers”