Building high-level features using large scale unsupervised learning

By Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Greg S. Corrado, Kai Chen, Joseph Dean, Andrew Y. Ng. 2011

2,318 citations recorded by the source corpus.

We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a deep sparse autoencoder on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200×200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting from these learned features, we trained our network to recognize 22,000 object categories from ImageNet and achieve a leap of 70% relative improvement over the previous state-of-the-art.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “uch better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning mo”
Sequence to Sequence Learning with Neural Networks (2014, background): “Deep Neural Networks (DNNs) are extremely powerful machine learning models that achieve excellent performance on difficult problems such as speech recognition [13, 7] and visual object recognition [19, 6, 21, 20].”
Fully convolutional networks for semantic segmentation (2014, background): “ully convolutional networks that take input of any size and output classiﬁcation maps. This transformation is illustrated in Figure2. (By contrast, nonconvolutional nets, such as the one by Le et al. [18], lack this capability.) Furthermore, while the resulting maps are equivalent to the evaluation ”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “At the cost of expressivity, deep linear networks gain theoretical tractability and may prove fertile for addressing other phenomena in deep learning, such as the impact of carefully-scaled initializations [13, 23], momentum [23], dropout regularization [1], and sparsity constraints [2].”
Intriguing properties of neural networks (2013, methodology): “For all the networks we studied (MNIST, QuocNet [10], AlexNet [9]), for each sample, we have always managed to generate very close, visually hard to distinguish, adversarial examples that are misclassified by the original network (see figure 5 and http://goo.gl/huaGPb for examples).”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “In Computer Vision, Convolutional Neural Networks (CNNs) [23] have recently been shown to learn powerful image representations that support state of the art image classification [24, 25, 26] and object detection [27, 28].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “To speed up the supervised training of this network, we made a simple modification to the algorithm described in Le et al. (2012): adding a “bottleneck” layer in between the last layer and the classifier.”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “Classical generative models include Restricted Bolztmann Machines (RBMs) [12, 39, 21], and Auto-encoders [40, 20].”
Towards Conceptual Compression (2016, methodology): “Numerous techniques exist for unsupervised learning in deep networks, e.g. sparse auto-encoders and sparse coding (Kavukcuoglu et al., 2010; Le, 2013), denoising auto-encoders (Vincent et al., 2010), deconvolutional networks (Zeiler et al., 2010), restricted Boltzmann machines (Hin-ton &…”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Unsupervised learning methods span a very broad spectrum of approaches going back to the roots of artificial neural networks [57, 43, 3, 62, 30, 45, 31, 33, 27], and are too numerous to fully review here.”
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition (2013, background): “However, reported successes with such models in convolutional networks have been limited to relatively small datasets such as CIFAR and MNIST, and efforts on larger datasets have had only modest success (Le et al., 2012).”
Generating Videos with Scene Dynamics (2016, background): “, we expect the network to learn motion features on its own. Finally, this paper is related to a growing body of work that capitalizes on large amounts of unlabeled video for visual recognition tasks [18, 47, 37, 13, 24, 25, 3, 32, 26, 27, 19, 41, 42, 1]. We instead leverage large amounts of unlabel”
Deep multi-scale video prediction beyond mean square error (2015, background): “A signiﬁcant difference with the more classical problem of image reconstruction (Vincent et al., 2008; Le, 2013) is that the ability of a model to predict future frames requires to build accurate, non trivial internal representations, even in the absence of other constraints (such as sparsity).”
Predicting Parameters in Deep Learning (2013, background): “It is now possible to train networks with tens of millions [13] or even over a billion parameters [7, 16].”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “Exploiting scale in deep learning has been central to the success of the field thus far (Krizhevsky et al., 2012; Le et al., 2012).”
Axiomatic Attribution for Deep Networks (2017, background): “…of this work has been on networks trained on computer vision tasks, and deals with understanding what a speciﬁc neuron computes (Erhan et al., 2009; Le, 2013) and interpreting the representations captured by neurons during a prediction (Mahendran & Vedaldi, 2015; Dosovitskiy & Brox, 2015; Yosinski…”