Discriminative Unsupervised Feature Learning with Convolutional Neural Networks

By Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, Thomas Brox. 2014

1,065 citations recorded by the source corpus.

Current methods for training convolutional neural networks depend on large amounts of labeled samples for supervised training. In this paper we present an approach for training a convolutional neural network using only unlabeled data. We train the network to discriminate between a set of surrogate classes. Each surrogate class is formed by applying a variety of transformations to a randomly sampled 'seed' image patch. We find that this simple feature learning algorithm is surprisingly successful when applied to visual object recognition. The feature representation learned by our algorithm achieves classification results matching or outperforming the current state-of-the-art for unsupervised learning on several popular datasets (STL-10, CIFAR-10, Caltech-101).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

U-Net: Convolutional Networks for Biomedical Image Segmentation (2015, background): “[3] in the scope of unsupervised feature learning.”
A Simple Framework for Contrastive Learning of Visual Representations (2020, positive): “Discriminative approaches based on contrastive learning in the latent space have recently shown great promise, achieving state-of-theart results (Hadsell et al., 2006; Dosovitskiy et al., 2014; Oord et al., 2018; Bachman et al., 2019).”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “, transformations of a single (“exemplar”) image [17], patch orderings [13, 45], tracking [59] or segmenting objects [47] in videos, or clustering features [3, 4].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “To define the frame/shot-level loss LS, we propose to build on any SSL loss designed for images, such as classifying exemplars [15], solving jigsaw puzzles of image patches [43], or rotation prediction [18].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “Methods applied to the entire image (R OTATION , E XEMPLAR ) outperform patch-based methods (J IGSAW , R EL .”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “We demonstrate our framework on two prominent selfsupervised techniques: predicting image rotation [9] and exemplar [6].”
Self-Supervised MultiModal Versatile Networks (2020, background): “single modality), such as predicting the relative position of patches [18, 59], colorization [95], predicting orientation [27] or invariance to transformation [20, 36].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In particular, relative patch prediction [23, 40], colorizing grayscale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful.”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “TextTopicNet (Wiki) - CNN-ﬁnetuning * 76.51% TextTopicNet (Wiki) - fc7+SVM * 66.00% Semi-supervised auto-encoder [44] 74.33 % Convolutional k-means [8] 74.10% CNN with Target Coding [43] 73.15% Exemplar convnets [7] 72.80% Unsupervised pre-training [26] 70.20% Swersky et al .”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “Exemplar CNN [5] appears similar to our work.”
Unsupervised Representation Learning by Predicting Image Rotations (2018, methodology): “Dosovitskiy et al. (2014) train a ConvNet model to yield representations that are discriminative between images and at the same time invariant on geometric and chromatic transformations.”
Learning to generate chairs with convolutional neural networks (2014, methodology): “We include artiﬁcial transformations T θ described by the randomly generated parameter vector θ to increase the amount of variation in the training data and reduce overﬁt-ting, analogous to data augmentation in discriminative CNN training [18, 7].”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, methodology): “2) Estimating Between-image Labels, it usually estimates between-image labels using the clustering technique [3, 9, 26] or kNN-based methods [41], which provide label information.”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “Exemplar [9]: In this technique, every individual image”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, neutral): “to train a model on a pretext task like predicting rotation angle or relative location of an image patch, and then extracting representations from the resulting networks [23, 24, 25].”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “nvariance as a training objective. Optimising for function outputs to be persistent through spatio-temporal or non-material distortion is an idea shared by IIC with several works, including exemplars [17], IMSAT [27], proximity prediction [29], the denoising objective of Tagger [21], temporal slowne”
Multi-task Self-Supervised Visual Learning (2017, methodology): “Finally, one can form a surrogate “class” by taking a single image and altering it many times via translations, rotations, and color shifts [8], to create a synthetic categorization problem.”
Scattering Networks for Hybrid Representation Learning (2018, methodology): “The best reported result in the purely supervised case is a CNN [19], [52] whose hyper parameters have been automatically tuned using 4,000 images for validation achieving 70.”