Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks

By Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin A. Riedmiller, Thomas Brox. 2014

636 citations recorded by the source corpus.

Deep convolutional networks have proven to be very successful in learning task specific features that allow for unprecedented performance on various computer vision tasks. Training of such networks follows mostly the supervised learning paradigm, where sufficiently many input-output pairs are required for training. Acquisition of large training sets is one of the key challenges, when approaching a new task. In this paper, we aim for generic feature learning and present an approach for training a convolutional network using only unlabeled data. To this end, we train the network to discriminate between a set of surrogate classes. Each surrogate class is formed by applying a variety of transformations to a randomly sampled `seed' image patch. In contrast to supervised network training, the resulting feature representation is not class specific. It rather provides robustness to the transformations that have been applied during training. This generic feature representation allows for classification results that outperform the state of the art for unsupervised learning on several popular datasets (STL-10, CIFAR-10, Caltech-101, Caltech-256). While features learned with our approach cannot compete with class specific features from supervised training on a classification task, we show that they are advantageous on geometric matching problems, where they also outperform the SIFT descriptor.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “While there exist many approaches on how to transfer to the new task, the most common approach in modern deep learning, which we use, is to (i) train a model on Ppre−train (using perhaps an auxiliary, self-supervised task [15, 22]), and then (ii) train a model on Ptrain by initializing the model wei”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “→ The methods [1, 2, 3, 4, 5] are from: [Gidaris et al., 2018, Dosovitskiy et al., 2014, Doersch and Zisserman, 2017, Noroozi and Favaro, 2016, van den Oord et al., 2018].”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “However, explicitly learning a classiﬁer to discriminate be-tween all images [20] does not scale well with the number of images.”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Selfsupervised learning has a long history in computer vision with methods based on autoencoders [42, 51], clustering [1, 5, 11] or instance-level discrimination [3, 15, 21, 52].”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, methodology): “Finally, we applied the method to invert the representation learned by AlexNet and analyzed some properties of the method.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Representations ought to be invariant under image transformations to be useful for image recognition [14, 31] because the transformations do not alter visual semantics.”
Large Scale Adversarial Representation Learning (2019, methodology): “Other approaches include colorization [34, 35], motion segmentation [25], rotation prediction [9, 2], GAN-based discrimination [26, 2], and exemplar matching [6].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, neutral): “, 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al., 2018) and counting (Noroozi et al.”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, methodology): “Given a test sample, we retrieve its top-k ( k = 200 ) nearest neighbors based on cosine similarity, then apply weighted voting to predict its label [46] plar CNN [8], NPSoftmax [46], NCE [46] and Triplet loss with and without hard mining.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “…approaches include deep volume-preserving maps (Dinh et al., 2014, 2016), deep clustering (Xie et al., 2016; Chang et al., 2017), noise as targets (NAT, Bojanowski & Joulin, 2017), self-supervised methods (Doersch & Zisserman, 2017), and co-learning (Dosovitskiy et al., 2016; Sajjadi et al., 2016).”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “Note it also includes a number of clustering methods (DeepCluster [7], exemplars [18]) where the clustering is only auxiliary; a clustering-style objective is used but does not produce groups with semantic correspondence.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Many pretext tasks have been designed for images that exploit their spatial structure [12, 48–50], color information [11, 37, 38, 74], illumination [16], rotation [24] etc.”
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, background): “The performance of DCGANs is still less than that of Exemplar CNNs (Dosovitskiy et al., 2015), a technique which trains normal discriminative CNNs in an unsupervised fashion to differentiate between speciﬁcally chosen, aggressively augmented, exemplar samples from the source dataset.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many recent state-of-the-art methods build upon the instance discrimination task that considers each image of the dataset (or “instance”) and its transformations as a separate class [16].”
Self-labelling via simultaneous clustering and representation learning (2019, neutral): “, 2019), geometry (Dosovitskiy et al., 2016), predicting transformations (Gidaris et al.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “applicable to control tasks, the RL context does require modiﬁcations to be made to the underlying algorithm. Data augmentation methods have also been used in the context of self-supervised learning. [14] use per-exemplar perturbations in a unsupervised classiﬁcation framework. More recently, a seve”
Learning to Compose Domain-Specific Transformations for Data Augmentation (2017, background): “Heuristic Data Augmentation Most state-of-the-art image classification pipelines use some limited form of data augmentation [11, 7].”
Boosting Self-Supervised Learning via Knowledge Transfer (2018, background): “A natural question is then: How do we design such a task? Some SSL approaches define pretext tasks through explicit desirable invariances of the metric [32, 33, 10, 23] or such that they implicitly require a good object representation [7, 21, 39].”