Unsupervised Feature Learning via Non-parametric Instance Discrimination

By Zhirong Wu, Yuanjun Xiong, Stella X. Yu, Dahua Lin. 2018

3,981 citations recorded by the source corpus.

Neural net classifiers trained on data with annotated class labels can also capture apparent visual similarity among categories without being directed to do so. We study whether this observation can be extended beyond the conventional domain of supervised learning: Can we learn a good feature representation that captures apparent similarity among instances, instead of classes, by merely asking the feature to be discriminative of individual instances? We formulate this intuition as a non-parametric classification problem at the instance-level, and use noise-contrastive estimation to tackle the computational challenges imposed by the large number of instance classes. Our experimental results demonstrate that, under unsupervised learning settings, our method surpasses the state-of-the-art on ImageNet classification by a large margin. Our method is also remarkable for consistently improving test performance with more training data and better network architectures. By fine-tuning the learned feature, we further obtain competitive results for semi-supervised learning and object detection tasks. Our non-parametric model is highly compact: With 128 features per image, our method requires only 600MB storage for a million images, enabling fast nearest neighbour retrieval at the run time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, background): “Recently, contrastive learning [3, 22] has been popular, e.g ., [62, 43, 23, 7], which models image similarity and dissimilarity (or only similarity [21, 8]) between two or more views.”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “We do not train the model with a memory bank (Wu et al., 2018).”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “As the focus of this paper is not on designing a new pretext task, we use a simple one mainly following the instance discrimination task in [61], to which some recent works [63, 2] are related.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020; He et al., 2020).”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Recent methods [67, 25, 4, 45, 24, 60] have advanced the state of the art in terms of learning representations that can linearly separate between the 1000 ImageNet categories [50].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Instance Discrimination is another contrastive objective which encourages representations that can discriminate between individual examples in the dataset (Wu et al., 2018).”
Learning Visual Representations with Caption Annotations (2020, background): “Besides, recently, contrastive learning-based unsupervised meth-ods [3,26,43,62] have showed signiﬁcant improvements.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “architecture are provided in AppendixB. As done in [15], we normalize vectors prior to computing their dot products in the NCE and MIL-NCE losses and use a temperature of ˝= 0:07 in the softmax as in [31, 64, 89]. When training with all three modalities on HowTo100M, we observe that a larger weight ”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “3 As in previous work, we also report performance when training a linear classiﬁer on top of a ﬁxed representation with all labels [31, 16, 17, 1] to directly evaluate SimCLRv2 representations.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “[39] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “h datasets. This veriﬁes our hypothesis that a good embedding plays an important role in few-shot recognition. 4.4. Embeddings from self-supervised representation learning Using unsupervised learning [61,49,16,51] to improve the generalization of the meta-learning algorithms [58] removes the needs o”
Improved Baselines with Momentum Contrastive Learning (2020, methodology): “In the instance discrimination pre-text task [16] (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.”
Contrastive Multiview Coding (2019, methodology): “ization [62] 13.1 24.8 31.0 32.6 31.8 Jigsaw [43] 19.2 30.1 34.7 33.9 28.3 BiGAN [15] 17.7 24.5 31.0 29.9 28.0 SplitBrainy[63] 17.7 29.3 35.4 35.2 32.8 Counting [44] 18.0 30.6 34.3 32.5 25.7 Inst-Dis [59] 16.8 26.5 31.8 34.1 35.6 RotNet [18] 18.8 31.7 38.7 38.2 36.5 DeepCluster [9] 12.9 29.2 38.2 39”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Our augmentation policy includes random resized cropping, horizontal flipping, color jittering (Wu et al., 2018).”
Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “A large body of work on self-supervised learning focuses on discriminative approaches coined instance classiﬁcation [12, 20, 33, 73], which considers each image a different class and trains the model by discriminating them up to data augmentations.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “Other details follow the kNN implementation in [44].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Selfsupervised learning has a long history in computer vision with methods based on autoencoders [42, 51], clustering [1, 5, 11] or instance-level discrimination [3, 15, 21, 52].”
Exploring Simple Siamese Representation Learning (2020, methodology): “This methodology has been recently popularized for un-/self-supervised representation learning [36, 30, 20, 37, 21, 2, 35, 17, 29, 8, 9].”
Dual Contradistinctive Generative Autoencoder (2020, positive): “Inspired by the literature on instance-level classiﬁcation [50], approximating likelihood by classiﬁcation [60], and contrastive learning [20, 65, 22], we propose to model instance-level ﬁdelity by contrastive loss (commonly referred to as InfoNCE loss) [62].”