Unsupervised Pre-Training of Image Features on Non-Curated Data

By Mathilde Caron, Piotr Bojanowski, Julien Mairal, Armand Joulin. 2019

256 citations recorded by the source corpus.

Pre-training general-purpose visual features with convolutional neural networks without relying on annotations is a challenging and important task. Most recent efforts in unsupervised feature learning have focused on either small or highly curated datasets like ImageNet, whereas using uncurated raw datasets was found to decrease the feature quality when evaluated on a transfer task. Our goal is to bridge the performance gap between unsupervised methods trained on curated data, which are costly to obtain, and massive raw datasets that are easily available. To that effect, we propose a new unsupervised approach which leverages self-supervision and clustering to capture complementary statistics from large-scale data. We validate our approach on 96 million images from YFCC100M, achieving state-of-the-art results among unsupervised methods on standard benchmarks, which confirms the potential of unsupervised learning when only uncurated data are available. We also show that pre-training a supervised VGG-16 with our method achieves 74.9% top-1 classification accuracy on the validation set of ImageNet, which is an improvement of +0.8% over the same network trained from scratch. Our code is available at https://github.com/facebookresearch/DeeperCluster.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “, transformations of a single (“exemplar”) image [17], patch orderings [13, 45], tracking [59] or segmenting objects [47] in videos, or clustering features [3, 4].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “In contrast, many existing methods for learning image representations from video data often rely on short curated videos consisting of single clips, or even the treat training set as a bag of frames [7, 13].”
Learning Visual Representations with Caption Annotations (2020, methodology): “they require either extremely large training sets (up to a hundred million images) [7,24] or need to be trained much longer with larger networks to express their full potential [9,26].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “[13] further explain the performance gap between training on such curated data versus uncurated ones, truly available at scale.”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several variants allow automatic grouping of instances in the form of clustering [2, 7, 8, 21, 29, 35, 61, 65, 69].”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “Prior attempts to train self-supervised models on uncurated data [6, 13, 19, 27] have only used a few millions of images for which using small model architectures is sufﬁcient.”
Exploring Simple Siamese Representation Learning (2020, methodology): “Like contrastive learning, clustering-based methods require either a memory bank [5, 6, 1], large batches [7], or a queue [7] to provide enough samples for clustering.”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, methodology): “Previously, this problem has been mitigated by workarounds such as re-initializing codewords or applying additional regularizers to the loss function (Caron et al., 2019).”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Our work is also related to clustering-based methods [2, 3, 6, 7, 17, 25, 50, 53, 54, 59].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “Examples include image inpainting [21], colorization [22, 23], prediction of patch orderings [24, 25] and image transformations [26, 27, 28, 29].”
The Challenges of Continuous Self-Supervised Learning (2022, background): “Given its success, a few attempts have been made to scale SSL to large uncurated datasets, such as YFCC-100M [8, 26] and Instagram-1B [9, 24].”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Previous works (Caron et al., 2019; He et al., 2020) have leveraged especially large datasets for pre-training, such as YFCC 100M (Thomee et al., 2016) and Instagram 1B (Mahajan et al., 2018).”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “Instead of viewing each sample as its own class, clustering-based methods [3, 46, 44, 27, 51, 6, 1, 45, 18, 5, 7] regroup them into clusters based on some similarity measure.”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “Self-supervised methods often try to reconstruct the input data or impose constraints on the representation, such as sparsity [48, 59, 60], noise [82] or invariance [8, 10, 11, 17, 28, 35, 53, 69] to learn a useful and transferable feature representation.”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “Self-supervised representation learning has been widely studied in computer vision (Dosovitskiy et al., 2014; 2015; Doersch et al., 2015; Xie et al., 2016; Yang et al., 2016; van den Oord et al., 2018; Caron et al., 2018; Asano et al., 2019; Caron et al., 2019; Huang et al., 2019; Zhuang et al., 201”
Radioactive data: tracing through training (2020, background): “, 2018) and unsupervised learning (Caron et al., 2019) where annotations are scarcer, state-of-the-art results are obtained on large-scale datasets collected from the Web (Thomee et al.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “Other self-supervised tasks include predicting rotation [20], relative patch location [13], clusters [7, 8], and number of objects [42], etc.”
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations (2021, background): “Methods that use prototypes/clusters [1, 2, 5, 6, 7, 23, 33, 53, 56, 57, 58, 60] also attempt to learn features by associating multiple samples with the same cluster.”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, unknown): “YFCC or IG) data using instance discrimination, this usually requires order of magnitude more data than Ima-geNet [22, 51].”