Self-supervised Pretraining of Visual Features in the Wild

By Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin and 1 more. 2021

302 citations recorded by the source corpus.

Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, positive): “On Places, our MAE outperforms the previous best results [19, 40], which were obtained via pre-training on billions of images.”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “2, 5 [28] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al.”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “Though image SSL has great promises in leveraging nearly infinite amounts of unlabelled image data in training [18], the lack of language association renders it hardly applicable to zero-shot recognition.”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “Even methods trained on larger datasets like Instagram-1B [ 24 , 26 ] are less likely to encounter heavily correlated samples in the mini-batches.”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “In the past two years, the CNN architectures have been further scaled up to about 1 billion parameters [18, 28], however, absolute performance may not be so encouraging, per-”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, methodology): “Using 4.5× fewer data, our DeCLIP-RegNetY-64GF achieves 73.7% zero-shot ImageNet top1 accuracy, which is on-pair with CLIP-R50×64.”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, unknown): “When applied on uncurated data, these methods degrade significantly and require larger datasets to obtain similar performance [22].”
AAVAE: Augmentation-Augmented Variational Autoencoders (2021, methodology): “This enables SSL methods to leverage large unlabeled datasets to learn representations [28] which can then be used to solve downstream tasks, such as classiﬁcation and segmentation, for which it is often expensive to collect a large number of annotations.”
PreDet: Large-scale weakly supervised pre-training for detection (2021, methodology): “9% absolute improvement in AP compared to ImageNet pre-trained model and self-supervised SEER [13] model pre-trained on 1B images respectively.”
Point Transformer V3: Simpler, Faster, Stronger (2023, background): “Deep learning models have experienced rapid advancements in various areas, such as 2D vision [24, 38, 78, 86] and natural language processing (NLP) [1, 37, 56, 79], with their progress often attributed to the effective utilization of scale, encompassing factors such as the size of datasets, the…”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, background): “The impact of pre-training dataset size/quality is explored for ConvNets in SSL [23, 68].”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “Self-supervised representation learning has been widely studied in computer vision , including: context prediction [24,75], clustering-based methods [88,93,8,1,105, 45,9,36], contrastive self-supervised learning [55,65,41,”
Self-Supervised Learning in Remote Sensing: A review (2022, methodology): “Indeed, by distilling representative features from unlabeled data, SSL algorithms are already outperforming supervised pre-training on many problems [7].”
Online Task-free Continual Learning with Dynamic Sparse Distributed Memory (2022, background): “[ 20] has shown that an SSL encoder can be trained in online learning with random images.”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “Of particular interest, Goyal et al. (2021) have also shown that these methods benefit from scaling in model size given enough pretrained data.”
Artificial intelligence and machine learning for medical imaging: A technology review. (2021, unknown): “It is gaining attention for computer vision tasks [70 – 73] and it could play an important role in future research directions for medical imaging applications.”