Are Large-scale Datasets Necessary for Self-Supervised Pre-training?

By Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Herv'e J'egou, Edouard Grave. 2021

170 citations recorded by the source corpus.

Pre-training models on large scale datasets, like ImageNet, is a standard practice in computer vision. This paradigm is especially effective for tasks with small training sets, for which high-capacity models tend to overfit. In this work, we consider a self-supervised pre-training scenario that only leverages the target task data. We consider datasets, like Stanford Cars, Sketch or COCO, which are order(s) of magnitude smaller than Imagenet. Our study shows that denoising autoencoders, such as BEiT or a variant that we introduce in this paper, are more robust to the type and size of the pre-training data than popular self-supervised methods trained by comparing image embeddings.We obtain competitive performance compared to ImageNet pre-training on a variety of classification datasets, from different domains. On COCO, when pre-training solely using COCO images, the detection and instance segmentation performance surpasses the supervised ImageNet pre-training in a comparable setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Finally, while progress on ImageNet has been encouraging, there has not been strong evidence that current self-supervised methods scale well to larger uncurated datasets [34,11].”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “In terms of its scalability, the ability of MIM tasks to train large-capacity models has been well demonstrated in [17, 45, 29], however, the ability to benefit from larger data currently sounds negative, as shown in [12].”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training (He et al., 2022; Bao et al., 2021; El-Nouby et al., 2021), potentially in feature space (Assran et al., 2023; Baevski et al., 2022).”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “For inpainting, it adopts a similar architecture as MAE in that a lightweight (shallow) ViT decoder is used to recover the masked patches from the representation of unmasked patches [63].”
DeiT III: Revenge of the ViT (2022, background): “This greater flexibility typically comes with the downside that they need larger datasets, or the training must be adapted when the data is scarcer [14, 48].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “With the introduction of Vision Transformers (ViT) [22] and motivated by the success of BERT [20] in NLP, many recent works propose a variety of masked image prediction methods for pre-training vision models in a self-supervised way, using reconstruction targets such as pixels [5,13,22,25,28,74], di”
A Cookbook of Self-Supervised Learning (2023, background): “, 2021], or on datasets that different from ImageNet such as COCO [El-Nouby et al., 2021], or iNaturalist [Assran et al.”
Masked Siamese Networks for Label-Efficient Learning (2022, methodology): “, 2021) and SplitMask (El-Nouby et al., 2021) apply a joint-embedding loss to an output representing the global sequence (either the [CLS] token or a global average pool of the patch vectors).”
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers (2022, background): “…to the reconstruction targets: low-level image elements (e.g., raw pixels; He et al. 2022; Fang et al. 2022; Liu et al. 2022), handcrafted features (e.g., HOG features; Wei et al. 2021), and visual tokens; Bao et al. 2022; Wang et al. 2022; Dong et al. 2021; El-Nouby et al. 2021; Chen et al. 2022.”