ImageNet-21K Pretraining for the Masses

By Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, Lihi Zelnik-Manor. 2021

956 citations recorded by the source corpus.

ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, methodology): “, [35, 40]) that uses WordNet [28] to expand single labels to multiple labels.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “ImageNet-21k dataset contains approximately 14 million images with about 21’000 distinct object categories [9, 26].”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “Benchmark: Following L2P [43] and DualPrompt [42], we adopt pre-training from ImageNet-21K dataset [30], also known as the full ImageNet [6] consisting of 14,197,122 images with 21,841 classes.”
A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, methodology): “The use of large-scale datasets, such as ImageNet-1k[23] and ImageNet-21k [22] and pretraining [24] has led to significant advances.”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “3) Pre-trained vision models are useful in practice for many fields [50, 20], including continual learning.”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Concretely, except the 1000 classes in ILSVRC 2012, we also randomly sample 1000 classes from the remaining classes in ImageNet-21K (Ridnik et al., 2021) to build ImageNet-2K dataset with 2.29M images.”
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications (2022, methodology): “We follow [33] to preprocess the pretraining data by removing classes with fewer examples and split it into training and validation sets containing around 11M and 522K images respectively over 10,450 classes.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “The other half is constructed from image and class data, such as Imagenet21k (Ridnik et al., 2021) or image and caption data, such as YFCC15M (Radford et al., 2021).”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “In fact, a common practice to evaluate the quality of these vision models is through linear probing or ﬁne-tuning on ImageNet [45, 47].”
TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, background): “This is also recognized by existing literature [77,5,55] and approximately 10% images in ImageNet are considered as hard samples.”
Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning (2021, methodology): “We select state-of-the-art self-supervision models SwAV [3] (ResNet50 backbone), CLIP [47] trained via natural language supervision on a large dataset of 400 million image and sentence pairs (VisionTransformer [10] backbone), BiT(-M) [29], which trains a ResNet50-V2 [29] on both the standard ImageNe”
Detecting Twenty-thousand Classes using Image-level Supervision (2022, methodology): “To show our method is compatible with better pretraining, we use ImageNet-21k pretrained backbone weights [43].”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, methodology): “We consider supervised pre-training on ImageNet1K [11] and ImageNet21K [11, 36], self-supervised pre-training via MoCoV2 [16] and DenseCL [43], and the vision-language pre-training.”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, positive): “We use ImageNet-21k (Ridnik et al., 2021) pretraining for ResNet and CLIP (Radford et al., 2021) pretraining for ViT-B/16.”
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022, methodology): “We further investigate the effect of different pre-training methods on the tracking performance by comparing four different pre-training strategies: no pre-training; ImageNet-1k [10] pre-trained model provided by [41]; ImageNet21k [39] pre-trained model provided by [40]; unsupervised pre-training mo”
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training (2022, methodology): “In detail, we instantiate the video encoder with MViT-Base model [28] pretrained on ImageNet-21K [37].”
Adversarial vulnerability of powerful near out-of-distribution detection (2022, methodology): “Recently, large models (such as the Vision Transformer [11]) pre-trained on large datasets (such as ImageNet21k [12]) produced sufficiently high-quality image embeddings that allowed us to close the gap to human performance in many challenging near-OOD tasks in vision (such as distinguishing CIFAR-1”
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models (2022, background): “For example Imagenet-21K [Ridnik et al. 2021] has 21,000 object categories.”