Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels

By Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, Sanghyuk Chun. 2021

176 citations recorded by the source corpus.

ImageNet has been the most popular image classification benchmark, but it is also the one with a significant level of label noise. Recent studies have shown that many samples contain multiple classes, despite being assumed to be a single-label benchmark. They have thus proposed to turn ImageNet evaluation into a multi-label task, with exhaustive multi-label annotations per image. However, they have not fixed the training set, presumably because of a formidable annotation cost. We argue that the mismatch between single-label annotations and effectively multi-label images is equally, if not more, problematic in the training setup, where random crops are applied. With the single-label annotations, a random crop of an image may contain an entirely different object from the ground truth, introducing noisy or even incorrect supervision during training. We thus re-label the ImageNet training set with multi-labels. We address the annotation cost barrier by letting a strong image classifier, trained on an extra source of data, generate the multi-labels. We utilize the pixel-wise multi-label predictions before the final pooling layer, in order to exploit the additional location-specific supervision signals. Training on the re-labeled samples results in improved model performances across the board. ResNet-50 attains the top-1 accuracy of 78.9% on ImageNet with our localized multi-labels, which can be further boosted to 80.2% with the CutMix regularization. We show that the models trained with localized multi-labels also outperforms the baselines on transfer learning to object detection and instance segmentation tasks, and various robustness benchmarks. The re-labeled ImageNet training set, pre-trained weights, and the source code are available at https://github.com/naverai/relabel_imagenet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Baseline methods We compare the performance of models trained using ImageNet + with the following baseline methods: (1) KD [35, 6] (Online distillation): A standard knowledge distillation method with strong teacher models and model-specific augmentations, (2) MEALV2 [54] (Fine-tuning distillation): ”
ImageNet-21K Pretraining for the Masses (2021, methodology): “Knowledge distillation (KD) is a known method to improve not only upstream, but also downstream results [50, 51, 53].”
Knowledge distillation: A good teacher is patient and consistent (2021, background): “Second, knowledge distillation is also commonly used in different contexts (other than model compression), where authors recommend different or even opposite design choices [40,48,50], see Figure 2.”
All Tokens Matter: Token Labeling for Training Better Vision Transformers (2021, methodology): “The score map generation procedure is similar to [49], but we limit our experiment setting by training all models from scratch on ImageNet without extra data support, such as JFT-300M and ImageNet-22K.”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, methodology): “Comparison of ReLabel [52] and our fast knowledge distillation (FKD) framework.”
TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, background): “This is also recognized by existing literature [77,5,55] and approximately 10% images in ImageNet are considered as hard samples.”
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, methodology): “This applies also to ResNet50 model, where we compare USI to a recently proposed dedicated scheme [42], and a KD-based scheme [47].”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “Whereas many of the label issues in ImageNet are due to multi-labeled examples (Yun et al., 2021), next we consider a dataset with disjoint classes.”
Concept Generalization in Visual Representation Learning (2020, background): “3.4 for the justification of our protocol and the choice of metrics.”
Asymmetric Loss For Multi-Label Classification (2020, background): “Typical natural images contain multiple objects and concepts [34, 37], highlighting the importance of multi-label classification for real-world tasks.”