Label Refinery: Improving ImageNet Classification through Label Progression

By Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, Ali Farhadi. 2018

201 citations recorded by the source corpus.

Among the three main components (data, labels, and models) of any supervised learning system, data and models have been the main subjects of active research. However, studying labels and their properties has received very little attention. Current principles and paradigms of labeling impose several challenges to machine learning algorithms. Labels are often incomplete, ambiguous, and redundant. In this paper we study the effects of various properties of labels and introduce the Label Refinery: an iterative procedure that updates the ground truth labels after examining the entire dataset. We show significant gain using refined labels across a wide range of models. Using a Label Refinery improves the state-of-the-art top-1 accuracy of (1) AlexNet from 59.3 to 67.2, (2) MobileNet from 70.6 to 73.39, (3) MobileNet-0.25 from 50.6 to 55.59, (4) VGG19 from 72.7 to 75.46, and (5) Darknet19 from 72.9 to 74.47.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “As previously reported (Bagherinezhad et al., 2018; Beyer et al., 2021), we find that it improves accuracy to run distillation with data augmentation.”
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, methodology): “We use label-reﬁnery (Bagherinezhad et al. (2018)) in training and we use the same data augmentation as this Pytorch example 1 .”
Knowledge Distillation: A Survey (2020, methodology): “Recently, knowledge distillation has been used successfully for solving the complex image classiﬁcation problems (Zhu et al. 2019; Bagherinezhad et al. 2018; Peng et al. 2019b; Li and Hoiem 2017; Chen et al. 2018a; Wang et al. 2019b; Mukherjee et al. 2019).”
Relational Knowledge Distillation (2019, methodology): “Likewise, many other methods [1, 2, 8, 12, 25, 45, 47] can also be formulated as a form of Eq.”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, methodology): “applied knowledge distillation to data argumentation, increasing the numerical value of labels to a higher entropy [2].”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, background): “In several works, multi-stage KD was proposed to improve both teacher and student by training a sequence of models [16, 37, 3].”
Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks (2021, background): “KD has also shown its potential in other ﬁelds such as adversarial robustness (Papernot et al., 2016), multi-granularity lip reading (Zhao et al., 2020b), and data augmentation (Bagherinezhad et al., 2018).”
Online Knowledge Distillation via Collaborative Learning (2020, background): “A few recent papers about self-distillation [29, 3, 6, 28] have shown that a converged teacher model supervising a student model of identical architecture could improve the generalization ability over the teacher.”
Highlight Every Step: Knowledge Distillation via Collaborative Teaching (2019, background): “…been proposed for KD in [61] and Lopes et al. [62] use the teacher model to provide metadata for data-free KD. Recently, researchers note that it is effective in improving a teacher model itself by self-distillation [63] [64], namely, a few models with the same architecture are trained one by one.”
Data Augmentation in Classification and Segmentation: A Survey and New Strategies (2023, background): “The conventional strategy for training modern state-of-the-art architectures is to crop patches as small as 8% of the given image and label them the same as the given image [38].”
Adaptive Regularization of Labels (2019, methodology): “Not coincidentally, previous researches [7, 20, 22, 31, 30, 15, 17, 1] have suggested that the cross entropy loss based on one-hot encoded labels may not be optimal in the classification task.”
Confidence Regularized Self-Training (2019, background): “Enforcing a model to be very confident on only one of the class during training can hurt the learning behavior [2], particularly within the under-determined context of UDA.”
Snapshot Distillation: Teacher-Student Optimization in One Generation (2018, background): “See Section 2 for a detailed survey. found its effectiveness in providing complementary cues to training the same network [11][2].”
Subclass Distillation (2020, background): “…methods focus on teachers and students with the same architecture which can be trained sequentially (Furlanello et al., 2018; Xie et al., 2019) (using unlabeled data and noisy student), (Bagherinezhad et al., 2018) (using extensive data augmentation) or in parallel (Anil et al., 2018) (ensemble).”