Self-Training With Noisy Student Improves ImageNet Classification

By Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, Quoc V. Le. 2019

2,812 citations recorded by the source corpus.

We present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. To achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as accurate as possible. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, background): “The use of additional data sources allows to achieve state-of-the-art results on standard benchmarks (Mahajan et al., 2018; Touvron et al., 2019; Xie et al., 2020).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “, 2020c) and self-training (Lee; Xie et al., 2020) methods is a promising direction given their demonstrated ability to improve data efficiency over standard supervised learning.”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “Similarly, Transformer models are pretrained with large amounts of data, which may also aid robustness (Orhan, 2019; Xie et al., 2020; Hendrycks et al., 2019a).”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “While state-ofthe-art performance on many competitive datasets is attained via transfer learning [70, 36], the implications for final robustness metrics remain unclear.”
Are we done with ImageNet? (2020, methodology): “Importantly, several models [30, 39] already surpass the ReaL accuracy obtained by the original ImageNet labels (Figure 4, red point and dashed line).”
Fixing the train-test resolution discrepancy: FixEfficientNet (2020, methodology): “Additionally, the ImageNet validation images were used to filter the images from the unlabelled set [9].”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “Specialized Representations Rather than pre-train generic representations, recent works have shown strong performance by training task-speciﬁc representations [63,38,61].”
Learning Visual Representations with Caption Annotations (2020, background): “Leveraging ImageNet labels, those approaches produce pseudo-labels for additional unlabeled images [64,65].”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “efficientnet-l2-noisystudent [102].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Inspired by the recent successes of learning from unlabeled data [19, 20, 1, 11, 24, 12], the proposed semi-supervised learning framework leverages unlabeled data in both task-agnostic and task-specific”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Some recent approaches employ teacher-student learning [45, 46], where a teacher (trained on supervised data) predicts pseudo-labels on unlabeled data that the student mimics.”
Towards Robust Vision Transformer (2021, background): “, stronger data augmentation [16, 18, 33], carefully designed [36,44] or searched [9,13] network architecture, improved training strategy [22, 26, 47], quantization [24] and pruning [49] of the weights, better pooling [41, 53] or activation functions [46], etc.”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “This propagation can either be done with hard assignments of labels [41, 78, 79] or with a soft assignment [76].”
Large Language Models Can Self-Improve (2022, neutral): “Majority Voting by answer ···”
Learning by Distilling Context (2022, background): “There has been a large literature on distilling knowledge in a neural network (Hinton et al., 2015b; Adriana et al., 2015; Liu et al., 2019; Yang et al., 2020; Xie et al., 2020).”
Self-training Improves Pre-training for Natural Language Understanding (2020, background): “Existing work, such as UDA (Xie et al., 2019), self-training (He et al., 2019; Xie et al., 2020) and back-translation for machine translation (Bojar and Tamchyna, 2011; Sennrich et al., 2015; Edunov et al., 2018), assumes the existence of unannotated data in the same domain as the downstream task.”