Meta Pseudo Labels

By Hieu Pham, Qizhe Xie, Zihang Dai, Quoc V. Le. 2020

774 citations recorded by the source corpus.

We present Meta Pseudo Labels, a semi-supervised learning method that achieves a new state-of-the-art top-1 accuracy of 90.2% on ImageNet, which is 1.6% better than the existing state-of-the-art [16]. Like Pseudo Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted by the feedback of the student’s performance on the labeled dataset. As a result, the teacher generates better pseudo labels to teach the student.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “This approach uses unlabeled data in a task-speciﬁc way to encourage class label prediction consistency on unlabeled data among different models [11, 12, 2] or under different data augmentations [13–15].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “This is the second highest validation accuracy achieved to date with extra training data, second only to a strong recent semi-supervised learning baseline (Pham et al., 2020), and the highest accuracy achieved using transfer learning.”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The learned features can be transferred to recognition tasks [16], and the classifier can be used to label images for semi-supervised learning [36, 55, 57].”
MLP-Mixer: An all-MLP Architecture for Vision (2021, methodology): “MPL is pre-trained at very large-scale on JFT-300M images, using meta-pseudo labelling from ImageNet instead of the original labels.”
Going deeper with Image Transformers (2021, background): “Hybrid architectures combining CNNs and transformers blocks have also been used on ImageNet [58, 70] and on COCO [11].”
Meta-Learning in Neural Networks: A Survey (2020, background): “include direct transductive training of labels [156] , or training a teacher network to generate labels for a student [257].”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, background): “Meta pseudo-labels (Pham et al., 2020), an improved semi-supervised learning technique, currently holds the ImageNet state-of-the-art (90.2%).”
The Computational Limits of Deep Learning (2020, methodology): “For example, in 2012 AlexNet trained using 2 GPUs for 5-6 days[22], in 2017 ResNeXt-101 [39] was trained with 8 GPUs for over 10 days, and in 2020 Meta Pseudo Labels (EfﬁcientNet-L2) was trained with 2048 TPU cores for 11 days[40].”
Debiased Pseudo Labeling in Self-Training (2022, background): “…conﬁdence threshold [21; 41] or prediction uncertainty [28], assign pseudo labels with the density of local neighborhood [30; 15], meta-learning [27], or simply use the predictions from a weaker augmented images [31] or the average predictions from multiple augmentations [2] to generate more…”
Extract Free Dense Labels from CLIP (2021, methodology): “Self-training has firstly become popular in the semisupervised classification task [33,29,35,43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [27,37,4,28,40,36,56,8].”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, background): “Meta Pseudo Labels [235] is constantly adapted through the feedback of the student’s performance on the labeled dataset so that the student can learn better pseudo labels from the teacher.”
A Survey on Deep Semi-Supervised Learning (2021, background): “MPL [236] modiﬁes Pseudo-label [224] by deriving the teacher network’s update rule from the feedback of the student network.”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “For simpler classification tasks, a plethora of SSL methods have been developed, based on pseudo-labeling [50, 8, 72], self-supervision [8], entropy-minimization [27], consistency-regularization [80, 49, 87, 23], adversarial training [63], data augmentation [91], and combinations thereof [7, 83, 6].”