Realistic Evaluation of Semi-Supervised Learning Algorithms

By Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, Ian J. Goodfellow. 2018

866 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Similar concerns have been raised in the field of semi-supervised learning (Oliver et al., 2018).”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “Firstly, we follow the settings in [19] and employ Wide-ResNet-282 [26, 27] as the backbone model and evaluate UDA with varied supervised data sizes.”
Do Better ImageNet Models Transfer Better? (2018, methodology): “When investigating the effect of preprocessing upon performance, we tested the performance of ResNet-50 v1, ResNet-101 v1, and ResNet-152 v1 models retrained with the preprocessing from Szegedy et al. [6] with label smoothing of 0.1 and an input size of 299× 299.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “However, we use this setting, which is common in prior art (Oliver et al., 2018), because it signiﬁcantly reduces evaluation noise.”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “[28] discovered that reported baselines trained on labeled examples alone are unfairly weak, perhaps given that there is not a strong community behind tuning those baselines.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, neutral): “Aside from the representation learning paradigm, there is a large and diverse set of approaches for semi-supervised learning, we refer readers to [52–54] for surveys of classical approaches.”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, positive): “In the quest to draw sound and robust conclusions, we join other recent papers in highlighting the under-acknowledged existence of conclusion replication failure [47, 36, 39, 26, 19].”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “73% [2] in the standard experimental setting from [31].”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “Unless otherwise noted, in all experiments we use the “Wide ResNet-28” model from [35].”
True Few-Shot Learning with Language Models (2021, methodology): “Inspired by prior work [77, 78], we offer recommendations for future work in true few-shot learning: • Report all hyperparameters (prompts) considered and the hyperparameter selection criteria.”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “This makes the recipe a realistic option for few-shot learning settings where validation sets are tiny by definition [31, 32].”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “38%, probably due to hyperparameter tuning) [65].”
Self-Training for End-to-End Speech Recognition (2019, background): “This prior work in semisupervised ASR with end-to-end models tends to build from a weak or poorly tuned supervised baseline model, a common issue with semi-supervised learning in general [37].”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “Secondly, it has been shown in the semi supervised setting that out-of-distribution samples can cause significant performance drop when used for training (Oliver et al., 2018).”
Continual semi-supervised learning through contrastive interpolation consistency (2021, methodology): “only a few annotations at disposal: as shown in our experiments, this encourages the model to overfit the limited supervised data available [35].”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, methodology): “SSL on GAN is generally built on a two-network architecture, that the discriminator is responsible for both sample quality and label prediction, including CatGAN [38], Improved-GAN [34], CGAN [23] and its variants [30, 26].”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, methodology): “Some recent methods [9,37] train neural networks with a small number of labeled examples.”
Invertible Residual Networks (2018, background): “• Semi-supervised learning from few labeled examples (Oliver et al., 2018; Kingma et al., 2014)”
Meta Pseudo Labels (2020, background): “However, since these methods do not share the same controlled environment, the comparison to them is not direct, and should be contextualized (Oliver et al., 2018).”
Interpolation Consistency Training for Semi-Supervised Learning (2019, methodology): “To justify the use of a SSL algorithm, one must compare its performance against the state-of-the-art supervised learning algorithm (Oliver et al., 2018).”