There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average

By Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson. 2018

262 citations recorded by the source corpus.

Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, background): “Works in this category include Mean Teacher (Tarvainen & Valpola, 2017), fast-Stochastic Weight Averaging (Athiwaratkun et al., 2018) and Smooth Neighbors on Teacher Graphs (Luo et al.”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “visit a simple paradigm: pre-train on a large supervised source dataset, and ne-tune the weights on the target task. Numerous improvements to deep network training have recently been introduced, e.g. [55,62,26,35,22,1,64,67,54,60]. We aim not to introduce a new component or complexity, but to provid”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “Semi-supervised learning algorithms are typically [28, 22, 2, 39, 1, 21] evaluated on small-scale datasets such as CIFAR-10 [17] and SVHN [23].”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, background): “Another class of approaches was derived with the idea of imposing similar predictions from the network for two samples obtained with different input perturbations [3,4,27,33, 35, 41, 47, 50, 60].”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, methodology): “Because mean teacher (MT) [41] can be easily extended to many recent works [9, 2, 21], we choose it as the base model of SSL classifier in ORDisCo.”
Interpolation Consistency Training for Semi-Supervised Learning (2019, methodology): “Consistency-regularization semi-supervised learning methods (Athiwaratkun et al., 2019; Laine & Aila, 2017; Luo et al., 2018; Miyato et al., 2018; Sajjadi et al., 2016; Tarvainen & Valpola, 2017) encourage that realistic perturbations u+ δ of unlabeled samples u should not change the model predictio”
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2020, background): “We thus investigate the effect of using a consistency loss (Xie et al., 2019; Athiwaratkun et al., 2019) for few-shot training which enforces consistency between the original and round-trip translated documents with respect to the original summary.”
SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations (2020, methodology): “The Lagrange multiplier form of (14 tive models and data augmentation, FixMatch (Sohn et al. 2020), ReMixMatch (Berthelot et al. 2019a) and UDA (Xie et al. 2019) combining data augmentation and consistency learning, and SWSA (Athiwaratkun et al. 2019) combining Π − model and fast-SWA.”
Semi-Supervised Learning with Normalizing Flows (2019, methodology): “While FlowGMM does not yet outperform the most powerful discriminative approaches for semi-supervised image classification (Athiwaratkun et al., 2019; Verma et al., 2019), we believe it is a promising step towards making fully generative approaches more practical for semi-supervised tasks.”
Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, methodology): “In the SSL literature, recent semi-supervised learning methods have achieved remarkable results [7, 8, 9, 10, 11, 12, 13, 14].”
High-Quality Self-Supervised Deep Image Denoising (2019, methodology): “Impulse α ∈ [0, 1] Baseline, N2C no 31.”
Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning (2021, background): “In consistency based semi- and self-supervised learning, training could be less stable [1], where the EMA-teacher framework with improved generalization can help.”
An Overview of Deep Semi-Supervised Learning (2020, background): “[5] observed that Π-Model and Mean Teacher continue taking significant steps in the weight space at the end of training, given that the models stochastic gradient descent (SGD) traverses a large flat region of the weight space late in training, ending up iterating at the periphery of this flat regio”
Guided Collaborative Training for Pixel-wise Semi-Supervised Learning (2020, unknown): “For example, Temporal Model [25] accumulates the predictions over epochs as the targets; Mean Teacher [44] defines an explicit model by exponential moving average to generate the targets; FastSWA [4] further averages the models between epochs to produce better targets.”
Adversarially Robust Generalization Just Requires More Unlabeled Data (2019, methodology): “Following many previous works (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Miyato et al., 2018; Athiwaratkun et al., 2019), we sample 5k/10k labeled data from the training set and use them as labeled data.”