Weight-averaged consistency targets improve semi-supervised deep learning results

By Antti Tarvainen, Harri Valpola. 2017

1,218 citations recorded by the source corpus.

The recently proposed temporal ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, temporal ensembling becomes unwieldy when using large datasets. To overcome this problem, we propose a method that averages model weights instead of label predictions. As an additional benefit, the method improves test accuracy and enables training with fewer labels than earlier methods. We report state-of-the-art results on semi-supervised SVHN, reducing the error rate from 5.12% to 4.41% with 500 labels, and achieving 5.39% error rate with 250 labels. By using extra unlabeled data, we reduce the error rate to 2.76% on 500-label SVHN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Text Chunking : We use the CoNLL-2000 data (Tjong Kim Sang and Buchholz, 2000).”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “To our knowledge, Mean Teacher [38] currently holds the state-of-the-art result on ILSVRC-2012 when using only 10% of the labels.”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “We tried applying the state of the art technique by Tarvainen and Valpola (2017) without much improvement in sample complexity while facing more training stability issues.”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “This motivated researchers to develop many variants of it to various domains, and many researchers studied ways to better teach the student (Frosst and Hinton 2017; Pham et al. 2018; Radosavovic et al. 2018; Tan et al. 2018; Tarvainen and Valpola 2017).”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, methodology): “Thus, we consider them as unlabeled data and use consistency regularization, which is widely used in the semisupervised learning literature [7, 35].”
Static-Dynamic Co-Teaching for Class-Incremental 3D Object Detection (2021, methodology): “We adopt a rampup technique (Tarvainen and Valpola 2017) to schedule the respective contributions of λd and λc.”
Continual semi-supervised learning through contrastive interpolation consistency (2021, methodology): “This principle can be applied either by comparing the predictions on the same exemplar by different learners [27, 6] or the predictions on different augmentations of the same data-point by the same learner [28].”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, methodology): “Notably, we learn various strong semi-supervised classifiers, including MT [41], VAT [24], PL [14] and PI [13] models in SSCL.”
Meta Pseudo Labels (2020, background): “On the other hand, label propagation losses typically enforce that the model is invariant against certain transformations of the data such as data augmentations, adversarial attacks, or proximity in the latent space [35, 64, 44, 5, 76, 30, 71, 58, 32, 51, 20].”
Interpolation Consistency Training for Semi-Supervised Learning (2019, methodology): “Results We provide the results for CIFAR10 and SVHN datasets using CNN-13 architecture in Tables 1 and 2, respectively.”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, methodology): “Our label guessing method improves over (Tarvainen and Valpola, 2017) which utilizes teacher and student models to predict labels for unlabeled data, and UDA (Xie et al., 2019) that just uses p ( x ui ) as generated labels.”
ImageNet-21K Pretraining for the Masses (2021, methodology): “where KDLoss is a standard measurement for the distance between distributions, that can be chosen 258 as Kullback-Leibler divergence [19, 57], or as MSE loss [2, 51].”
Unlabeled Data Improves Adversarial Robustness (2019, background): “A recent family of approaches operate by enforcing consistency in the model’s predictions under various perturbations of the unlabeled data [30, 51], or over the course of training [45, 40, 23].”
End-to-End Semi-Supervised Object Detection with Soft Teacher (2021, methodology): “As the training progresses, the teacher model is continuously updated by the student model, and we follow the common practices [29, 26] that the teacher model is updated by exponential moving average (EMA) strategy.”
Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities (2017, background): “However, more recent results (Sajjadi et al. 2016; Laine and Aila 2016; Tarvainen and Valpola 2017) show better performance can be obtained by different categories of semisupervised methods from the GAN-based models.”
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2019, methodology): “…proximal point methods in existing literature on optimization, including vanilla proximal point method proposed in Rockafellar (1976), generalized proximal point method (Teboulle, 1997; Eckstein, 1993), accelerated proximal point method, and other variants (Güler, 1991, 1992; Parikh et al., 2014).”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, methodology): “The encoding of the unmasked training sample is parameterized by an exponentially moving average (EMA) of the model parameters (θ; Tarvainen & Valpola 2018; Grill et al. 2020; Caron et al. 2021) where the weights of the model in target-mode ∆ are: ∆← τ∆ + (1− τ) θ We use a schedule for τ that…”