Interpolation Consistency Training for Semi-Supervised Learning

By Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, David Lopez-Paz. 2019

919 citations recorded by the source corpus.

We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. In classification problems, ICT moves the decision boundary to low-density regions of the data distribution. Our experiments show that ICT achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark datasets. Our theoretical analysis shows that ICT corresponds to a certain type of data-adaptive regularization with unlabeled points which reduces overfitting to labeled points under high confidence values.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, background): “In Appendix C, we also compare UDA with recently proposed methods including ICT [61] and mixmixup [19] which enforce interpolation smootheness similar to mixup [69] and LGA + VAT [28], an algorithm based on gradient similarity.”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “Semi-supervised learning algorithms are typically [28, 22, 2, 39, 1, 21] evaluated on small-scale datasets such as CIFAR-10 [17] and SVHN [23].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Another family of methods are based on label consistency regularization [55–58, 14, 13, 59, 15], where unlabeled examples are directly used as a regularizer to encourage task prediction consistency.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “Other works employ the Mix&Cut approaches for consistency regularization [44, 4, 51], i.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “MixUp has been previously applied to semi-supervised learning; in particular, the concurrent work of [45] uses a subset of the methodology used in MixMatch.”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “Notable examples of this consistency based category are [4, 5, 27,33,35,43,49,52,60].”
Meta Pseudo Labels (2020, background): “On the other hand, label propagation losses typically enforce that the model is invariant against certain transformations of the data such as data augmentations, adversarial attacks, or proximity in the latent space [35, 64, 44, 5, 76, 30, 71, 58, 32, 51, 20].”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, neutral): “…have been recently proposed for supervised learning (Zhang et al., 2017; Verma et al., 2019a) and semi-supervised learning (Berthelot et al., 2019; Verma et al., 2019b) for image-format data by overlaying two input images and combining image labels as virtual training data and have achieved…”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, background): “, 2017) which trains a model on linear interpolants of inputs and labels has recently been applied to SSL (Berthelot et al., 2019; Verma et al., 2019).”
Selfie: Self-supervised Pretraining for Image Embedding (2019, background): “Successful recent semi-supervised learning methods for deep learning are based on consistency training (Miyato et al., 2018; Sajjadi et al., 2016; Laine and Aila, 2016; Verma et al., 2019; Xie et al., 2019).”
Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning (2020, background): “This setting has been revisited in the context of semi-supervised learning where the teacher is no longer ﬁxed but evolves during training [45, 65, 70].”
A Fourier-based Framework for Domain Generalization (2021, background): “[43] also demonstrate that a interpolation consistency with MixUp [49] samples could be helpful.”
Efficient Semi-supervised Gross Target Volume of Nasopharyngeal Carcinoma Segmentation via Uncertainty Rectified Pyramid Consistency (2020, background): “In addition, we furthercompared our methods with five state-of-the-art semi-supervised segmentation meth- ods, including Mean Teacher (MT) [21], Interpolation Consistency Training (ICT) [22], Entropy Minimization (EM) [23], Uncertainty Aware Mean Teacher (UAMT) [27] and Deep Adversarial Network (DAN”
Hard Negative Mixing for Contrastive Learning (2020, background): “At the same time, data mixing techniques operating at either the pixel [41, 49, 50] or the feature level [40] help models learn more robust features that improve both supervised and semi-supervised learning on subsequent (target) tasks.”
Unsupervised Data Augmentation (2019, methodology): “When compared with the previous SOTA model ICT [Verma et al., 2019], UDA reduces the error rate from 7.”
On Adversarial Mixup Resynthesis (2019, methodology): “…mixing based methods have shown to improve strong baselines in supervised learning (Zhang et al., 2018; Verma et al., 2018) and semi-supervised learning (Verma et al., 2019; Berthelot et al., 2019), there has been relatively less exploration of these methods in the context of unsupervised learning.”