ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

By David Berthelot, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Kihyuk Sohn, Colin Raffel. 2019

796 citations recorded by the source corpus.

We improve the recently-proposed "MixMatch" semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input. To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between $5\times$ and $16\times$ less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach $93.73\%$ accuracy (compared to MixMatch's accuracy of $93.58\%$ with $4{,}000$ examples) and a median accuracy of $84.92\%$ with just four labels per class. We make our code and data open-source at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “[5,50] used semi-supervised learning to attain strong performance on CIFAR-10 and SVHN using only 40 or 250 labels.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Apart from self-training, another important line of work in semi-supervised learning [12, 101] is based on consistency training [5, 64, 47, 83, 56, 52, 62, 13, 16, 60, 2, 49, 86, 89, 8, 96, 46, 7].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Recent work has shown in-domain beneﬁts of semi-supervised learning (H´enaff et al., 2019; Berthelot et al., 2020); our study reveals that semi-supervised methods also learn good representations for transfer to new tasks.”
Grafit: Learning fine-grained image representations with coarse labels (2020, background): “Recent works on semi-supervised learning [4, 5, 48, 62, 66, 69] rely on both supervised and self-supervised losses to get information from unlabelled data.”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, methodology): “Recently, sample mixing techniques, such as MixUp, were also investigated in the context of semi-supervised learning for improving the model performance on low sample density regions [5,6,31].”
Meta Pseudo Labels (2020, methodology): “For this finetuning phase, we use SGD with a fixed learning rate of 10−5 and a batch size of 512, running for 2,000 steps for ImageNet-10% and 1,000 steps for CIFAR-10 and SVHN.”
End-to-End Semi-Supervised Object Detection with Soft Teacher (2021, background): “Recently, some works [32, 3, 2, 26] explore the importance of data augmentation in semi-supervised learning, which inspire us to use the weak augmentation to generate pseudo-labels and the strong augmentation for the learning of detection models.”
An Overview of Deep Semi-Supervised Learning (2020, methodology): “[12] propose to improve MixMatch by introducing two new techniques: distribution alignment and augmentation anchoring.”
Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations. (2020, methodology): “Other semi-supervised learning methods not based on self-supervised learning include Unsupervised Data Augmentation (UDA) [53] and the MixMatch trilogy of work [3, 4, 45].”
One Million Scenes for Autonomous Driving: ONCE Dataset (2021, methodology): “Methods on semi-supervised learning are mainly of two branches: The first branch of methods try to annotate those unlabeled data with pseudo labels [25, 3, 2, 33] by self-training [47] or teacher model [41].”
Unbiased Teacher for Semi-Supervised Object Detection (2021, methodology): “As an alternative, Semi-Supervised Learning (SSL) methods have received growing attention (Sohn et al., 2020a; Berthelot et al., 2020; 2019; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Sajjadi et al., 2016; Lee, 2013; Grandvalet & Bengio, 2005).”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, background): “There are several recent semi supervised learning methods [45, 64, 6, 52, 5, 2, 9, 60] that produce pseudo labels for unlabeled data using a model trained on a limited labeled set.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, methodology): “Inspired by the advances of AutoAugment [76], ReMixMatch [27] introduces CTAugment to learn an automated augmentation policy.”
Recovering Petaflops in Contrastive Semi-Supervised Learning of Visual Representations (2020, methodology): “Other semi-supervised learning methods not based on self-supervised learning include Unsupervised Data Augmentation (UDA) [50] and the MixMatch trilogy of work [4, 3, 43].”
Give Me Your Trained Model: Domain Adaptive Semantic Segmentation without Source Data (2021, methodology): “[34] uses auto-augmentation to generate augmented inputs, and [35] extends the idea by dividing the augmentation set into strong and weak operations.”
VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain (2020, methodology): “MixMatch [29] and ReMixMatch [30] combine entropy minimization with consistency regularization in one unified framework with MixUp [10] as the data augmentation method.”
ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning (2020, methodology): “Some methods of semi-supervised learning for classification [37, 33, 2, 1] integrate entropy minimization in the consistency regularization framework.”
A Comprehensive Survey of Image Augmentation Techniques for Deep Learning (2022, background): “Furthermore, semi-28 supervised learning beneﬁts from image augmentation [79, 110, 111].”
Milking CowMask for Semi-Supervised Image Classification (2020, methodology): “MixMatch (Berthelot et al., 2019b) guesses labels for un-supervised samples by sharpening the averaged predictions from multiple rounds of standard augmentation and blends images and corresponding labels (ground truth for supervised samples, guesses for unsupervised) using MixUp (Zhang et al.,…”