FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

By Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Colin Raffel. 2020

4,872 citations recorded by the source corpus.

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “[5,50] used semi-supervised learning to attain strong performance on CIFAR-10 and SVHN using only 40 or 250 labels.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Another family of methods are based on label consistency regularization [60–62, 39, 14, 13, 63, 15], where unlabeled examples are directly used as a regularizer to encourage task prediction consistency.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In the semi-supervised setting [55, 56], an unsupervised loss is combined with a classification loss over a handful of labels to ground the training [19, 20, 57, 58, 59, 60, 61, 62].”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “3 [64] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Semi-supervised methods on full ImageNet FixMatch [46] RN50 24M - 71.”
Grafit: Learning fine-grained image representations with coarse labels (2020, background): “Recent works on semi-supervised learning [4, 5, 48, 62, 66, 69] rely on both supervised and self-supervised losses to get information from unlabelled data.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Attempts to remedy this have focused either on regularization (He et al., 2020) or on variations of consistency (Xie et al., 2019; Sohn et al., 2020) for a given task–such as round-trip consistency of question generation and answer prediction (Alberti et al., 2019; Puri et al., 2020) for QA or…”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “Another big theme in semi-supervised learning is consistency [34–36].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “On semi-supervised learning (Table 1), SwAV outperforms other self-supervised methods and is on par with state-of-the-art semi-supervised models [44], despite the fact that SwAV is not specifically designed for semi-supervised learning.”
Class Gradient Projection For Continual Learning (2022, background): “Supervised contrastive learning (SCL) is extended from the standard contrastive loss by incorporating the label information to construct positive and negative pairs [37].”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, methodology): “Consistency regularization techniques are used to ensure that a model’s output is invariant to various augmentations, which is common in semisupervised learning [2, 19, 30, 32, 39].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Yang et al. [216] combined regularization and parameter isolation based methods.”
Continual semi-supervised learning through contrastive interpolation consistency (2021, background): “ers [37, 9] or the predictions on different augmentations of the same data-point by the same learner [8, 38, 39, 40].”
Reinforcement Learning with Augmented Data (2020, methodology): “For unsupervised and semi-supervised learning, several unsupervised data augmentation techniques have been proposed [18, 16, 26].”