MixMatch: A Holistic Approach to Semi-Supervised Learning

By David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel. 2019

3,592 citations recorded by the source corpus.

Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “A similar consistency idea, but for class label prediction, has been explored in other contexts such as semisupervised learning (Xie et al., 2019; Berthelot et al., 2019).”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “We compare UDA with Pseudo-Label [36], an algorithm based on self-training, Virtual adversarial training (VAT) [44], an algorithm that generates adversarial Gaussian perturbations on input, Π -Model [35], which combines simple input augmentation with hidden state perturbations, Mean Teacher [58], wh”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “A common approach is to increase the source side data, for instance by adding modifications or sampling from generative models (Hannun et al., 2014; Jia & Liang, 2016; Krizhevsky et al., 2017; Antoniou et al., 2017; Yasunaga et al., 2018; Yu et al., 2018a; Lee et al., 2019; Berthelot et al., 2019; X”
Shortcut learning in deep neural networks (2020, background): “Extreme dataaugmentation strategies are also the core ingredient of the most successful semi-supervised [124] and self-supervised learning approaches to date [125, 126].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Apart from self-training, another important line of work in semi-supervised learning [9, 85] is based on consistency training [6, 4, 53, 36, 70, 45, 41, 51, 10, 12, 49, 2, 38, 72, 74, 5, 81].”
Learning Representations by Maximizing Mutual Information Across Views (2019, neutral): “Examples of such results include [Ji et al., 2019] and [Berthelot et al., 2019], which achieve 88.8% and 94.4% accuracy respectively.”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “Semi-supervised learning algorithms are typically [28, 22, 2, 39, 1, 21] evaluated on small-scale datasets such as CIFAR-10 [17] and SVHN [23].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “This approach uses unlabeled data in a task-speciﬁc way to encourage class label prediction consistency on unlabeled data among different models [11, 12, 2] or under different data augmentations [13–15].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In the semi-supervised setting [55, 56], an unsupervised loss is combined with a classification loss over a handful of labels to ground the training [19, 20, 57, 58, 59, 60, 61, 62].”
Grafit: Learning fine-grained image representations with coarse labels (2020, background): “Recent works on semi-supervised learning [4, 5, 48, 62, 66, 69] rely on both supervised and self-supervised losses to get information from unlabelled data.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “Other works employ the Mix&Cut approaches for consistency regularization [44, 4, 51], i.”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “Following [3], we use a WRN-37-2 network (comprising 23.8M parameters).”
Semi-Supervised Neural Architecture Search (2020, methodology): “Furthermore, we will explore advanced semi-supervised learning methods [33, 3] to improve SemiNAS.”
Faster AutoAugment: Learning Augmentation Strategies using Backpropagation (2019, background): “assiﬁers. [27, 28] use GANs to modify the outputs of simulators to look like real objects. Automating data augmentation can also be applied to representation learning such as semi-supervised learning [4, 33] and domain generalization [32]. 3. Preliminaries In this section, we describe the common bas”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “Several approaches mitigate such an issue through dataaugmentation techniques, either by generating different versions of the same buffer datapoint [7, 15] or by combining different examples into a single one [14, 11].”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “CL [31] ✓ ✗ ✗ ✗ SSL [11] ✗ ✓ ✗ ✗ CFSL [6] ✓ ✗ ✓ ✗ SSCL [60] ✓ ✓ ✗ ✗ SSFSL [47] ✗ ✓ ✓ ✓ SETS (Ours) ✓ ✓ ✓ ✓ We summarize and compare different benchmarks in Table 1, including continual learning (CL), semi-supervised learning (SSL), continual few-shot learning (CFSL), semi-supervised continual learni”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, methodology): “Thus, we consider them as unlabeled data and use consistency regularization, which is widely used in the semisupervised learning literature [7, 35].”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, methodology): “Consistency regularization techniques are used to ensure that a model’s output is invariant to various augmentations, which is common in semisupervised learning [2, 19, 30, 32, 39].”
Continual semi-supervised learning through contrastive interpolation consistency (2021, methodology): “ment our proposal with MixMatch [28]: the predictions of the network are not meant as training targets, but rather as means for applying consistency regularization [6, 29].”