Unsupervised Data Augmentation for Consistency Training

By Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, Quoc V. Le. 2019

2,706 citations recorded by the source corpus.

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods such as RandAugment and back-translation, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 5.43 with only 250 examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. Code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “A similar consistency idea has been explored in other contexts such as semi-supervised learning (Xie et al., 2019; Berthelot et al., 2019).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “A common approach is to increase the source side data, for instance by adding modifications or sampling from generative models (Hannun et al., 2014; Jia & Liang, 2016; Krizhevsky et al., 2017; Antoniou et al., 2017; Yasunaga et al., 2018; Yu et al., 2018a; Lee et al., 2019; Berthelot et al., 2019; X”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “…butter-fingers 9 with two thresholds 0.02 and 0.05, which respectively correspond to lower and higher error frequencies; (b) removal of the final punctuation sign (if any); (c) substitution of the input text by a backtranslated version, using the backtranslation implementation by Xie et al. (2020).”
Improving and Simplifying Pattern Exploiting Training (2021, background): “However, performance can suffer when there is very limited labeled data available for a downstream task (Xie et al., 2020; Chen et al., 2020).”
Making Pre-trained Language Models Better Few-shot Learners (2021, background): “learning (Miyato et al., 2017; Xie et al., 2020; Chen et al., 2020), where a set of unlabeled examples are given; (2) meta-learning (Yu et al.”
Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “…NLP commonly require large sets of examples from re-lated tasks (Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019; Ye et al., 2020), parallel data for consistency training (Xie et al., 2020; Chen et al., 2020), or highly specialized methods tailored towards a speciﬁc task (Laban et al., 2020).”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Similar techniques are used in Virtual AT [46], TRADES [84], and UDA [75].”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “2 [75] Qizhe Xie, Zihang Dai Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V.”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “[62] is similar to Multigrain, except that the Kullback-Leibler divergence replaces the triplet loss.”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “This has led to a plethora of SSL methods that are designed for deep networks [28, 39, 21, 43, 3, 45, 2, 22, 38].”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Attempts to remedy this have focused either on regularization (He et al., 2020) or on variations of consistency (Xie et al., 2019; Sohn et al., 2020) for a given task–such as round-trip consistency of question generation and answer prediction (Alberti et al., 2019; Puri et al., 2020) for QA or…”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, methodology): “This makes it challenging to apply data augmentation (Chen et al., 2020; Xie et al., 2020) and self-training techniques (Meng et al.”
A Survey of Data Augmentation Approaches for NLP (2021, methodology): “It has received active attention in recent machine learning (ML) research in the form of well-received, general-purpose techniques such as UDA (Xie et al., 2020) (3.1), which used backtranslation (Sennrich et al., 2016), Au-toAugment (Cubuk et al., 2018), and RandAugment (Cubuk et al., 2020), and M ”