Data Augmentation using Pre-trained Transformer Models

By Varun Kumar, Ashutosh Choudhary, Eunah Cho. 2020

411 citations recorded by the source corpus.

Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Big Bird: Transformers for Longer Sequences (2020, background): “The pretraining has led to signiﬁcant improvement in low data regime downstream tasks [51] as well as tasks with sufﬁcient data [102] and thus have been a major force behind the ubiquity of transformers in contemporary NLP.”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “unfair outcomes and negative experiences to the end users of the downstream applications [11, 18, 20, 33, 34].”
Generating Datasets with Pretrained Language Models (2021, neutral): “Some approaches augment existing datasets with automatically generated examples (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Yang et al., 2020; Mohapatra et al., 2020; Kumar et al., 2021), but in contrast to our work, all of these approaches require that there already exists a labele”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “Some works explore automatic data synthesis/expansion by ﬁnetuning LMs on existing labeled data (Anaby-Tavor et al., 2020; Papaniko-laou and Pierleoni, 2020; Kumar et al., 2020; Yang et al., 2020), but are limited by data quality.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Recently, generative language models have been used to synthesize examples for various NLP tasks (Kumar et al., 2020; Anaby-Tavor et al., 2020; Puri et al., 2020; Yang et al., 2020).”
Towards Zero-Label Language Learning (2021, background): “…modiﬁcation has been applied to diversify training samples, more recently generative data augmentation has gained popularity due to the progress made in language modeling (Anaby-Tavor et al., 2019; Papanikolaou and Pier-leoni, 2020; Juuti et al., 2020; Lee et al., 2021; Kumar et al., 2021).”
Pre-trained models for natural language processing: A survey (2020, background): “Indeed, mining knowledge from PTMs can be more flexible, such as feature extraction, knowledge distillation [203], data augmentation [192, 89], using PTMs as external knowledge [132].”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, background): “novel examples for text classification and reading comprehension (Anaby-Tavor et al., 2020; Kumar et al., 2020; Puri et al., 2020).”
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020, methodology): “Other methods focus on fine-tuning contextual language models (Kobayashi, 2018; Wu et al., 2019b; Kumar et al., 2020) or large generative models (AnabyTavor et al., 2020; Yang et al., 2020; Kumar et al., 2020) to generate synthetic examples.”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, methodology): “They either fine-tune the generator on the initial training set with the standard maximum likelihood objective (Anaby-Tavor et al., 2020; Kumar et al., 2020) or use the training samples as demonstrations (Yoo et al.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “Early efforts achieve this goal with fine-tuned generative models (Anaby-Tavor et al., 2020; Puri et al., 2020; Kumar et al., 2020; Lee et al., 2021).”
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021, background): “For example, BART has been proven to be effective in populating text samples for data-scarce labels (Kumar et al., 2020).”
Quantifying the Evaluation of Heuristic Methods for Textual Data Augmentation (2020, methodology): “…et al., 2016; Fadaee et al., 2017; Xie et al., 2019) used back-translation, and (Hu et al., 2017; Iyyer et al., 2018; Anaby-Tavor et al., 2020; Kumar et al., 2020) However, with all these textual augmentation options, trying all of them for a (classiﬁer) training task might be impractical,…”
Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classification (2020, background): “The expectation is that large pre-trained LMs, being trained on massive amount of data, can produce a sensible text span given a particular sentence context, and possibly produce slot values that do not occur in D.”
Improving CLIP Training with Language Rewrites (2023, background): “Common practices in data augmentation include synonym replacement [58], random masking [30], and back translation [50].”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “At the same time, limited by data resources, augmentation methods of low resource languages are scarce [29], although they have more demand for data augmentation.”
A Survey on Data Augmentation for Text Classification (2021, neutral): “EDA is very popular in the research field and was used as a method for comparisons in the works of Qiu et al. [42], Huong and Hoang [43], Anaby-Tavor et al. [44], Kumar et al. [45], Bayer et al. [46], Feng et al. [39], Luu et al. [47], and Kashefi and Hwa [48].”
Textual Data Augmentation for Efficient Active Learning on Tiny Datasets (2020, background): “Kumar et al. (2020) further explores this approach with different trans-former based models (Vaswani et al., 2017) for data generation.”