A Survey of Data Augmentation Approaches for NLP

By Steven Y. Feng, Varun Prashant Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, Eduard H. Hovy. 2021

1,000 citations recorded by the source corpus.

Data augmentation has recently seen increased interest in NLP due to more work in low-resource domains, new tasks, and the popularity of large-scale neural networks that require large amounts of training data. Despite this recent upsurge, this area is still relatively underexplored, perhaps due to the challenges posed by the discrete nature of language data. In this paper, we present a comprehensive and unifying survey of data augmentation for NLP by summarizing the literature in a structured manner. We first introduce and motivate data augmentation for NLP, and then discuss major methodologically representative approaches. Next, we highlight techniques that are used for popular NLP applications and tasks. We conclude by outlining current challenges and directions for future research. Overall, our paper aims to clarify the landscape of existing literature in data augmentation for NLP and motivate additional work in this area. We also present a GitHub repository with a paper list that will be continuously updated at https://github.com/styfeng/DataAug4NLP

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A variety of works have proposed using LMs for data generation (Schick and Schütze, 2021; Wang et al., 2021; Liu et al., 2022; Meng et al., 2023) or augmentation (Feng et al., 2021; Yang et al., 2020; Mekala et al., 2022).”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “translation or synonymous word replacement (Feng et al., 2021; Chen et al., 2021a) were shown to be effective for increasing NLP models’ accuracy and/or coverage.”
Natural language processing applied to mental illness detection: a narrative review (2022, result): “In addition, feature enrichment and data augmentation 190 are useful to achieve comparable results.”
Is GPT-3 a Good Data Annotator? (2022, methodology): “One approach to address the need for labeled data is through data augmentation (Feng et al., 2021; Meng et al., 2022; Chen et al., 2023), which involves generating new data by modifying existing data points using transformations based on prior knowledge about the problem’s structure (Yang et al.,…”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, positive): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “Compared to traditional knowledge distillation algorithms (Gou et al., 2021), data augmentation (DA) (Feng et al., 2021) has emerged as a prevalent paradigm to achieve knowledge distillation of LLMs, where a small seed of knowledge is used to prompt the LLM to generate more data with respect to a…”
Data Augmentation techniques in time series domain: a survey and taxonomy (2022, background): “In the latest times, several high-quality data augmentation review papers have been published [12, 13, 14].”
AugLy: Data Augmentations for Robustness (2022, background): “Data augmentations are a key component in the computer vision model development life cycle[24], and are also becoming increasingly prevalent in other domains[7].”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “Recently, several surveys also explore the data augmentation techniques for NLP (Hedderich et al., 2020; Feng et al., 2021).”
DialFact: A Benchmark for Fact-Checking in Dialogue (2021, methodology): “, 2021) and improve the complexity of test sets (Sakaguchi et al., 2021; Feng et al., 2021).”
Data-centric Artificial Intelligence: A Survey (2023, neutral): “Some previous surveys only focus on specific scopes of data-centric AI, such as data augmentation [74, 214, 249], data labeling [283], and feature selection [138].”
DataLab: A Platform for Data Analysis and Intervention (2022, methodology): “(ii) Data augmentation (Wei and Zou, 2019; Dhole et al., 2021; Feng et al., 2021).”
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023, methodology): “We evaluate the debiasing performance of five different methods (see Section 5.3), including three loss-based methods: Language Model Debiasing (Qian et al., 2019), Attribute Distance Debiasing (Lauscher et al., 2020), and Hard Debiasing (Bordia and Bowman, 2019; Barikeri et al., 2021), as well as t”
A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, methodology): “It has also been used for natural language processing [201, 202] and text classification [203].”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, background): “Most data augmentation techniques create examples through transformations of existing examples which are based on prior task-specific knowledge (Feng et al., 2021; Chen et al., 2021).”