NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

By Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan and 113 more. 2021

104 citations recorded by the source corpus.

Data augmentation is an important method for evaluating the robustness of and enhancing the diversity of training data for natural language processing (NLP) models. In this paper, we present NL-Augmenter, a new participatory Python-based natural language (NL) augmentation framework which supports the creation of transformations (modifications to the data) and filters (data splits according to specific features). We describe the framework and an initial set of 117 transformations and 23 filters for a variety of NL tasks annotated with noisy descriptive tags. The transformations incorporate noise, intentional and accidental human mistakes, socio-linguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguous to humans. We demonstrate the efficacy of NL-Augmenter by using its transformations to analyze the robustness of popular language models. We find different models to be differently challenged on different tasks, with quasi-systematic score decreases. The infrastructure, datacards, and robustness evaluation results are publicly available on GitHub for the benefit of researchers working on paraphrase generation, robustness analysis, and low-resource NLP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Dhole et al. (2021) expand their framework to over 100 different transformations that include dialectal variations, OCR errors, and others that can be used to create more realistic scenarios.”
Multi-lingual Evaluation of Code Generation Models (2022, methodology): “We evaluate the robustness of models across r-MBXP datasets perturbed by common transformations in NL-Augmenter (Dhole et al., 2021), a standard collection of data augmentations for robustness evaluation on text.”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, background): “GEM further introduced evaluation suites (Mille et al., 2021; Dhole et al., 2021) that are compatible with its datasets and test various robustness and fairness aspects of models. ar X iv :2 20 6.”
How Many Data Samples is an Additional Instruction Worth? (2022, methodology): “They are allowed to use automated tools such as Semantic Control (Ross et al., 2021), Text Style Transfer (Reif et al., 2021), NL-Augmenter (Dhole et al., 2021).”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, methodology): “To this end, we employ the perturbation methods deﬁned in HELM, which were originally inspired by NL-Augmenter [55].”
Intriguing Properties of Compression on Multilingual Models (2022, methodology): “The technique we use to study robustness expands on studies that perturb training (Yaseen and Langer, 2021; Dai and Adel, 2020) or evaluation data (Dhole et al., 2021) in NER by introducing perturbations speciﬁc to languages, language families, and scripts.”
DataLab: A Platform for Data Analysis and Intervention (2022, background): “(2) Operation Standardization : There are a number of well-designed packages for data-oriented operations such as preprocessing (Loper and Bird, 2002; Manning et al., 2014; Kudo, 2020) or editing (Ribeiro et al., 2021; Dhole et al., 2021).”
ReCode: Robustness Evaluation of Code Generation Models (2022, methodology): “For docstrings, we use the NLAugmenter (Dhole et al., 2021) library which is designed for data augmentation and robustness evaluation on text.2 We carefully select ten transformations, including character-level, wordlevel and sentence-level ones, that are likely to preserve semantic similarity.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “Back translation (Sen-nrich et al., 2016; Dhole et al., 2021) is a popular strategy for augmenting training data, but given that our goal is to improve multilingual generation, we simply translated our training datasets into our target languages without translating them back.”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “In addition, we perturb the natural language descriptions in docstrings with the back-translation method of NL-Augmenter [17], inspired by ReCode [57].”