Neural Data Augmentation via Example Extrapolation

By Kenton C. T. Lee, Kelvin Guu, Luheng He, Tim Dozat, Hyung Won Chung. 2021

75 citations recorded by the source corpus.

In many applications of machine learning, certain categories of examples may be underrepresented in the training data, causing systems to underperform on such"few-shot"cases at test time. A common remedy is to perform data augmentation, such as by duplicating underrepresented examples, or heuristically synthesizing new examples. But these remedies often fail to cover the full diversity and complexity of real examples. We propose a data augmentation approach that performs neural Example Extrapolation (Ex2). Given a handful of exemplars sampled from some distribution, Ex2 synthesizes new examples that also belong to the same distribution. The Ex2 model is learned by simulating the example generation procedure on data-rich slices of the data, and it is applied to underrepresented, few-shot slices. We apply Ex2 to a range of language understanding tasks and significantly improve over state-of-the-art methods on multiple few-shot learning benchmarks, including for relation extraction (FewRel) and intent classification + slot filling (SNIPS).

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “Ex2 (Lee et al., 2021) focus on generating synthetic data for underrepresented or few-shot slices using a model learned by simulating the example generation procedure on data-rich slices of the data.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, methodology): “Possible ways to use few-shot samples for generation include using them as demonstrations [5], for creating augmentations [29] and for tuning the generators.”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “The most similarly-motivated work to ours, Lee et al. (2021), trains a data generator on “data-rich slices” of an existing dataset, and applies it to under-represented slices.”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “, 2019), data augmentation (Lee et al., 2021) style transfer (Reid and Zhong, 2021; Madaan et al.”
A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Lee et al. (2021) use T5 to generate additional examples for data-scarce classes.”
Towards Zero-Label Language Learning (2021, background): “…modiﬁcation has been applied to diversify training samples, more recently generative data augmentation has gained popularity due to the progress made in language modeling (Anaby-Tavor et al., 2019; Papanikolaou and Pier-leoni, 2020; Juuti et al., 2020; Lee et al., 2021; Kumar et al., 2021).”
A Survey on Data Augmentation for Text Classification (2021, unknown): “[116], a first step is to subdivide the data into slices (informed by or based on the labels).”
Controllable Semantic Parsing via Retrieval Augmentation (2021, background): “7 Related works (Jia and Liang, 2016; Kumar et al., 2019; Andreas, 2020; Lee et al., 2021) can be used to amplify the data for the new task.”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “One potential solution is to leverage recent synthetic data generation recipes to train a smaller model (Lee et al., 2021; Schick et al., 2022).”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “…(Rad-ford et al., 2019) and T5 (Raffel et al., 2020), to generate synthetic training data for different tasks, including text classiﬁcation (Anaby-Tavor et al., 2020; Lee et al., 2021; Schick and Schütze, 2021), question answering (Puri et al., 2020), and commonsense reasoning (Yang et al., 2020).”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, background): “Data augmentation methods (Chen et al., 2020; Huang et al., 2022; Lee et al., 2021; Meng et al., 2021b; Miyato et al., 2017; Xie et al., 2020) aim to create similar samples to the existing ones so that the enlarged training set can benefit model generalization.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “LLMs have been explored for data generation in [101, 63, 68, 114], and Self-Instruct [119] proposes a semi-automated process for instruction instance generation.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “LLMs have been explored for data generation in [59, 38, 40, 66], and Self-Instruct [70] proposes a semi-automated process for instruction instance generation.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “One potential solution is to leverage recent synthetic data generation recipes to train a smaller model (Lee et al., 2021; Schick et al., 2022).”