Do Not Have Enough Data? Deep Learning to the Rescue!

By Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, Naama Zwerdling. 2019

410 citations recorded by the source corpus.

Based on recent advances in natural language modeling and those in text generation capabilities, we propose a novel data augmentation method for text classification tasks. We use a powerful pre-trained neural network model to artificially synthesize new labeled data for supervised learning. We mainly focus on cases with scarce labeled data. Our method, referred to as language-model-based data augmentation (LAMBADA), involves fine-tuning a state-of-the-art language generator to a specific task through an initial training phase on the existing (usually small) labeled data. Using the fine-tuned model and given a class label, new sentences for the class are generated. Our process then filters these new sentences by using a classifier trained on the original data. In a series of experiments, we show that LAMBADA improves classifiers' performance on a variety of datasets. Moreover, LAMBADA significantly improves upon the state-of-the-art techniques for data augmentation, specifically those applicable to text classification tasks with little data.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “A common approach is to automatically augment existing datasets (Anaby-Tavor et al., 2020; Andreas, 2020; Yang et al., 2020; Kaushik et al., 2020; Lee et al., 2021, inter alia).”
Generating Datasets with Pretrained Language Models (2021, neutral): “Some approaches augment existing datasets with automatically generated examples (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Yang et al., 2020; Mohapatra et al., 2020; Kumar et al., 2021), but in contrast to our work, all of these approaches require that there already exists a…”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “Some works explore automatic data synthesis/expansion by ﬁnetuning LMs on existing labeled data (Anaby-Tavor et al., 2020; Papaniko-laou and Pierleoni, 2020; Kumar et al., 2020; Yang et al., 2020), but are limited by data quality.”
A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Anaby-Tavor et al. (2020) learn a label-conditioned generator by ﬁnetuning GPT-2 (Radford et al., 2019) on the training data, using this to generate candidate examples per class.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Recently, generative language models have been used to synthesize examples for various NLP tasks (Kumar et al., 2020; Anaby-Tavor et al., 2020; Puri et al., 2020; Yang et al., 2020).”
Neural Data Augmentation via Example Extrapolation (2021, methodology): “…literature, our approach is most related to recent work on data augmentation for NLP using pre-trained language models (LMs): Kumar et al. (2019); Anaby-Tavor et al. (2020) perform data augmentation for text classification by fine-tuning an LM to synthesize new inputs x for a given label y —…”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “Early efforts achieve this goal with fine-tuned generative models (Anaby-Tavor et al., 2020; Puri et al., 2020; Kumar et al., 2020; Lee et al., 2021).”
A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, background): “Fine-tuning pre-trained models on imbalanced datasets can lead to improved classification performance since these models can learn to assign more significance to the underrepresented classes [349].”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “tation (Zhang et al., 2020a; Yang et al., 2020; Peng et al., 2020; Kumar et al., 2020; Anaby-Tavor et al., 2020), requiring no significant change to the model architecture.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[28] propose a data augmentation method named LAMBDA.”
Leveraging QA Datasets to Improve Generative Data Augmentation (2022, methodology): “We fix the few-shot supervision size to 8 samples per label and vary the number of generated samples per label and plot the performance of C ON DA-TweetQA, C ON DA-SocialIQA, and baselines such as LAMBADA and EDA on AGNews and IMDb datasets, shown in Figure 3.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, methodology): “Instead, a language model can also be used for augmenting text classification datasets (Kumar et al., 2020; Anaby-Tavor et al., 2020).”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “…as GPT-2 (Rad-ford et al., 2019) and T5 (Raffel et al., 2020), to generate synthetic training data for different tasks, including text classiﬁcation (Anaby-Tavor et al., 2020; Lee et al., 2021; Schick and Schütze, 2021), question answering (Puri et al., 2020), and commonsense reasoning (Yang et…”