G-DAug: Generative Data Augmentation for Commonsense Reasoning

By Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey. 2020

92 citations recorded by the source corpus.

Recent advances in commonsense reasoning depend on large-scale human-annotated training sets to achieve peak performance. However, manual curation of training sets is expensive and has been shown to introduce annotation artifacts that neural models can readily exploit and overfit to. We propose a novel generative data augmentation technique, G-DAUGˆC, that aims to achieve more accurate and robust learning in a low-resource setting. Our approach generates synthetic examples using pretrained language models and selects the most informative and diverse set of examples for data augmentation. On experiments with multiple commonsense reasoning benchmarks, G-DAUGˆC consistently outperforms existing data augmentation methods based on back-translation, establishing a new state-of-the-art on WinoGrande, CODAH, and CommonsenseQA, as well as enhances out-of-distribution generalization, proving to be robust against adversaries or perturbations. Our analysis demonstrates that G-DAUGˆC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Recently, generative language models have been used to synthesize examples for various NLP tasks (Kumar et al., 2020; Anaby-Tavor et al., 2020; Puri et al., 2020; Yang et al., 2020).”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “Yang et al. (2020) generate randomly initialized samples from finetuned GPT2 as augmentation data for target datasets.”
Data Boost: Text Data Augmentation through Reinforcement Learning Guided Conditional Generation (2020, methodology): “Existing methods tend to produce augmentation with low readability or unsatisfying semantic consistency (Yang et al., 2020).”
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020, methodology): “Other methods focus on ﬁne-tuning contextual language models (Kobayashi, 2018; Wu et al., 2019b; Kumar et al., 2020) or large generative models (Anaby-Tavor et al., 2020; Yang et al., 2020; Kumar et al., 2020) to generate synthetic examples.”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “To evaluate this, we use the TextFooler system (Jin et al., 2020; Yang et al., 2020; Wei and Zou, 2019) to generate adversarial text by computing word importance ranking and replacing the most influential words with their synonym in the vector space.”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, methodology): “This does not mean that the tasks would be unsolvable in general: Several studies [Huang et al., 2021, Ouyang et al., 2021, Wang et al., 2021b, Yang et al., 2020, Lourie et al., 2021, Xu et al., 2021] show that the addition of symbolic knowledge (such as date normalization, quasi-logical reasoning,…”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, background): “…the PLMs can be either fine-tuned on the labeled samples to learn label-conditioned generation probability (Kumar et al., 2020; Lee et al., 2021; Yang et al., 2020) or take the labeled data as demonstrations (Wang et al., 2021; Yoo et al., 2021) to generate similar samples pertaining to the…”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “tation (Zhang et al., 2020a; Yang et al., 2020; Peng et al., 2020; Kumar et al., 2020; Anaby-Tavor et al., 2020), requiring no significant change to the model architecture.”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, unknown): “Building on earlier work (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Yang et al., 2020; Mohapatra et al., 2020; Kumar et al., 2020; Schick and Sch¨utze, 2021), we investigate whether synthetic examples can replace unlabeled data.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “Similarly, new answers are generated on the basis of given questions in question answering and are ﬁltered by customized metrics like question answering probability (Zhang and Bansal, 2019) and n-gram diversity (Yang et al., 2020).”