Good-Enough Compositional Data Augmentation

By Jacob Andreas. 2019

242 citations recorded by the source corpus.

We propose a simple data augmentation protocol aimed at providing a compositional inductive bias in conditional and unconditional sequence models. Under this protocol, synthetic training examples are constructed by taking real training examples and replacing (possibly discontinuous) fragments with other fragments that appear in at least one similar environment. The protocol is model-agnostic and useful for a variety of tasks. Applied to neural sequence-to-sequence models, it reduces error rate by as much as 87% on diagnostic tasks from the SCAN dataset and 16% on a semantic parsing task. Applied to n-gram language models, it reduces perplexity by roughly 1% on small corpora in several languages.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “A common approach is to automatically augment existing datasets (Anaby-Tavor et al., 2020; Andreas, 2020; Yang et al., 2020; Kaushik et al., 2020; Lee et al., 2021, inter alia).”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “There are two representative approaches: building structured neural networks with explicit phrase-based structures or segments [37, 51, 41, 34]; and using data augmentation techniques [3, 17, 2].”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Akyürek et al., 2021; Lake, 2019).”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “…related are the creation of minimally edited datasets, either through manual rewriting (Gard-ner et al., 2020; Kaushik et al., 2020), or creating perturbation templates (Andreas, 2020; Li et al., 2020; Ribeiro et al., 2020; Wu et al., 2019); T ailor reduces the human e ﬀ orts these studies require.”
A Survey of Data Augmentation Approaches for NLP (2021, neutral): “, 2020) × 3 Input × GECA (Andreas, 2020) × × tok Input × SEQMIXUP (Guo et al.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “Data augmentation methods have also been used to mitigate gender (Zhao et al., 2018), racial (Dixon et al., 2018), and other biases (Kaushik et al., 2020) during training, or to introduce useful inductive biases (Andreas, 2020).”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, methodology): “Finally, data recombination has been used to achieve compositional generalization (Jia and Liang, 2016; Andreas, 2020).”
Controllable Semantic Parsing via Retrieval Augmentation (2021, background): “If such examples are available at training time, one could fine-tune the model on them, in which case data augmentation (Jia and Liang, 2016; Kumar et al., 2019; Andreas, 2020; Lee et al., 2021) can be used to amplify the data for the new task.”
Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning (2021, result): “The two-stage curriculum outperformed standard augmentation by a small margin, although both dropped in performance at Nc = 20, consistent with prior findings on the diminished effect of data augmentation for larger datasets (Xie et al., 2020; Andreas, 2020).”
Towards Ecologically Valid Research on Language User Interfaces (2020, unknown): “, by data augmentation on real language (Andreas, 2019)—so as to meet the big data requirements of deep learning methods.”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, methodology): “Additionally, we include the metrics from GECA(4) (Andreas, 2019), a data augmentation method, as well as LANE (Liu et al.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “To increase the compositional generalization abilities of models, recent efforts have also focused on compositional augmentations (Jia and Liang, 2016; Andreas, 2020) where different fragments from different sentences are re-combined to create augmented examples.”
Sequence-level Mixed Sample Data Augmentation (2020, methodology): “Consider the following example from Andreas (2020): (1a) She picks the wug up in Fresno.”
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2020, methodology): “Numerous prior works (Berant & Liang, 2014; Wang et al., 2015a; Jia & Liang, 2016; Iyer et al., 2017; Andreas, 2020) have demonstrated the beneﬁts of augmenting data using context-free grammar.”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, methodology): “It also illustrates modification-based compositionality: once learning how to walk somewhere ‘cautiously’ (Figure 1 left), can models walk elsewhere cautiously or push an object cautiously (Figure 1 right)? On these and other generalizations, we test a baseline multi-modal model representative of co”