Sequence-level Mixed Sample Data Augmentation

By Demi Guo, Yoon Kim, Alexander M. Rush. 2020

107 citations recorded by the source corpus.

Despite their empirical success, neural networks still have difficulty capturing compositional aspects of natural language. This work proposes a simple data augmentation approach to encourage compositional behavior in neural models for sequence-to-sequence problems. Our approach, SeqMix, creates new synthetic examples by softly combining input/output sequences from the training set. We connect this approach to existing techniques such as SwitchOut and word dropout, and show that these techniques are all approximating variants of a single objective. SeqMix consistently yields approximately 1.0 BLEU improvement on five different translation datasets over strong Transformer baselines. On tasks that require strong compositional generalization such as SCAN and semantic parsing, SeqMix also offers further improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Grammar-Based Grounded Lexicon Learning (2022, methodology): “…(while keeping the same label); (5) similarly, SwitchOut [45] randomly replaces an input token with a random token from the vocabulary; (6) SeqMix [17] uses soft augmentation techniques following [49], which composes an “weighted average” of different input sequences; (7) recomb-2 [2] learns…”
A Survey of Data Augmentation Approaches for NLP (2021, methodology): “, 2020) × 3 Input × GECA (Andreas, 2020) × × tok Input × SEQMIXUP (Guo et al., 2020) × × tok Input × SWITCHOUT (Wang et al.”
Syntax-Aware Data Augmentation for Neural Machine Translation (2020, methodology): “[42] proposed SeqMix to create new synthetic examples by softly combining sequences from the training set.”
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning (2020, background): “…several works have attempted to apply mixup on language representations to improve performance on downstream NLP tasks (Sun et al., 2020; Chen et al., 2020; Cheng et al., 2020), on knowledge distillation (Liang et al., 2020) and on compositional generalization benchmarks (Guo et al., 2020).”
Substructure Substitution: Structured Data Augmentation for NLP (2021, methodology): “…method (Zhang et al., 2018), which augments the original dataset by linearly interpolating the vector representations of text and labels, to text classiﬁcation (Guo et al., 2019; Sun et al., 2020), named entity recognition (Chen et al., 2020) and compositional generalization (Guo et al., 2020).”
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021, background): “On the other hand, inspired by the mix-up technique (Zhang et al., 2017) proposed for the vision domain, there have also been works to mix existing text samples to produce realistic augmentation texts based on statistical methods (Guo et al., 2020; Sun et al., 2020; Chen et al., 2020).”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “…has since been a large body of work on compositional sequence-to-sequence learning through various approaches including modiﬁcations to existing architectures [70, 94, 44, 17, 26], grammars and neuro-symbolic models [86, 98, 85, 19, 76], meta-learning [67, 25], and data augmentation [6, 49, 50, 4].”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “Different strategies to select samples to mix have also been explored (Chen et al., 2020a; Guo et al., 2020; Zhang et al., 2020) like k-nearestneighbours (Chen et al.”