AdvAug: Robust Adversarial Augmentation for Neural Machine Translation

By Yong Cheng, Lu Jiang, Wolfgang Macherey, Jacob Eisenstein. 2020

129 citations recorded by the source corpus.

In this paper, we propose a new adversarial augmentation method for Neural Machine Translation (NMT). The main idea is to minimize the vicinal risk over virtual sentences sampled from two vicinity distributions, in which the crucial one is a novel vicinity distribution for adversarial sentences that describes a smooth interpolated embedding space centered around observed training sentence pairs. We then discuss our approach, AdvAug, to train NMT models using the embeddings of virtual sentences in sequence-to-sequence learning. Experiments on Chinese-English, English-French, and English-German translation benchmarks show that AdvAug achieves significant improvements over theTransformer (up to 4.9 BLEU points), and substantially outperforms other data augmentation techniques (e.g.back-translation) without using extra corpora.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Cheng et al. (2020) Propose AdvAug , an adversarial augmentation method for NMT, by sampling adversarial examples from a new vicinity distribution and using their embeddings to augment training.”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “Recent methods have shown successes in overcoming training data bias such as label noise [3, 16, 23, 24, 52, 40, 49, 66] and biased data distributions [56, 10, 35, 38, 9].”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[36] employ curriculum learning to encourage the model to focus on the difficult training examples.”
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, methodology): “With such a characterization, adversarial training (Zhu et al., 2020; Jiang et al., 2020; Liu et al., 2020; Cheng et al., 2020) also performs label-preserving transformation in embedding space, and thus is considered as an alternative to data augmentation methods in this work.”
Token-Aware Virtual Adversarial Training in Natural Language Understanding (2021, methodology): “These methods help improve the robustness of NLP models (Ebrahimi et al. 2017; Alzantot et al. 2018; Jin et al. 2019), or improve the performances of some NLP tasks such as machine translation (Cheng, Jiang, and Macherey 2019; Cheng et al. 2020).”
Sometimes We Want Ungrammatical Translations (2021, background): “Uncovering these sorts of errors has lead the research community to develop new NMT models that are more robust to noisy inputs, using techniques such as targeted data augmentation (Belinkov and Bisk, 2018) and adversarial approaches (Cheng et al., 2020).”
Syntax-Aware Data Augmentation for Neural Machine Translation (2020, methodology): “Furthermore, unlike other works using generated representation [7], [39], embedding [57] or loss [55] to generate new sentences during model training, our approach is an offline method that generates new sentences to enlarge the dataset before model training, which makes our method generally applica”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “, 2017), and have more recently been generalized to the text domain (Miao et al., 2020; Chen et al., 2020c; Cheng et al., 2020b) by performing interpolation between original data and token-level augmented data in the output space (Miao et al.”
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning (2020, background): “…several works have attempted to apply mixup on language representations to improve performance on downstream NLP tasks (Sun et al., 2020; Chen et al., 2020; Cheng et al., 2020), on knowledge distillation (Liang et al., 2020) and on compositional generalization benchmarks (Guo et al., 2020).”