Robust Neural Machine Translation with Doubly Adversarial Inputs

By Yong Cheng, Lu Jiang, Wolfgang Macherey. 2019

269 citations recorded by the source corpus.

Neural machine translation (NMT) often suffers from the vulnerability to noisy perturbations in the input. We propose an approach to improving the robustness of NMT models, which consists of two parts: (1) attack the translation model with adversarial source examples; (2) defend the translation model with adversarial target inputs to improve its robustness against the adversarial source inputs. For the generation of adversarial inputs, we propose a gradient-based method to craft adversarial examples informed by the translation loss over the clean inputs. Experimental results on Chinese-English and English-German translation tasks demonstrate that our approach achieves significant improvements (2.8 and 1.6 BLEU points) over Transformer on standard clean benchmarks as well as exhibiting higher robustness on noisy data.

Subfields: Artificial Intelligence, Materials Chemistry, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Training for Large Neural Language Models (2020, background): “In NLP, applying adversarial training is not straight-forward, since the input are discrete elements (to-ken or subword sequences), but there have been some recent successes (Zhu et al., 2019; Jiang et al., 2019; Cheng et al., 2019; Wang et al., 2019; Mi-naee et al., 2020b).”
FreeLB: Enhanced Adversarial Training for Natural Language Understanding (2019, background): “Surprisingly, people observe the opposite result for language models (Miyato et al., 2017; Cheng et al., 2019), showing that adversarial training can improve both generalization and robustness.”
Contextualized Perturbation for Textual Adversarial Attack (2020, methodology): “Besides exposing system vulnerabilities and helping improve their robustness and security (Zhao et al., 2018; Wallace et al., 2019; Cheng et al., 2019; Jia et al., 2019, inter alia ), adversarial examples are also used to analyze and interpret the models’ decisions (Jia and Liang, 2017; Ribeiro et…”
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation (2020, methodology): “We also directly incorporate two adversarial examples to NMT models without interpolating their embed-dings, but we do not observe any further gain over Cheng et al. (2019).”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, result): “This result also resonates with the observations of [365].”
Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back-Translation (2019, methodology): “Ad-versarial methods are proposed to inject random noise into clean training data in Cheng et al. (2018, 2019).”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, background): “, 2014b) to increase neural models’ robustness, and can also be utilized as data augmentation to increase the models’ generalization ability (Miyato et al., 2016; Cheng et al., 2019).”
Syntax-Aware Data Augmentation for Neural Machine Translation (2020, methodology): “Furthermore, unlike other works using generated representation [7], [39], embedding [57] or loss [55] to generate new sentences during model training, our approach is an offline method that generates new sentences to enlarge the dataset before model training, which makes our method generally applica”
Token-Aware Virtual Adversarial Training in Natural Language Understanding (2021, methodology): “Gradient-based methods such as Ebrahimi et al. (2017); Papernot et al. (2016); Cheng, Jiang, and Macherey (2019) used in generating adversarial samples in the texts domains usually ﬁnd the substitutes that are similar to the gradient-based perturbations.”
The Curious Case of Hallucinations in Neural Machine Translation (2021, unknown): “Subsequently, approaches to making NMT models more robust to small perturbations in the input have been actively explored (Cheng et al., 2019), however, no coherent theory to explain the phenomena of hallucinations has been empirically validated in the existing literature.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “Given the original samples, Cheng et al. [25] ﬁrstly construct their adversarial samples following [74], and then apply two Mixup strategies named P adv and P aut : The former interpolates between adversarial samples, and the latter interpolates between the two corresponding original samples.”
FreeLB: Enhanced Adversarial Training for Language Understanding (2019, background): “Cheng et al. (2019) also ﬁnd the craft adversarial examples can help neural machine translation signiﬁcantly.”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…et al., 2019; Jiang et al., 2015; Liang et al., 2016), cross-validation (Northcutt et al., 2019), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017; Li et al., 2020; Zhang et al., 2020), data augmentation (Zhang et al., 2018; Cheng et al., 2019; 2020; Liang et al., 2020), among others.”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, background): “Cheng et al. (2019a) study adversarial learning in negotiation dialogues (Lewis et al., 2017), where agents divide several items with different values based on conversation, so that each item is assigned to one agent.”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “[3] PE embedding binary MalConv [110] L∞ MAD [23] word N Transformer [137] Language model MT”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, background): “[180] proposes an interesting adversarial framework for robustifying NMT against noisy inputs and reports performance gains over the Transformer model.”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…2017), example weighting (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017), data augmentation (Zhang et al., 2018; Cheng et al., 2019 et al., 2015; Li et al., 2017a; Krause et al., 2016).”