Towards Robust Neural Machine Translation

By Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, Yang Liu. 2018

167 citations recorded by the source corpus.

Small perturbations in the input can severely distort intermediate representations and thus impact translation quality of neural machine translation (NMT) models. In this paper, we propose to improve the robustness of NMT models with adversarial stability training. The basic idea is to make both the encoder and decoder in NMT models robust against input perturbations by enabling them to behave similarly for the original input and its perturbed counterpart. Experimental results on Chinese-English, English-German and English-French translation tasks show that our approaches can not only achieve significant improvements over strong NMT systems but also improve the robustness of NMT models.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Robust Neural Machine Translation with Joint Textual and Phonetic Embedding (2018, methodology): “To improve the robustness of NMT, Cheng et al. (2018) recently proposed to use adversarial stability training to improve the robustness of both encoder and decoder with perturbed inputs.”
Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation (2019, neutral): “Although the M (cid:48) may contain potential parallel sentences t (cid:48) for s , we cannot directly use ( s, t (cid:48) ) as ground-truth sentence pairs to train the translation model V s → t because the NMT system is sensitive to noises (Cho et al., 2014a; Cheng et al., 2018).”
Sentence Boundary Augmentation for Neural Machine Translation Robustness (2020, background): “removal [4] where redundant and unnecessary words are removed before translating the transcript, domain adaptation [5] where NMT models are augmented with in-domain training data, and synthetic noise [6, 7], where random edits are made to training data.”
More Bang for Your Buck: Natural Perturbation for Robust Question Answering (2020, background): “There is a handful of work that studies semi-automatic contextual augmentation (Kobayashi, 2018; Cheng et al., 2018), often with the goal of creating better systems.”
Zero-Shot Cross-Lingual Abstractive Sentence Summarization through Teaching Generation and Attention (2019, unknown): “Compared to Cheng et al. (2018) on Chinese-to-English translation, which targets at robust machine translation and uses the same data to ours, our Transformer signiﬁcantly outperforms their work, indicating that we build a solid system for machine translation.”
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, methodology): “One way of achieving this is to train robust translation models by introducing synthetic ASR errors into the source side of MT corpora (Peitz et al., 2012; Tsvetkov et al., 2014; Ruiz et al., 2015; Sperber et al., 2017b; Cheng et al., 2018, 2019).”
Improving the Robustness of Speech Translation (2018, methodology): “It also demonstrates that the conventional NMT is indeed fragile to permuted inputs, which is consistent with prior work (Belinkov and Bisk 2017; Cheng et al. 2018).”
Hallucinations in Neural Machine Translation (2018, methodology): “Followed by explorations in interpretability, recent work has focused on robust NMT, studying the effects of input noise, aiming to reduce variation from typos [4] and synonym choice [10] in the discrete input set used by NMT systems.”
Robust Neural Machine Translation with Doubly Adversarial Inputs (2019, methodology): “Cheng et al. (2018) employed adversarial stability training to improve the robustness of NMT.”
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation (2020, background): “…of noise can be distinguished: (1) continuous noise which is modeled as a real-valued vector applied to word embeddings (Miyato et al., 2016, 2017; Cheng et al., 2018; Sato et al., 2019), and (2) discrete noise which adds, deletes, and/or replaces characters or words in the observed sentences…”
Sometimes We Want Ungrammatical Translations (2021, background): “…can be surprisingly brittle when presented with out-of-domain data (Luong and Manning, 2015), or when trained with noisy input data containing small orthographic (Sakaguchi et al., 2017; Belinkov and Bisk, 2018; Vaibhav et al., 2019; Niu et al., 2020) or lexical perturbations (Cheng et al., 2018).”
Natural Perturbation for Robust Question Answering (2020, background): “There is a handful of work that studies semi-automatic contextual augmentation (Kobayashi, 2018; Cheng et al., 2018), often with the goal of creating better systems.”
Data Augmentation by Data Noising for Open-vocabulary Slots in Spoken Language Understanding (2019, background): “Cheng et al. (2018) improved the robustness of neural machine translation models against noisy inputs by randomly adding Gaussian noise to the word embedding and maintaining consistent behavior of the encoder to normal and perturbed inputs through adversarial learning.”
Neural Machine Translation: A Review (2019, background): “studied by Neubig & Hu (2018). Approaches that do not rely on resources from a third language include Östling & Tiedemann (2017) who supervised the generation order of an insertion-based low-resource translation model with word alignments.”
Neural Machine Translation with Adequacy-Oriented Learning (2018, unknown): “Adversarial Learning Recently, adversarial learning (Goodfellow et al. 2014) has been successfully applied to neural machine translation (Wu et al. 2017; Yang et al. 2018; Cheng et al. 2018).”
Cross-Lingual Machine Reading Comprehension (2019, methodology): “7 We evaluated GNMT system on NIST MT02/03/04/05/06/08 Chinese-English set and achieved an average BLEU score of 43.24, compared to previous best work (43.20) (Cheng et al., 2018), yielding state-of-the-art performance.”
Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back-Translation (2019, methodology): “Ad-versarial methods are proposed to inject random noise into clean training data in Cheng et al. (2018, 2019).”