Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection

By Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, Ciprian Chelba. 2018

94 citations recorded by the source corpus.

Measuring domain relevance of data and identifying or selecting well-fit domain data for machine translation (MT) is a well-studied topic, but denoising is not yet. Denoising is concerned with a different type of data quality and tries to reduce the negative impact of data noise on MT training, in particular, neural MT (NMT) training. This paper generalizes methods for measuring and selecting data for domain MT and applies them to denoising NMT training. The proposed approach uses trusted data and a denoising curriculum realized by online data selection. Intrinsic and extrinsic evaluations of the approach show its significant effectiveness for NMT to train on data with severe noise.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Some studies also explore partial parameter sharing strategies tailored for speciﬁc architectures (Sachan and Neubig, 2018; Wang et al., 2018b).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “, 2014), to neural approaches like (contrastive) language-model scoring (Axelrod et al., 2011; Moore and Lewis, 2010; Wang et al., 2018).”
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, positive): “Our choice of a small, trusted dataset for adaptation speciﬁcally to a debiased domain connects to recent work in data selection by Wang et al. (2018), in which ﬁne-tuning on less noisy data improves translation performance.”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, background): “58 Neural machine translation model training may combine data selection and model training, taking advantage of the increasing quality of the model to better detect noisy data or to increasingly focus on cleaner parts of the data (Wang et al., 2018; Kumar et al., 2019).”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “Data weighted training [82] assigns a score for each training example measuring its “noise,” a technique adapted from NMT [138].”
Language-agnostic BERT Sentence Embedding (2020, methodology): “Note that the pairs may still be noisy, which is why we perform additional filtering before training NMT models (Wang et al., 2018) .”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, methodology): “This development of extracting parallel pairs with increasing precision is in fact an instantiation of a denoising curriculum as described by Wang et al. (2018).”
The practical ethics of bias reduction in machine translation: why domain adaptation is better than data debiasing (2021, background): “For instance, adapting to small datasets with desired style, desired vocabulary, or simply reduced data noise has been previously explored (e.g., Farijian et al. 2017, Michel and Neubig 2018; Wang et al. 2018).”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, methodology): “Wang et al. (2018) use a similar approach, where the NMT model is trained on progressively noisereduced data batches.”