Dynamic Data Selection for Neural Machine Translation

By Marlies van der Wees, Arianna Bisazza, Christof Monz. 2017

156 citations recorded by the source corpus.

Intelligent selection of training data has proven a successful technique to simultaneously increase training efficiency and translation performance for phrase-based machine translation (PBMT). With the recent increase in popularity of neural machine translation (NMT), we explore in this paper to what extent and how NMT can also benefit from data selection. While state-of-the-art data selection (Axelrod et al., 2011) consistently performs well for PBMT, we show that gains are substantially lower for NMT. Next, we introduce ‘dynamic data selection’ for NMT, a method in which we vary the selected subset of training data between different training epochs. Our experiments show that the best results are achieved when applying a technique we call ‘gradual fine-tuning’, with improvements up to +2.6 BLEU over the original data selection approach and up to +3.1 BLEU over a general baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Research on exploiting task-relatedness (Lee et al., 2016; Neubig and Hu, 2018), curriculum learning on noisy data (van der Wees et al., 2017; Wang et al., 2018a), and automated curriculum learning from model state (Bengio et al., 2009; Graves et al., 2017) have demonstrated success in multitask…”
Tagged Back-Translation (2019, methodology): “Similarly, tagging could be investigated as an alternative to data selection, as in van der Wees et al. (2017); Axelrod et al. (2011), or curriculum learning approaches like ﬁne-tuning on in-domain data (Thompson et al., 2018; Hassan Sajjad and Nadir Durrani and Fahim Dalvi and Yonatan Belinkov and…”
Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, methodology): “It was successfully used in many phrase-based MT systems, but it was reported to be less successful for NMT (van der Wees et al., 2017).”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “Unfortunately it has not proved as useful for NMT; while it reduces the training data it does not lead to improvements in system quality [37].”
Learning to select data for transfer learning with Bayesian Optimization (2017, background): “A recent study investigates data selection for neural machine translation (van der Wees et al., 2017).”
Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering (2018, neutral): “…on data selection which aims at sub-sampling parallel data relevant for a task-speciﬁc machine translation system (Axelrod et al., 2011). van der Wees et al. (2017) ﬁnd that the existing data selection methods developed for statistical machine translation are less effective for neural machine…”
On the Impact of Various Types of Noise on Neural Machine Translation (2018, methodology): “…literature on data selection which aims at sub-sampling parallel data relevant for a task-speciﬁc machine translation system (Axelrod et al., 2011). van der Wees et al. (2017) ﬁnd that the existing data selection methods developed for statistical machine translation are less effective for neural…”
An Empirical Exploration of Curriculum Learning for Neural Machine Translation (2018, background): “We consider a wide range of schedules, based not only on the easy-to-difﬁcult ordering, but also on strategies developed independently from curriculum learning, such as dynamic sampling and boosting (Zhang et al., 2017; van der Wees et al., 2017; Wang et al., 2018).”
Learning Hidden Unit Contribution for Adapting Neural Machine Translation Models (2018, methodology): “Previous methods can of course still be applied as long as they are model-independent, e.g. data selection methods as shown in (van der Wees et al., 2017) or self-training approaches like (Bertoldi and Federico, 2009), of which back-translation (Sennrich et al., 2015) can be considered a special…”
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, methodology): “It has been used in machine translation (van der Wees et al., 2017; Wang et al., 2018), parsing (Plank and van Noord, 2011; Ruder and Plank, 2017) and sentiment analysis (Ruder et al., 2017).”
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation (2018, background): “Earlier work has explored to what extent data selection of parallel corpora can beneﬁt translation quality (Axelrod et al., 2011; van der Wees et al., 2017), but such selection techniques have not been investigated in the context of back-translation.”
Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection (2018, methodology): “The noise annealing is inspired by (van der Wees et al., 2017), but we anneal data quality at per step to make the approach more friendly to NMT online optimization, instead of per data epoch.”
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (2019, methodology): “These approaches have been applied in machine translation (van der Wees et al., 2017), language models (Moore and Lewis, 2010; Duh et al., 2013), dependency parsing (Søgaard, 2011), etc.”
Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, background): “new relevant training data in several (related or not) languages might become available continuously. Taking advantage of relevant data for adaptation is crucial to the performance of the ﬁnal models [4, 5]. Recently, building a large scale MNMT model was shown to be beneﬁcial for LRL [6],evenoutper”
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (2019, background): “Among the very extensive work on handling heterogeneous data in NMT, the closest to ours are techniques that re-weight (Chen et al., 2017) or re-order examples to deal with domain mismatch (van der Wees et al., 2017; Sajjad et al., 2017) or noise (Wang et al., 2018).”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, methodology): “For model-centric domain adaptation methods, our work is related to van der Wees et al. (2017).”
A Survey of Domain Adaptation for Neural Machine Translation (2018, background): “…and Zong, 2016b; Cheng et al., 2016; Currey et al., 2017; Domhan and Hieber, 2017), synthetic corpora (Sennrich et al., 2016b; Zhang and Zong, 2016b; Park et al., 2017), or parallel copora (Chu et al., 2017; Sajjad et al., 2017; Britz et al., 2017; Wang et al., 2017a; van der Wees et al., 2017).”
Choosing Transfer Languages for Cross-Lingual Learning (2019, methodology): “Tsvetkov et al. (2016a) and Ruder and Plank (2017) use Bayesian optimization for data selection. van der Wees et al. (2017) study the effect of data selection of neural machine translation, as well as propose a dynamic method to select relevant training data that improves translation performance.”