Regularization techniques for fine-tuning in neural machine translation

By Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, Rico Sennrich. 2017

112 citations recorded by the source corpus.

We investigate techniques for supervised domain adaptation for neural machine translation where an existing model trained on a large out-of-domain dataset is adapted to a small in-domain dataset. In this scenario, overfitting is a major challenge. We investigate a number of techniques to reduce overfitting and improve transfer learning, including regularization techniques such as dropout and L2-regularization towards an out-of-domain prior. In addition, we introduce tuneout, a novel regularization technique inspired by dropout. We apply these techniques, alone and in combination, to neural machine translation, obtaining improvements on IWSLT datasets for English→German and English→Russian. We also investigate the amounts of in-domain training data needed for domain adaptation in NMT, and find a logarithmic relationship between the amount of training data and gain in BLEU score.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Optimization & Training

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “A number of prior studies observed power-law relations between the generalization error and training data size (Cho et al., 2015; Miceli Barone et al., 2017; Johnson et al., 2018).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “In addition to the growing number of models, ﬁne-tuning requires very careful hyper-parameter tuning (eg. learning rate, regularization knobs etc.) during adaptation, and is prone to rapid over-ﬁtting (Sennrich et al., 2016; Miceli Barone et al., 2017).”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, neutral): “Thompson et al. (2019), Barone et al. (2017), and Khayrallah et al. (2018) propose regularization-based methods that introduce an additional loss to the original ob-jective to help the model trade off between the general-domain and in-domain.”
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models (2019, background): “Barone et al. (2017) introduced tuneout, which is a special case of mixout.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, methodology): “Our work applies a regularization term to continued training, similar to Miceli Barone et al. (2017) and Khayrallah et al. (2018), but for the purpose of retaining generaldomain performance as opposed to improving indomain performance.”
Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation (2018, methodology): “Miceli Barone et al. (2017) share our goal of improving in-domain results and compare three methods of regularization to improve leaning during continued training: 1) Bayesian dropout 2) L2 regularization, and 3) tuneout , which is similar to Bayesian dropout but instead of setting weights to zero,…”
Explicit Inductive Bias for Transfer Learning with Convolutional Networks (2018, background): “2014) and neural machine translation (Barone et al. 2017), where it proved to be helpful.”
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, positive): “Regularized training is a well-established approach for minimizing catastrophic forgetting during domain adaptation of machine translation (Barone et al., 2017).”
Extreme Adaptation for Personalized Neural Machine Translation (2018, methodology): “Overfitting Training each speaker model with very little data is a challenge, necessitating careful and heavy regularization (Miceli Barone et al., 2017) and an early stopping procedure.”
Domain Adaptive Inference for Neural Machine Translation (2019, methodology): “Barone et al. (2017) fine-tune with L2 regularization to reduce forgetting.”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, methodology): “This ﬁnding may explain the recent success of techniques which regularize a continued training model using the initial, out-of-domain model (Miceli Barone et al., 2017; Dakwale and Monz, 2017; Khayrallah et al., 2018).”
Softmax Tempering for Training Neural Machine Translation Models (2020, methodology): “Thus, it is related to techniques, such as L N regularization (Ng, 2004), dropout (Srivastava et al., 2014), and tuneout (Miceli Barone et al., 2017).”
Predicting accuracy on large datasets from smaller pilot data (2018, unknown): “Sun et al. (2017), Barone et al. (2017) and the concurrent unpublished work by Hestness et al. (2017) point out that these power-law models describe modern ML and NLP systems quite well, including complex deep-learning systems, so we expect our results to generalise to these systems.”
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, methodology): “Regularization-based methods (Dakwale and Monz, 2017; Thompson et al., 2019; Barone et al., 2017; Khayrallah et al., 2018) instead introduce an additional loss to the original objective so that the translation model can trade off between general-domain and in-domain.”
Improving Robustness of Machine Translation with Synthetic Noise (2019, background): “…the target domain (Li et al., 2010; Axelrod et al., 2011), Luong and Manning (2015) suggest the use of fine-tuning on varying amounts of target domain data, and Barone et al. (2017) note a logarithmic relationship between the amount of data used in fine-tuning and the relative success of MT models.”
Online Distilling from Checkpoints for Neural Machine Translation (2019, methodology): “For further performance improvement, deeper and more expressive structures (Johnson et al., 2017; Barone et al., 2017b; Gehring et al., 2017; Vaswani et al., 2017) have been exploited.”