Domain Adaptive Inference for Neural Machine Translation
By Danielle Saunders, Felix Stahlberg, Adrià de Gispert, Bill Byrne. 2019
32 citations recorded by the source corpus.
We investigate adaptive ensemble weighting for Neural Machine Translation, addressing the case of improving performance on a new and potentially unknown domain without sacrificing performance on the original domain. We adapt sequentially across two Spanish-English and three English-German tasks, comparing unregularized fine-tuning, L2 and Elastic Weight Consolidation. We then report a novel scheme for adaptive NMT ensemble decoding by extending Bayesian Interpolation with source information, and report strong improvements across test domains without access to the domain label.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- James Thorne
- Andreas Vlachos
- Danielle Saunders
- Minlie Huang
- Xipeng Qiu
- Bill Byrne
- Fei Mi
- Mingxuan Wang
In Natural Language Processing
- James Thorne
- Andreas Vlachos
- Danielle Saunders
- Bill Byrne
- Minlie Huang
- Lei Li
- Chengqi Zhao
- Jianze Liang
Citations in the corpus
- Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “(2019) and Saunders et al. (2019) minimize catastrophic forgetting when adapting the domain of models through fine-tuning.”
- Continual Learning for Neural Machine Translation (2021, background): “There is an improvement of 0.3-0.8 BLEU scores over the fine-tuned model, 0-0.3 BLEU scores over the knowledge distillation-based model, and 0.2-0.5 BLEU scores over the EWC regularization-based model.”
- Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, methodology): “3462 applied to the domain adaptation task for Neural Machine Translation (Thompson et al., 2019; Saunders et al., 2019).”
- Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, positive): “One effective form is Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) which in NMT has been shown to maintain or even improve original domain performance (Thompson et al., 2019; Saunders et al., 2019).”
- Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, background): “Fine-tuning is very common in domain adaptation, which first trains a base model on the general domain data and then fine-tunes it on each target domain (Luong and Manning 2015; Chen et al. 2017; Gu, Feng, and Liu 2019; Saunders et al. 2019).”
- Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, unknown): “Saunders et al. (2019) investigates adaptive ensemble weighting for inference.”