Continual Learning for Neural Machine Translation

By Yue Cao, Haochen Wei, Boxing Chen, Xiaojun Wan. 2021

34 citations recorded by the source corpus.

Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus. However, this bears the risk of catastrophic forgetting that the performance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT models. We consider a scenario where the training is comprised of multiple stages and propose a dynamic knowledge distillation technique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and propose a bias-correction module to eliminate the bias. We conduct experiments on three representative settings of NMT application. Experimental results show that the proposed method achieves superior performance compared to baseline models in all settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The applications of continual learning in NLP are also highly diverse, such as dialogue system [130], [131], [260], [260], [282], [298], [437], text classification [87], [172], [325], [460], sentence generation [246], [298], [499], relation learning [343], [357], [455], [465], neural machine transla”
Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, background): “…same model as both a solver and generator and instead of replaying actual samples, replay latent representations. et al. (2021); Qian et al. (2021); Cao et al. (2021) use different variants of knowledge distillation and representative sample selection meth-ods to alleviate catastrophic forgetting…”
Fine-tuned Language Models are Continual Learners (2022, methodology): “Cao et al. (2021) propose a new CL framework for NMT models, while Ke et al. (2021) proposes a novel capsule network based model called B-CL (Bert based CL) for sentiment classification tasks.”