Towards Making the Most of BERT in Neural Machine Translation

By Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang, Lei Li. 2019

167 citations recorded by the source corpus.

GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTnmt) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTnmt} consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTnmt gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “Despite its wide adoption for other NLP tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019b; Lewis et al., 2019; Raffel et al., 2019), self-supervised pre-training is not yet common practice in machine translation (MT).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “First, pre-trained language models such as BERT are not easy to directly fine-tune unless using some sophisticated techniques (Yang et al., 2020).”
Mitigating Political Bias in Language Models Through Reinforced Calibration (2021, background): “Large-scale language models (LMs) can generate human-like text and have shown promise in many Natural Language Generation (NLG) applications such as dialogue generation (Zhang et al. 2020; Peng et al. 2020) and machine translation (Yang et al. 2020; Zhu et al. 2020).”
MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2021, background): “Previous studies can be roughly divided into three categories: (1) finetuning pre-trained language models (Yang et al., 2020; Weng et al., 2020).”
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021, unknown): “The ﬁrst category usually uses pre-trained models as feature extractors of a source language, or initializes the encoder and decoder with pre-trained models separately (Lample and Conneau, 2019; Ren et al., 2019; Yang et al., 2020a; Zhu et al., 2020).”
On Learning Universal Representations Across Languages (2020, methodology): “Our approach also outperforms the BERT-fused model (Yang et al., 2020), a method treats BERT as an extra context and fuses the representations extracted from BERT with each encoder and decoder layer.”
Distilling the Knowledge of BERT for Text Generation (2019, methodology): “Concurrent with our work, Ghazvininejad et al. (2019) proposed a similar conditional MLM for constant-time translation, and Yang et al. (2019) studied how to ﬁne-tune BERT for NMT.”
Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation (2021, methodology): “With autoregressive decoding where BERT is utilized only on the source-side, we compare our framework with BERT-Fused [14], BERT-Distilled [13] and CT-NMT [12] with their reported scores.”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, methodology): “[182] achieves that aim primarily by using a dynamically gated model and asymptotic distillation.”
Incorporating BERT into Neural Machine Translation (2020, unknown): “…have been proposed: In XLM (Lample & Conneau, 2019), the model is pre-trained based on multiple languages and NSP task is removed; in RoBERTa (Liu et al., 2019), more unlabeled data is leveraged without NSP task neither; in XLNet (Yang et al., 2019b), a permutation based modeling is introduced.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Moreover, considering the excellent encoding capability of BERT, CTNMT [256] used asymptotic distillation and dynamic switching gate to integrate BERT embeddings.”
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, methodology): “…al., 2019; Lewis et al., 2019) Specifically for MT, Ramachandran et al. (2017) proposed pre-training the encoder-decoder modules as two separate language models, and Yang et al. (2019a); Zhu et al. (2020) explored approaches incorporating BERT model weights into the usual seq-to-seq architecture.”