On the use of BERT for Neural Machine Translation

By Stéphane Clinchant, K. Jung, Vassilina Nikoulina. 2019

99 citations recorded by the source corpus.

Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness.

Subfields: Artificial Intelligence, Large Language Models, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jingjing Liu
3. Mohit Bansal
4. Jianfeng Gao
5. Linjie Li
6. Yu Cheng
7. Xipeng Qiu
8. Yen-Chun Chen

In Large Language Models

1. Zhe Gan
2. Linjie Li
3. Yen-Chun Chen
4. Jingjing Liu
5. Yu Cheng
6. Julian Salazar
7. Andrew McCallum
8. Ning Dai

In Machine Translation

1. Won Ik Cho
2. Nam Soo Kim
3. Mohit Iyyer
4. Jingjing Liu
5. Jiwon Kim
6. Linjie Li

Citations in the corpus

• Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Satellited around BERT, there are also studies on model compression (Sun et al., 2019c; Jiao et al., 2019; Shen et al., 2019) and extension from understanding to generation (Chen et al., 2019a; Clinchant et al., 2019; Wang and Cho, 2019).”
• Masked Language Model Scoring (2019, methodology): “This handicapped their use in generative tasks, where they at best served to bootstrap encoder-decoder models (Clinchant et al., 2019; Zhu et al., 2020) or unidirectional LMs (Wang et al.”
• Pre-trained models for natural language processing: A survey (2020, methodology): “hmark.com/ 16 QIU XP, et al. Pre-trained Models for Natural Language Processing: A Survey March (2020) pre-trained language model for source word embedding initialization. Similarly, Clinchant et al. [23] devised a series of dierent experiments for examining the best strategy to utilize BERT on the ”
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “Similar to deep fusion, Domhan and Hieber (2017) feed the unnormalized output of LM to the decoder of NMT. Domhan and Hieber (2017) jointly train the LM and NMT using monolingual target-side data and parallel data, respectively.”
• Towards Cross-Lingual Generalization of Translation Gender Bias (2021, background): “On top of the linguistic factors, we took the amount of available translation resources into consideration, since it is assumed to directly or indirectly affect the overall performance of (public) MT systems, either with training or pre-training [7, 12].”