Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation

By Asa Cooper Stickland, Xian Li, Marjan Ghazvininejad. 2020

51 citations recorded by the source corpus.

There has been recent success in pre-training on monolingual data and fine-tuning on Machine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates the benefits and drawbacks of freezing parameters, and adding new ones, when fine-tuning a pre-trained model on MT. We focus on 1) Fine-tuning a model trained only on English monolingual data, BART. 2) Fine-tuning a model trained on monolingual data from 25 languages, mBART. For BART we get the best performance by freezing most of the model parameters, and adding extra positional embeddings. For mBART we match or outperform the performance of naive fine-tuning for most language pairs with the encoder, and most of the decoder, frozen. The encoder-decoder attention parameters are most important to fine-tune. When constraining ourselves to an out-of-domain training set for Vietnamese to English we see the largest improvements over the fine-tuning baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Graham Neubig
2. Luke S. Zettlemoyer
3. Jiatao Gu
4. Mohit Bansal
5. Juan Pino
6. Michael Auli
7. Furu Wei
8. Changhan Wang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jiatao Gu
3. Jitendra Malik
4. Dacheng Tao
5. Trevor Darrell
6. Xiaogang Wang
7. Andrew Zisserman
8. Mohit Bansal

In Natural Language Processing

1. Graham Neubig
2. Luke S. Zettlemoyer
3. Jiatao Gu
4. Mohit Bansal
5. Michael Auli
6. Percy Liang
7. Angela Fan
8. Noah A. Smith

Citations in the corpus

• Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “For other conditional NLG tasks such as machine translation and dialogue generation, fine-tuning is also the prevalent paradigm (Zhang et al., 2020c; Stickland et al., 2020; Zhu et al., 2020; Liu et al., 2020).”
• Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Fine-tuning LayerNorm parameters was also found effective for adapting pretrained BART or mBART for machine translation (Stickland et al., 2020).”
• Multilingual Translation from Denoising Pre-Training (2021, background): “We extend Liu et al. (2020); Cooper Stickland et al. (2021) by investigating finetuning in a multilingual setting.”
• Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “A special case of multi-NMT based TL is ine-tuning large-scale multilingual language models such as mBART using small amounts of parallel data [33]).”
• Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “Examples include MT [24], text summarization [96], text style transfer [86], etc.”