Multilingual Denoising Pre-training for Neural Machine Translation

By Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Michael Lewis, Luke S. Zettlemoyer. 2020

2,142 citations recorded by the source corpus.

Abstract This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART—a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective (Lewis et al., 2019). mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, whereas previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine-tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task- specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show that it enables transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.1

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “This shortfall can be seen in benchmarks like DecaNLP (McCann et al., 2018) and GLGE (Liu et al., 2020a) which include NLG tasks but focus only on a single metric and, as a result, may mischaracterize a system’s performance.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “mBART (Liu et al., 2020a) is a multilingual encoder-decoder model that is based on BART (Lewis et al.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “Benchmark Model BLEU mBART Previous Work (Liu et al., 2020) 23.”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “In particular, we use mBART (Liu et al., 2020), which is a multi-lingual language model that has been trained on large, monolingual corpora in 25 languages.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “, 2019) and mBART (Liu et al., 2020) rely on auto-encoding objectives to pre-train the models, which are different from translation.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “For other conditional NLG tasks such as machine translation and dialogue generation, fine-tuning is also the prevalent paradigm (Zhang et al., 2020c; Stickland et al., 2020; Zhu et al., 2020; Liu et al., 2020).”
Generate & Rank: A Multi-task Framework for Math Word Problems (2021, positive): “In this work, we build our model based on BART (Lewis et al., 2020), a widely used pre-trained language model that achieves SOTA performance on various sequence-to-sequence tasks (Ahmad et al., 2021; Liu et al., 2020).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “A major line of work that does not use any labeled parallel data typically works as follow: They ﬁrst train an initial language model using a noisy reconstruction objective, then ﬁnetune it using on-the-ﬂy backtranslation loss [16, 32, 19, 18, 8].”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “The decoder network is initialized with weights from multilingually fine-tuned mBART [29, 30] and uses the same vocabulary with 250K types.”
Unified Pre-training for Program Understanding and Generation (2021, methodology): “This is done to help the model better fit the data (Liu et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “The noise functions of DAE usually contain two operations: replacing a span of tokens with a mask token as well as permuting the order of tokens. mBART (Liu et al., 2020c) extends DAE to support multiple languages by adding special symbols.”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, methodology): “We consider the multilingual model mBART (Liu et al., 2020) as well as all the WMT submissions that reported results on English Ø Kazakh.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “0 speech encoder [20] and an mBART text decoder [21], to S2UT with a wav2vec 2.”
Pre-training via Paraphrasing (2020, methodology): “4 Following Liu et al. [2020], we segment documents into chunks of 512 tokens for training and generation, and then concatenate chunks of the same document.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Another recent approach, mBART (Liu et al., 2020), leverages both monolingual and parallel data and also yields improvements in translation quality for lower-resource languages such as Nepali, Sinhala and Gujarati.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “, 2019) and mBART (Liu et al., 2020) pretrain sequence-to-sequence models to improve machine translation.”
Towards a Unified View of Parameter-Efficient Transfer Learning (2021, methodology): “For MT and text summarization, we use beam search for decoding and set the number of beams to be 6 and 5 following previous work (Li & Liang, 2021; Liu et al., 2020b).”