Cross-lingual Language Model Pretraining

By Guillaume Lample, Alexis Conneau. 2019

3,011 citations recorded by the source corpus.

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT’16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT’16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “The state of the art on the low-resource English to Romanian benchmark also uses additional forms of cross-lingual unsupervised training [Lample and Conneau, 2019].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “, 2017), and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019).”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “ised benchmarks §3.1, and directly use the pre-training data (CC25) for backtranslation to avoid introducing new information. Learning Following the same procedure described in Lample et al. (2018c); Lample and Conneau (2019), we ﬁrst initialize the translation model with the pre-trained weights, an”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, background): “The largest improvements have come from pre-training on both source and target languages (Song et al., 2019; Lample & Conneau, 2019), but this requires pre-training on all languages of interest.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “We pre-train the model with BERT+LM and DAE , and ﬁne-tune on the unsupervised translation pairs with same ﬁne-tuning strategy of XLM ( i.e. , DAE loss + back-translation).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…(Bengio et al., 2003) have played an important role in natural language processing through transferrable word vectors (Mikolov et al., 2013), contextualized word vectors (Peters et al., 2018; Devlin et al., 2018; Lample & Conneau, 2019), and models (Howard & Ruder, 2018; Radford et al., 2018).”
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, methodology): “It is straightforward to apply the Transformer to this task (Radford et al., 2018; Lample and Conneau, 2019).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “XLM (Lample and Conneau, 2019) explores cross-lingual pre-training for multilingual tasks such as translation and crosslingual classification.”
ERNIE: Enhanced Representation through Knowledge Integration (2019, methodology): “ERNIE use multi-layer Transformer (Vaswani et al., 2017) as basic encoder like previous pre-traning model such as GPT, BERT and XLM.”
Learning Compact Metrics for MT (2021, neutral): “In the case of cross-lingual transfer, this phenomenon is known as the curse of multilinguality: to allow for positive transfer, the model must be scaled up with the number of languages (Conneau and Lample, 2019).”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “mBERT (Devlin, 2018) Encoder-only 110M 104 Wikipedia XLM (Lample and Conneau, 2019) Encoder-only 570M 100 Wikipedia XLM-R (Conneau et al.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “Multilingual Pre-training via RAS Recent work proves that cross-lingual language model pretraining could be a more effective way to representation learning (Conneau and Lample, 2019; Huang et al., 2019).”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “Following the recent trend of using multi-lingual pre-trained models for NLP tasks, such as Multilin-gual BERT (M-BERT) (Pires et al., 2019) 1 or XLM (Lample and Conneau, 2019), Chi et al. (2019) proposed to ﬁne-tune the models for summarization on English training data.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Removing the <2xx> token from the source sentence during mNMT training improves cross-lingual effectiveness on both POS tagging and XNLI task.”
Tagged Back-Translation (2019, methodology): “For example, Lample and Conneau (2019) report 33.3 B LEU on EnRo using unsupervised NMT, which at ﬁrst seems comparable to our reported 33.4 SacreB LEU from iterative TaggedBT.”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “Moreover, leveraging the pseudo corpus generated by unsupervised-SMT system can bring furthermore enhancement though the unsupervised SMT is inferior to NMT system.”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “This improves the quality of the resulting dataset and allows for the training of multilingual text level representations like XLM (Lample and Conneau, 2019). en ru de fr es ja zh it pt nl pl cs id fa ar ko el hu ro da fi uk no bg he 10 6 10 7 10 8 10 9 10 10 Figure 6: Number of documents per…”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “, 2017) and explosion of following open models (Devlin et al., 2019; Lample and Conneau, 2019; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2018, 2019), machine translation (Lample and Conneau, 2019), and zero-short language generation (Radford et al.”