Emerging Cross-lingual Structure in Pretrained Language Models

By Shijie Wu, Alexis Conneau, Haoran Li, Luke S. Zettlemoyer, Veselin Stoyanov. 2019

329 citations recorded by the source corpus.

We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of several factors that influence why these models are so effective for cross-lingual transfer. We show, contrary to what was previously hypothesized, that transfer is possible even when there is no shared vocabulary across the monolingual corpora and also when the text comes from very different domains. The only requirement is that there are some shared parameters in the top layers of the multi-lingual encoder. To better understand this result, we also show that representations from monolingual BERT models in different languages can be aligned post-hoc quite effectively, strongly suggesting that, much like for non-contextual word embeddings, there are universal latent symmetries in the learned embedding spaces. For multilingual masked language modeling, these symmetries are automatically discovered and aligned during the joint training process.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Emergent Cross-Lingual Alignment On understanding cross-lingual alignment from pretrained models, [38] [24] present empirical evidence that there exists cross-lingual alignment structure in the encoder which is trained with multiple languages on a shared masked language modeling task.”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Interestingly, for speech, no lexical overlap across languages is leveraged during training, but multilingual representations still emerge from parameter sharing of the Transformer network (Wu et al., 2019).”
Unsupervised Cross-lingual Representation Learning at Scale (2019, methodology): “Results of mBERT are reported from Wu and Dredze (2019). Note that we do not use a linear-chain CRF on top of XLM-R and mBERT representations, which gives an advantage to Akbik et al.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, background): “…of multilingual data—have been observed to perform surprisingly well when transferring to other languages (Pires et al., 2019; Wu and Dredze, 2019; Wu et al., 2020) and the current state-of-the-art model, XLM-R has been shown to be competitive with the performance of monolingual models on the…”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “We choose this baseline as its behavior has been thoroughly studied (K et al., 2020; Conneau et al., 2020b; Pires et al., 2019; Wu & Dredze, 2019).”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, background): “…on massive amounts of multilingual data, have recently surpassed (static) cross-lingual word embedding spaces (Ruder et al., 2019; Glavaš et al., 2019) as the state-of-the-art paradigm for cross-lingual transfer in NLP (Pires et al., 2019; Wu and Dredze, 2019; Wu et al., 2020; Hu et al., 2020).”
Probing Pretrained Language Models for Lexical Semantics (2020, background): “…for FI and TR. Encompassing 100+ training languages with limited model capacity, multilingual models suffer from the “curse of multilinguality” (Conneau et al., 2020): they must trade off monolingual lexical information coverage (and consequently monolingual performance) for a wider language…”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, background): “can only implicitly learn which words and structures correspond to each-other across languages in an entirely unsupervised fashion, but are nonetheless quite effective empirically (Wu et al., 2020; K et al., 2020).”
Emergent Structures and Training Dynamics in Large Language Models (2022, methodology): “Interestingly, Conneau et al. (2020b) and K et al. (2020) showed that the universal rep-361 The size of each language's corpus in the lan-377 guage model's training set has been shown to be decisive for transfer to that language.”
Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “Examples include mBERT (Devlin et al., 2018), mBART (Liu et al., 2020), XLM-R (Conneau et al., 2020a), mT5 (Xue et al., 2021), XGLM (Lin et al., 2022), mGPT (Shli-azhko et al., 2022), BLOOM (Scao et al., 2022), and PolyLM (Wei et al., 2023).”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, unknown): “These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18], representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38].”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “Follow-up work study the cross-lingual ability of these models and what contributes to it (Pires et al., 2019; Wu & Dredze, 2019; Wu et al., 2019; Artetxe et al., 2019; Kudugunta et al., 2019; Karthikeyan et al., 2020), the limitation of such training paradigm (Arivazhagan et al., 2019; Wang et…”