On the Cross-lingual Transferability of Monolingual Representations

By Mikel Artetxe, Sebastian Ruder, Dani Yogatama. 2019

968 citations recorded by the source corpus.

State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “Similar phenomena have also been reported in other multilingual pre-training approachesin other NLP applications (Pires et al., 2019; Wang et al., 2019; Artetxe et al., 2019).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “In this section we test this hypothesis on three wordlevel benchmarks: (i) transliteration, (ii) grapheme- Small Base Large XL XXL mT5 ByT5 mT5 ByT5 mT5 ByT5 mT5 ByT5 mT5 ByT5 In-language multitask (models fine-tuned on gold data in all target languages) WikiAnn NER 86.4 90.6 88.2 91.6 89.7 91.8 91.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “, 2018) entailment task covering 14 languages; the XQuAD (Artetxe et al., 2020), MLQA (Lewis et al.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “XQuAD (Artetxe et al., 2019) includes 1,190 QA instances from SQuAD 1.”
WARP: Word-level Adversarial ReProgramming (2021, methodology): “Another method to reprogram a BERT-based model is proposed by Artetxe et al. (2020), where a model tuned on an English version of a particular task is transformed to work in another language by changing only the embedding matrices.”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “Some of these difﬁculties may be due to to-kenisation and an under-representation of ideograms in the joint sentencepiece vocabulary, which has been shown to be important in a cross-lingual model’s performance (Artetxe et al., 2019; Conneau et al., 2019).”
Pretrained Transformers as Universal Computation Engines (2021, background): “…investigation of transfer with language pretraining, notably showing transfer from English to Python, which they consider to be reasonably distanced from English; many works have also looked at transferring from one langauge to another (Artetxe, Ruder, and Yogatama 2019; Ponti et al. 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “Artetxe et al. (2019) also show cross-lingual transfer is possible by swapping the model Figure 7: Language centroids of the mean-pooled mBERT representations (Libovick´y et al., 2019) vocabulary, without any shared word-pieces.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “Our framework goes beyond prior work on using adapters for cross-lingual transfer (Bapna and Firat, 2019; Artetxe et al., 2020) by enabling adaptation to languages unseen during pretraining and without learning expensive language-speciﬁc token-level embeddings.”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Additionally, in the translate-train setting C HARFORMER SBase,LongPT is on par with subword models on XQuAD and MLQA, and close to parity on PAWS-X.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “, 2018), which is a textual entailment task covering 14 languages, XQuAD (Artetxe et al., 2020) and TyDiQA-GoldP (Clark et al.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “formance of multilingual systems on different natural language processing tasks, including question answering (Artetxe et al., 2020; Lewis et al., 2019; Clark et al., 2020), natural language understanding (Conneau et al.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, neutral): “5 (Zeman et al., 2019), NER named entity recognition on the Wikiann (Pan et al., 2017; Rahimi et al., 2019) dataset, cross-lingual natu-ral language inference on XNLI (Conneau et al., 2018), cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019a), and cross-lingual ques”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “ngual models, multilingual models have another dimension of transferability: cross-lingual transfer, the ability to transfer knowledge from one language to another. Previous work (Pires et al., 2019; Artetxe et al., 2020) has found that MLM on multilingual data encourages cross-lingual alignment of ”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “C L ] 7 M a y 2020 trained MLM model can be rapidly ﬁne-tuned to another language (Artetxe et al., 2019).”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “Datasets that cover multiple languages for other natural understanding tasks, such as language inference (Conneau et al., 2018), question answering (Lewis et al., 2019; Artetxe et al., 2020a; Clark et al., 2020), and paraphrase identiﬁcation (Yang et al., 2019b) have received increasing attention.”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, methodology): “This approach was proposed by Artetxe et al. (2020): they show that it allows learning aligned representations for a new language but only evaluate on high-resource languages.”