From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
By Anne Lauscher, Vinit Ravishankar, Ivan Vulic, Goran Glavas. 2020
363 citations recorded by the source corpus.
Massively multilingual transformers (MMTs) pretrained via language modeling (e.g., mBERT, XLM-R) have become a default paradigm for zero-shot language transfer in NLP, offering unmatched transfer performance. Current evaluations, however, verify their efficacy in transfers (a) to languages with sufficiently large pretraining corpora, and (b) between close languages. In this work, we analyze the limitations of downstream language transfer with MMTs, showing that, much like cross-lingual word embeddings, they are substantially less effective in resource-lean scenarios and for distant languages. Our experiments, encompassing three lower-level tasks (POS tagging, dependency parsing, NER) and two high-level tasks (NLI, QA), empirically correlate transfer performance with linguistic proximity between source and target languages, but also with the size of target language corpora used in MMT pretraining. Most importantly, we demonstrate that the inexpensive few-shot transfer (i.e., additional fine-tuning on a few target-language instances) is surprisingly effective across the board, warranting more research efforts reaching beyond the limiting zero-shot conditions.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Graham Neubig
- Jianfeng Gao
- Noah A. Smith
- Yejin Choi
- Sebastian Ruder
In Computer Vision and Pattern Recognition
- Ivan Vulic
- Luc Van Gool
- Sebastian Ruder
- Luke S. Zettlemoyer
- Dacheng Tao
- Trevor Darrell
- Mohit Bansal
- Xiaogang Wang
In Natural Language Processing
- Ivan Vulic
- Sebastian Ruder
- Iryna Gurevych
- Mohit Bansal
- Luke S. Zettlemoyer
- Jonas Pfeiffer
- Graham Neubig
- Sebastian Riedel
In Machine Translation
- Ivan Vulic
- Sebastian Ruder
- Jonas Pfeiffer
- Iryna Gurevych
- Goran Glavas
- Anna Korhonen
- Siva Reddy
- Heike Adel
Citations in the corpus
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “, 2019), language modeling on unseen languages (Lauscher et al., 2020), as well as various NLP applications (Liu et al.”
- UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, background): “The largest drops are reported for resourcepoor target languages (Lauscher et al., 2020b), and”
- MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “(11)We quantify the linguistic proximity of languages as the cosine similarity between their respective URIEL-based language vectors (Lauscher et al., 2020).”
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “Each task is associated with an example of input and output (English translations at the bottom). can help reduce the gap between zero-shot and supervised performance (Lauscher et al., 2020; Zhao et al., 2021).”
- mGPT: Few-Shot Learners Go Multilingual (2022, background): “However, there are several less explored directions in the interconnected areas of zero/few-shot learning and multilingual NLP, including model development for typologically distant and low-resource languages underrepresented in LMs’ pre-training corpora (Wu and Dredze, 2020; Lauscher et al., 2020; ”
- MuRIL: Multilingual Representations for Indian Languages (2021, background): “Multilingual LMs are substantially less effective in resource-lean scenarios (Wu and Dredze, 2020; Lauscher et al., 2020), as limited data doesn’t help capture the various nuances of a language.”
- Language Models are Few-shot Multilingual Learners (2021, background): “Another line of work is to apply cross-lingual transfer on English with the same task as the target languages (Ponti et al., 2018; Artetxe and Schwenk, 2019; Liu et al., 2019b; Lauscher et al., 2020; Liu et al., 2020b, 2021c; Chen et al., 2021).”
- A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “less suited for transfer learning, as Lauscher et al. (2020) showed.”
- MEGA: Multilingual Evaluation of Generative AI (2023, background): “We also study the effect that the amount of data available for each language during pre-training (Wu and Dredze, 2020; Lauscher et al., 2020) has on the multilingual performance of these models.”
- XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages (2020, methodology): “…gain in the low-data GAO-GAO setup (with only 919 training instances available): this confirms recent findings in other language understanding tasks (Lauscher et al., 2020; Brown et al., 2020) that few-shot fine-tuning works much better with pretrained language models which were exposed to more…”
- Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “Next to the curse of multilinguality, recent works have shown substantially reduced cross-lingual and monolingual abilities of models for low-resource languages with smaller pre-training data (Wu and Dredze, 2020; Hu et al., 2020; Lauscher et al., 2020b; Artetxe et al., 2020; Pfeiffer et al., 2020b,”
- How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “Another observation concerns substantially reduced cross-lingual and monolingual abilities of the models for resource-poor languages with smaller pretraining data (Wu and Dredze, 2020; Lauscher et al., 2020b).”
- Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer (2020, background): “Languages with smallest pretraining corpora are most affected: the largest transfer performance drops are observed with those target languages (Lauscher et al., 2020; Wu and Dredze, 2020).”
- A Primer on Pretrained Multilingual Language Models (2021, neutral): “Lauscher et al. (2020) show that for higher level tasks such as XNLI and XQuAD, the performance of zero-shot transfer has a strong correlation with the amount of data in the target language used for pretraining the MLLM.”