From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers

By Anne Lauscher, Vinit Ravishankar, Ivan Vulic, Goran Glavas. 2020

363 citations recorded by the source corpus.

Massively multilingual transformers (MMTs) pretrained via language modeling (e.g., mBERT, XLM-R) have become a default paradigm for zero-shot language transfer in NLP, offering unmatched transfer performance. Current evaluations, however, verify their efficacy in transfers (a) to languages with sufficiently large pretraining corpora, and (b) between close languages. In this work, we analyze the limitations of downstream language transfer with MMTs, showing that, much like cross-lingual word embeddings, they are substantially less effective in resource-lean scenarios and for distant languages. Our experiments, encompassing three lower-level tasks (POS tagging, dependency parsing, NER) and two high-level tasks (NLI, QA), empirically correlate transfer performance with linguistic proximity between source and target languages, but also with the size of target language corpora used in MMT pretraining. Most importantly, we demonstrate that the inexpensive few-shot transfer (i.e., additional fine-tuning on a few target-language instances) is surprisingly effective across the board, warranting more research efforts reaching beyond the limiting zero-shot conditions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Luke S. Zettlemoyer
  2. Mohit Bansal
  3. Percy Liang
  4. Graham Neubig
  5. Jianfeng Gao
  6. Noah A. Smith
  7. Yejin Choi
  8. Sebastian Ruder

In Computer Vision and Pattern Recognition

  1. Ivan Vulic
  2. Luc Van Gool
  3. Sebastian Ruder
  4. Luke S. Zettlemoyer
  5. Dacheng Tao
  6. Trevor Darrell
  7. Mohit Bansal
  8. Xiaogang Wang

In Natural Language Processing

  1. Ivan Vulic
  2. Sebastian Ruder
  3. Iryna Gurevych
  4. Mohit Bansal
  5. Luke S. Zettlemoyer
  6. Jonas Pfeiffer
  7. Graham Neubig
  8. Sebastian Riedel

In Machine Translation

  1. Ivan Vulic
  2. Sebastian Ruder
  3. Jonas Pfeiffer
  4. Iryna Gurevych
  5. Goran Glavas
  6. Anna Korhonen
  7. Siva Reddy
  8. Heike Adel

Citations in the corpus