Language Models are Few-shot Multilingual Learners

By Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, Pascale Fung. 2021

173 citations recorded by the source corpus.

General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples. Here, we evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages without any parameter updates. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones. Finally, we find the in-context few-shot cross-lingual prediction results of language models are significantly better than random prediction, and they are competitive compared to the existing state-of-the-art cross-lingual models and translation models.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “Technical workarounds raise new challenges Various solutions are being explored to increase LM performance in different languages, such as translating a prompt to English, generating predictions in English, then translating these predictions back into the original language of the prompt (Caswell et ”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, methodology): “We illustrate the main results of NATIVECOT, EN-COT and TRANSLATE-EN with respect to the language frequency in PaLM training data (3)We focus on these two models due to their notable few-shot performance; in contrast, many other multilingual models are not as competitive as them in the same settings”
Taxonomy of Risks posed by Language Models (2022, background): “Current state of the art LMs are primarily trained in English or Mandarin Chinese [30, 37, 54, 148, 160] and perform better in these compared to any other languages [196].”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “, 2021), most notably in cross-lingual transfer (Winata et al., 2021) and”
Few-shot Learning with Multilingual Generative Language Models (2021, background): “Despite notable recent work at smaller scales (Zhao and Schütze, 2021) and for domain-specific tasks (Winata et al., 2021), the multilingual few-shot learning capabilities of language models are less well understood.”
Co-Writing with Opinionated Language Models Affects Users’ Views (2023, methodology): “It is known that language models work and perform better for the languages and contexts they are trained in (primarily English or Mandarin Chinese) [23, 91, 103].”
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study (2023, background): “, which shares the underlying LM with ChatGPT (Winata et al., 2021), 96% of the training corpus is in English.”
Few-shot Learning with Multilingual Language Models (2021, methodology): “Despite notable recent work at smaller scales (Zhao and Sch¨utze, 2021) and for domain-specific tasks (Winata et al., 2021), multilingual few-shot learning capabilities are less well understood.”