Few-shot Learning with Multilingual Generative Language Models

By Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du and 11 more. 2021

384 citations recorded by the source corpus.

Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative language models on a corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, our model outperforms GPT-3 on 171 out of 182 directions with 32 training examples, while surpassing the official supervised baseline in 45 directions. We conduct an in-depth analysis of different multilingual prompting approaches, showing in particular that strong few-shot learning performance across languages can be achieved via cross-lingual transfer through both templates and demonstration examples.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “In the 1-shot setting, we test several language directions in the Flores-101 (Goyal et al., 2022) devtest set using the XGLM prompt (Lin et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “This can lead to problematic usages of LMs, whether malicious (disinformation) or unintentional (misinformation), with models producing misleading content, conspiracy theories, and falsehoods (Xu, 2020; Shao et al., 2018; McGuffie and Newhouse, 2020; Lin et al., 2021a).”
Qwen2 Technical Report (2024, neutral): “Multi-lingual datasets can be grouped into four categories: (a) Exam: M3Exam (5-shot, we only choose examples that require no image), IndoMMLU (Koto et al., 2023) (3-shot), ruMMLU (Fenogenova et al., 2024) (5-shot), and translated MMLU (Chen et al., 2023b) (5-shot on Arabic, Spanish, French, Portugu”
PyThaiNLP: Thai Natural Language Processing in Python (2023, positive): “It is also a collaborative work with AIResearch.in.th. WangChanGLM (Polpanumas et al., 2023) is a multilingual instruction-following model ﬁne-tuned from XGLM (Lin et al., 2022).”
TinyLlama: An Open-Source Small Language Model (2024, methodology): “1, we test our models with the following tasks: xwinograd (Emelin and Sennrich, 2021), xstorycloze (Lin et al., 2022), xnli (Conneau et al., 2018), and xcopa (Ponti et al., 2020).”
Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, background): “Examples include mBERT (Devlin et al., 2018), mBART (Liu et al., 2020), XLM-R (Conneau et al., 2020a), mT5 (Xue et al., 2021), XGLM (Lin et al., 2022), mGPT (Shli-azhko et al., 2022), BLOOM (Scao et al., 2022), and PolyLM (Wei et al., 2023).”
High-Dimension Human Value Representation in Large Language Models (2024, unknown): “…values in a high-dimensional space without the needs of any prior survey knowledge, allowing better scalability for extracting, understanding, and manipulating values of LLMs. Left : our objective is to learn a value embedding Z that represents the latent value-relevant factor ϑ value in LLM f θ…”