mGPT: Few-Shot Learners Go Multilingual

By Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, Tatiana Shavrina. 2022

209 citations recorded by the source corpus.

This paper introduces mGPT, a multilingual variant of GPT-3, pretrained on 61 languages from 25 linguistically diverse language families using Wikipedia and the C4 Corpus. We detail the design and pretraining procedure. The models undergo an intrinsic and extrinsic evaluation: language modeling in all languages, downstream evaluation on cross-lingual NLU datasets and benchmarks in 33 languages, and world knowledge probing in 23 languages. The in-context learning abilities are on par with the contemporaneous language models while covering a larger number of languages, including underrepresented and low-resource languages of the Commonwealth of Independent States and the indigenous peoples in Russia. The source code and the language models are publicly available under the MIT license.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “in settings where they support the language of the evaluation dataset): • mGPT (Shliazhko et al., 2022), GPT-style models trained on 60 languages from Wikipedia and Common Crawl”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, unknown): “XGLM XGLM models (Lin et al., 2021) cover 30 natural languages and come in five different numbers of parameters: 564M, 1.7B, 2.9B, 4.9B, and 7.5B. mGPT mGPT (Shliazhko et al., 2022) is a GPT model trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus.”
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, unknown): “In summary, while instruction tuning has greatly advanced the capabilities of LLMs in following complex instructions, challenges remain in dataset diversity, validity, and the integration of multilingual content.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “Here, we are interested in pre-training efforts which are massively multilingual (Xue et al., 2020; Chung et al., 2023; Shliazhko et al., 2022; Scao et al., 2022; Lin et al., 2022; Devlin et al., 2019; Conneau et al., 2019; Khanuja et al., 2021; Oladipo et al., 2023; Alabi et al., 2022).”
In-context Examples Selection for Machine Translation (2022, methodology): “Generating translations from large-scale multilingual language models like mGPT (Shliazhko et al., 2022), XGLM (Lin et al., 2021) or AlexaTM 20B (Soltan et al., 2022) requires conditioning the decoder-only language model with in-context parallel examples.”