Crosslingual Generalization through Multitask Finetuning

By Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf and 9 more. 2023

602 citations recorded by the source corpus.

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task genrealization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/ bigscience-workshop/xmtf.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Additional analysis is provided in Muennighoff et al. (2022b).”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A series of works have found evidence that vanilla language models can be effective at following general language instructions if tuned with annotated “instructional” data – datasets containing language instructional commands and their desired outcome based on human judgement (Weller et al., 2020; M”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, methodology): “Bloomz-mt (Muennighoff et al., 2022) (Alibaba, 2023), and Xunfei Xinghuo (iFLYTEK, 2023), these models may have strong performance, yet we are not authorized to evaluate and publicize their results.”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, background): “Instruction tuning [29, 38, 9, 23, 37], also called supervised finetuning, is an approach used to enhance the performance of language models by training them to follow natural language instructions.”
MTEB: Massive Text Embedding Benchmark (2022, background): “It also performs competitively on languages like Russian or Japanese that unintentionally leaked into its pre-training data (Muennighoff et al., 2022).”
Qwen2 Technical Report (2024, neutral): “…Italian, Japanese, and Korean); (b) Understanding: BELEBELE (Bandarkar et al., 2023) (5-shot), XCOPA (Ponti et al., 2020) (5-shot), XWinograd (Muennighoff et al., 2023) (5-shot), XStoryCloze (Lin et al., 2022b) (0-shot) and PAWS-X (Yang et al., 2019) (5-shot); (c) Table 2: Performance of the…”
Scaling Data-Constrained Language Models (2023, neutral): “[65] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al.”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, positive): “Some work shows that multitask prompt-based training on a variety of tasks and English or translated prompts improves zero-shot cross-lingual and cross-task performance (Muennighoff et al., 2022; Fu et al., 2022).”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, background): “Thus, instead of learning meta-tasks in an implicit way from raw texts, recent approaches (Wei et al., 2021; Sanh et al., 2021; Muennighoff et al., 2022; Chung et al., 2022; Ouyang et al., 2022) proposed learning tasks in an explicit way with a large scale prompted (supervised) meta-pretraining…”
Aligning Large Multimodal Models with Factually Augmented RLHF (2023, background): “For the sake of simplicity, we did not adjust the mixture rate, earmark-ing that consideration for future research.”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, methodology): “The classification task re-uses the logistic regression classifier from MTEB (Muennighoff et al., 2022a), where the provided label is predicted based on the input embedding.”