Lifting the Curse of Multilinguality by Pre-training Modular Transformers

By Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, Mikel Artetxe. 2022

181 citations recorded by the source corpus.

Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Thirdly, we find a minimal and inconsistent “curse of multilinguality” (Conneau et al., 2020; Pfeiffer et al., 2022) for BLOOM.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “The Impact of Sampling Weights The selection and balance of training data sources play a key role in determining the resulting model’s capabilities and quality (Longpre et al., 2023c; Wang et al., 2023b; Pfeiffer et al., 2022; Ogueji et al., 2022).”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “This is consistent with a number of recent results that suggest that masked-language model pre-training efficacy may saturate [1], especially as more diverse or distinct data sources are added [90], or may only give significant gains for highly in-domain data [126].”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, background): “In addition, our work focuses on studying the effects of language adaptation on prompting, which has been underexplored in previous language adaptation work (Ebrahimi and Kann, 2021; Ansell et al., 2022; Parović et al., 2022; Pfeiffer et al., 2022).”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “Similarly, Pfeiffer et al. (2022) develop a multilingual expert model with language-speciﬁc routing, and Kudugunta et al. (2021) develop a multi-task expert model with task-speciﬁc routing.”