MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer

By Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder. 2020

754 citations recorded by the source corpus.

The main goal behind state-of-the-art pretrained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pretraining. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new language. MAD-X outperforms the state of the art in cross-lingual transfer across a representative set of typologically diverse languages on named entity recognition and achieves competitive results on question answering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Power of Scale for Parameter-Efficient Prompt Tuning (2021, methodology): “Pfeiffer et al. (2020) use multiple adapters in a multilingual context to explicitly separate language understanding from task speciﬁcation, similar to our approach.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “We compare the training and inference speed of full model ﬁne-tuning against the adapter architectures of Houlsby et al. (2019) and Pfeiffer et al. (2021a) (depicted in Figure 1) using the AdapterHub.ml framework (Pfeiffer et al., 2020a).”
Efficient Transformers: A Survey (2020, methodology): “A modest number of models have been proposed, such as PALS (Stickland and Murray, 2019), MAD-X (Pfeiffer et al., 2020) and HyperGrid (Tay et al.”
Memory Efficient Continual Learning with Transformers (2022, background): “BERT [12], have shown their effectiveness in various natural language processing (NLP) tasks such as classification [25], Natural Language Inference [41, 39], and Question Answering [16].”
AdapterHub: A Framework for Adapting Transformers (2020, methodology): “…from many more sources than was previously possible and makes research such as intermediate task training (Pruksachatkun et al., 2020), composing information from many tasks (Pfeiffer et al., 2020a), and training models for very low-resource languages (Pfeiffer et al., 2020b) much more accessible.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, methodology): “In particular, Adapters (Rebuffi et al., 2018), which learn to read and write to layers of a shared model, have been applied to obtain parameter-efficient BERT models (Houlsby et al., 2019; Pfeiffer et al., 2020a,b,c).”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “, 2019) and (Pfeiffer et al., 2020) represent the stream of work which adds additional “adaptor modules” to achieve fast”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, methodology): “Previous studies have shown the importance considering domains when adapting LMs (Ramponi and Plank, 2020; Gururangan et al., 2020).”
Rethinking embedding coupling in pre-trained language models (2020, background): “In order to achieve reasonable performance for most languages, these models need to allocate sufﬁcient capacity for each language, known as the curse of multilinguality (Conneau et al., 2020a; Pfeiffer et al., 2020).”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, methodology): “In all strategies, we simply continue training the XLM-R model via the masked language modeling (MLM) objective on different combinations of data (Pfeiffer et al., 2020), in particular:”
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, background): “…signiﬁcant progress has been made in cross-lingual learning for NLP tasks (Kle-mentiev et al., 2012; Ammar et al., 2016; Lample and Conneau, 2019; Pfeiffer et al., 2020): for example, some approaches appear capable of extending machine translation models to new languages with only monolingual…”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, methodology): “For further technical details, we refer the reader to the original work (Pfeiffer et al., 2020c).”
Multi-Concept Customization of Text-to-Image Diffusion (2022, methodology): “Relatedly, several methods [19,50,64] propose to train adapter modules for large-scale models in the discriminative setting.”
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021, methodology): “There are also studies focusing on parameter-efficient fine-tuning, for example, the adapter-based methods (Houlsby et al., 2019; Pfeiffer et al., 2020; Karimi Mahabadi et al., 2021), and the Diff-Pruning method (Guo et al.”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, positive): “Our work is closely related to recent studies on modular LMs, which have specialized parameters (or experts ) trained on different domains (Gururangan et al., 2022; Li et al., 2022; Gururangan et al., 2023), languages (Pfeiffer et al., 2020; 2022), or tasks (Chen et al., 2022b; Jang et al., 2023a).”
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021, methodology): “This design inspires most of the following work (Pfeiffer et al., 2020a,c; Bapna and Firat, 2019).”
Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “…the use of adapters in NLP (Pfeif-fer et al., 2020b, 2021) and a framework built on top of HuggingFace Transformers library (Wolf et al., 2020) was also released to facilitate the down-loading, sharing, and adapting state-of-the-art pre-trained models with adapter modules (Pfeiffer et al., 2020a).”
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020, background): “Finally, Pfeiffer et al. (2020) propose adapter modules that learn language and task representation for cross-lingual transfer, which also allow adaptation to languages that are not observed in the pretraining data.”