MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer

By Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavas, Ivan Vulic, Anna Korhonen. 2021

99 citations recorded by the source corpus.

Adapter modules have emerged as a general parameter-efﬁcient means to specialize a pre-trained encoder to new domains. Massively multilingual transformers (MMTs) have particularly beneﬁted from additional training of language-speciﬁc adapters. However, this approach is not viable for the vast majority of languages, due to limitations in their corpus size or compute budgets. In this work, we propose MAD-G ( M ultilingual AD apter G eneration), which contextually generates language adapters from language representations based on typological features. In contrast to prior work, our time-and space-efﬁcient MAD-G approach enables (1) sharing of linguistic knowledge across languages and (2) zero-shot inference by generating language adapters for unseen languages. We thoroughly evaluate MAD-G in zero-shot cross-lingual transfer on part-of-speech tagging, dependency parsing, and named entity recognition. While offering (1) improved ﬁne-tuning efﬁciency (by a factor of around 50 in our experiments), (2) a smaller parameter budget, and (3) increased language coverage, MAD-G remains competitive with more expensive methods for language-speciﬁc adapter training across the board. Moreover, it offers substantial beneﬁts for low-resource languages, particularly on the NER task in low-resource African languages. Finally, we demonstrate that MAD-G’s transfer performance can be further improved via: (i) multi-source training , i.e., by generating and combining adapters of multiple languages with available task-speciﬁc training data; and (ii) by further ﬁne-tuning generated MAD-G adapters for languages with monolingual data.

Subfields: Artificial Intelligence, Efficiency & Compression, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Machine Translation

Citations in the corpus

AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “…for machine translation (Bapna and Firat, 2019), cross-lingual transfer (Pfeiffer et al., 2020b, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell et al., 2021), community QA (Rücklé et al., 2020), and task composition for transfer learning (Stickland and Murray, 2019; Pfeiffer et al.,…”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, background): “2 Besides being more efficient, adapters are also highly modular, enabling a wider range of transfer learning techniques (Pfeiffer et al., 2020b, 2021a,b; Üstün et al., 2020; Vidoni et al., 2020; Rust et al., 2021; Ansell et al., 2021).”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “MAD-G [108] generates language adapters from language representations based on typological features, allowing the sharing of linguistic knowledge across languages for cross-lingual transfer.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…(2021)157 propose a transliteration based approach Vernikos 158 and Popescu-Belis (2021) propose subword map- 159 pings and Pfeiffer et al. (2020b, 2021b); Vidoni 160 et al. (2020); Ansell et al. (2021b) propose adapter- 161 based approaches to extend multilingual models to 162 unseen languages.”