UDapter: Language Adaptation for Truly Universal Dependency Parsing

By Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord. 2020

127 citations recorded by the source corpus.

Recent advances in the field of multilingual dependency parsing have brought the idea of a truly universal parser closer to reality. However, cross-language interference and restrained model capacity remain a major obstacle to this pursuit. To address these issues, we propose a novel multilingual task adaptation approach based on recent work in parameter-efficient transfer learning, which allows for an easy but effective integration of existing linguistic typology features into the parsing network. The resulting parser, UDapter, consistently outperforms strong monolingual and multilingual baselines on both high-resource and low-resource (zero-shot) languages, setting a new state of the art in multilingual UD parsing. Our in-depth analyses show that soft parameter sharing via typological features is key to this success.

Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Computational Linguistics & Semantics

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “[37] proposed a multilingual dependency parsing method based on adapters and contextual parameter generator networks [38], where they generate adapter parameters conditioned on trained input language embeddings.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “Adapters have been shown to work well for machine translation (Bapna and Firat, 2019), cross-lingual transfer (Pfeiffer et al., 2020b, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell et al., 2021), community QA (Rücklé et al., 2020), and task composition for transfer learning (Stickland and…”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “, 2020), and cross-lingual transfer (Pfeiffer et al., 2020c,d; Üstün et al., 2020; Vidoni et al., 2020).”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “In contemporaneous work, ¨Ust¨un et al. (2020) proposed a multilingual dependency parsing method based on adapters and contextual parameter generator networks (Platanios et al., 2018) where they generate adapter parameters conditioned on trained input language embeddings.”
Learning to Generate Task-Specific Adapters from Task Description (2021, methodology): “This concept has been broadly applied to visual reasoning (Perez et al., 2018), zero-shot image classification (Jin et al., 2020), etc. Closely related to our work, UDapter (Üstün et al., 2020) studies multilingual dependency parsing by generating adapter parameters.”
Prompt-free and Efficient Few-shot Learning with Language Models (2022, unknown): “Adapter Layers: Mahabadi et al. (2021b) and Üstün et al. (2020) proposed to generate adapters’ weights using hypernetworks (Ha et al., 2017), where Mahabadi et al. (2021b) proposed to share a small hypernetwork to generate conditional adapter weights efficiently for each transformer layer and task.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…et al.,116 2020a; Mahabadi et al., 2021b; Poth et al., 2021),117 or stacked for combinatory cross-lingual (Pfeiffer118 et al., 2020b, 2021b; Üstün et al., 2020; Vidoni119 et al., 2020; Ansell et al., 2021b,a; Wang et al.,120 2021b) as well as NMT scenarios (Bapna and Fi-121 rat, 2019;…”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, background): “2 Besides being more efficient, adapters are also highly modular, enabling a wider range of transfer learning techniques (Pfeiffer et al., 2020b, 2021a,b; Üstün et al., 2020; Vidoni et al., 2020; Rust et al., 2021; Ansell et al., 2021).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, background): “Another way to provide more language-specific capacity to a multilingual LM beyond a dedicated tokenizer, thereby potentially making gains in monolingual downstream performance, is to introduce adapters (Pfeiffer et al., 2020b,c; Üstün et al., 2020), a small number of additional parameters at every ”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, methodology): “Unlike prior CPG work (Platanios et al., 2018a; Üstün et al., 2020), MAD-G generates language adapters that are taskagnostic, thus allowing for an efficient and modular cross-lingual transfer across the board, i.”
Zero-shot Learning by Generating Task-specific Adapters (2021, background): “Closely related to our work, UDapter ( ¨Ust¨un et al., 2020) studies multilingual dependency parsing by generating adapter parameters using CPG.”