Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

By Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson. 2021

382 citations recorded by the source corpus.

State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “[30] use a single compact hypernetwork allowing to generate adapter weights efficiently conditioned on multiple tasks and layers of a transformer model.”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “Concurrent work by Mahabadi et al. [33] improves the efﬁciency of Adapters by generating the weights of task-speciﬁc adapters via a hypernetwork.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “For this, we analyze these parameter-efﬁcient training techniques in a uniﬁed multi-task learning setup, and we benchmark different adapter [16, 20, 33] and prompt-based methods [24].”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, background): “Within unimodal multi-task and continual learning, Hypernetworks [Mahabadi et al., 2021] and compositional Adapter modules [Zhang et al., 2022] have been shown to facilitate knowledge transfer by generating Adapter parameters from shared task information.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “Other work explores updating only small parts of the model (Zaken et al., 2021) or task-specific modules, such as adapters (Houlsby et al., 2019; Karimi Mahabadi et al., 2021) or low-rank structures (Mahabadi et al., 2021; Hu et al., 2021), while keeping the rest of the model fixed.”
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, background): “Many parameter-efficient adaption methods exist (Rebuffi et al., 2017; Houlsby et al., 2019; Karimi Mahabadi et al., 2021; Zaken et al., 2021; Hu et al., 2022) and some have shown strong performance under domain shift (Lester et al., 2021; Li and Liang, 2021).”
Large Language Models: A Survey (2024, neutral): “Fine-tuning does not need to be performed to a single task though, and there are different approaches to multi-task fine-tuning (see e.g. Mahabi et al. [132]).”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
Fine-tuning Image Transformers using Learnable Memory (2022, neutral): “Such approaches were explored in [17, 29, 31].”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Compared to STN, AdaIN, FiLM and so on, our method is input-independent and these scale and shift parameters model the distribution of the whole dataset so that they can be absorbed into the original pre-trained model weights in the inference phase.”
HyperPrompt: Prompt-based Task-Conditioning of Transformers (2022, methodology): “To this end, this paradigm is strongly reminiscent of adapter layers (Houlsby et al., 2019a; Karimi Mahabadi et al., 2021; Zaken et al., 2021; He et al., 2021) which are also eﬃciently ﬁnetuned.”
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “Recent studies (He et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al., 2021) find that some PELT methods could be more effective than fine-tuning when the training data is limited, possibly due to the reduced risk of overfitting.”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, background): “LoRA (Hu et al., 2022; Karimi Mahabadi et al., 2021; Panahi et al., 2021) proposes to add a new vector calculated by recovering an hidden vector from a lower dimension space.”
DoRA: Weight-Decomposed Low-Rank Adaptation (2024, methodology): “The first category is referred to as Adapter-based methods, which involve introducing additional trainable modules into the original frozen backbone, such as (Houlsby et al., 2019; He et al., 2021; Karimi Mahabadi et al., 2021; mahabadi et al., 2021).”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “Hyperformer++ [21] utilizes the shared hypernetwork [80] to learn task-specific and layer-specific adapter parameters that condition on task and layer id embeddings.”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, background): “Parameter-efﬁcient transfer learning (PETL) [31, 23, 39, 40, 34, 51, 58, 17, 24, 18, 42, 60, 14, 52, 59] is a recent research direction for online or multi-task learning.”
Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, background): “For example, multi-task learning 29,30 is an advantageous setting for adapter-based methods, inserted with adapter modules in parallel with the self-attention module, PLMs could demonstrate impressive representational capacity in the multi-task setting.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…newly learned knowledge can114 be combined (Stickland and Murray, 2019; Wang115 et al., 2021a; Pfeiffer et al., 2021a; Lauscher et al.,116 2020a; Mahabadi et al., 2021b; Poth et al., 2021),117 or stacked for combinatory cross-lingual (Pfeiffer118 et al., 2020b, 2021b; Üstün et al., 2020;…”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, background): “Can more conceptually-diverse tasks from different languages communicate together? To answer this question, researchers have made preliminary efforts along this direction (Yang et al., 2016; Lin et al., 2018; Hasan et al., 2021; Dong et al., 2015; Mahabadi et al., 2021).”