Parameter-Efficient Transfer Learning for NLP
By Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. 2019
6,877 citations recorded by the source corpus.
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Quoc V. Le
- Yejin Choi
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Noah A. Smith
- Percy Liang
- Kyunghyun Cho
In Efficiency & Compression
- Luke S. Zettlemoyer
- Mohit Bansal
- Quoc V. Le
- Song Han
- Yi-Lin Sung
- Barret Zoph
- Kurt Keutzer
- Jaemin Cho
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “The first, “adapter layers” [Houlsby et al., 2019; Bapna et al., 2019], is motivated by the goal of keeping most of the original model fixed while fine-tuning.”
- Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “Houlsby et al. (2019) utilized a new formulation of adapters to adapt BERT (Devlin et al., 2018) to multiple tasks simultaneously.”
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…and Vocabulary Extending existing approaches for vocabulary construction and neural modeling to work better in multitask settings (Ma et al., 2018a; Houlsby et al., 2019) in order to strike the right balance between shared and task-specific capacity, or learning network structure in order to…”
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “efficient fine-tuning (Stickland and Murray, 2019; Houlsby et al., 2019) and better target task adaptation via multitask fine-tuning (Phang et al.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “of a small number of additional task-specific parameters instead (Lester et al., 2021; Li and Liang, 2021; Houlsby et al., 2019).”
- Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “For example, several researchers might concurrently fine-tune (Houlsby et al., 2019; Zhang et al., 2021) a foundation model for different tasks, using the same accelerators to hold the fixed foundation model layers.”
- Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, methodology): “For (b), we study lightweight finetuning alternatives that update a smaller set of parameters: BitFit (Ben-Zaken et al., 2021), Adapters (Houlsby et al., 2019), and calibration layers (Zhao et al., 2021 Figure 1 : Different Methods of Few-Shot Learning.”
- The Power of Scale for Parameter-Efficient Prompt Tuning (2021, methodology): “Adapters offer another means of reducing task-specific parameters, with Houlsby et al. (2019) achieving GLUE performance close to full model tuning when freezing BERT-Large and only adding 2 – 4 % additional parameters.”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “88 The rationale for skipping the first λ0 layers is that relevance classification effectiveness is too poor for the model to be useful; this is observation is consistent with findings across many NLP tasks [Houlsby et al., 2019, Lee et al., 2019a, Xin et al., 2020]. The [CLS] vector representation ”
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “For example, adapter-tuning (Rebuffi et al., 2017; Houlsby et al., 2019) inserts additional taskspecific layers between the layers of pretrained language models.”
- LoRA: Low-Rank Adaptation of Large Language Models (2021, methodology): “Adapter tuning [15] inserts adapter layers between the self-attention module (and the MLP module) and the subsequent residual connection.”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “…(PEFT) methods have been proposed such as prompt tuning [49, 33, 34], tuning the embedding layer inputs [1], tuning hidden states (IA 3 ) [37], adding full layers [27], tuning biases [71], learning a mask over weights based on Fisher information [55], and a combination of approaches [23].”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In contrast, gradient-based few-shot learning approaches [43] have constant complexity with respect to the number of shots during inference.”
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “We start by introducing the required background on the Kronecker product and adapter layers [1, 15].”
- WARP: Word-level Adversarial ReProgramming (2021, methodology): “In order to directly compare WARP with existing methods, we report in Table 2 different methods that use RoBERTa, including fine-tuning, linear classifiers on top, AutoPrompt, and Adapters.”
- AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “We compare the training and inference speed of full model fine-tuning against the adapter architectures of Houlsby et al. (2019) and Pfeiffer et al. (2021a) (depicted in Figure 1) using the AdapterHub.ml framework (Pfeiffer et al., 2020a).”