Towards a Unified View of Parameter-Efficient Transfer Learning
By Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig. 2021
1,223 citations recorded by the source corpus.
Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Graham Neubig
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Yu Qiao
- Yi Yang
- Jitendra Malik
In Alignment & RLHF
In Efficiency & Compression
Citations in the corpus
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…tuning (a.k.a. continuous or soft “prompt tuning”) shows that only updating a small subset of model parameters can reach comparable performance as fully tuning all model parameters (Lester et al., 2021; Vu et al., 2022; Hu et al., 2021; see He et al., 2022 for a detailed analysis).”
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, neutral): “Relatedly, prompt tuning [14] and prefix tuning [29] concatenate learned continuous embeddings to the model’s input or activations to induce it to perform a task; this can be seen as a PEFT method [30].”
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, background): “[13, 34] combine such approaches to propose a unified parameterefficient training framework.”
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “Following research demonstrating that LMs become strong zero-shot learners after being finetuned on collections of structured, task-based datasets [42,56], we also opt to include the training sets of various downstream tasks – adding the train sets of VQA [2], GQA [23], OKVQA [31], VizWiz [17], Hatef”
- Visual Prompt Tuning (2022, background): “On the other hand, given the dominance of large-scale pre-trained Transformerbased Language Models (LM) [14,62,5], many approaches [25,24,31] have been proposed to efficiently fine-tune LM for different downstream NLP tasks [72,71].”
- Continual Training of Language Models for Few-Shot Learning (2022, background): “, 2019), CL-plugin adopts the parallel adapter idea in (He et al., 2021).”
- Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, background): “Moreover, foregoing prompts as expressed in natural language makes it much harder to study their semantics, and it is not clear if continuous prompts serve as task-specific instructions or simply more efficient model parameters (see He et al., 2021 for a detailed analysis).”
- CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, positive): “Subsequent work such as Parallel Adapter [19] and VL-Adapter [58] further extend [24]’s ability to language and multimodal tasks.”
- PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, background): “(2) Evaluating the few-shot performance of other parameter-efficient tuning approaches (He et al., 2022) and adapting unified task pre-training to them.”
- Structured Pruning Adapters (2022, methodology): “However, prior work has largely ignored the computational efficiency aspects of Adapters, which either increase the complexity of the network (He et al., 2022; Zhu et al., 2021; Li & Liang, 2021; Houlsby et al., 2019; Pfeiffer et al., 2021; Mahabadi et al., 2021) or leave it unaltered, at best, by u”
- AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The Transformer architecture is first introduced in [81] and has re-energized the natural language processing (NLP) field from then on [27, 10].”
- Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, background): “In order to mitigate this issue, we explore a lightweight alternative [19] to training the full network where residual adapters with a small number of parameters are added for each individual language while the pre-trained USM is entirely frozen during fine-tuning.”
- RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, background): “Soft prompt tuning can be seen as a variant of parameter-efficient transfer learning (Houlsby et al., 2019; He et al., 2021; Ding et al., 2022), and inspires a number of follow-up works that boost its performance (e.g., Liu et al., 2021c; Gu et al., 2021; Vu et al., 2021; Clive et al., 2021) or…”
- Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, background): “See Liu et al. (2021), He et al. (2022), and Liu et al. (2022) for detailed discussion of the differences between these methods.”
- Personalize Segment Anything Model with One Shot (2023, methodology): “Prompt tuning (Lester et al., 2021; Zhou et al., 2022; Jia et al., 2022; Liu et al., 2021) suggests using learnable soft prompts alongside frozen models to perform specific downstream tasks, achieving more competitive performance with scale and robust domain transfer compared to full model tuning.”
- Black-Box Tuning for Language-Model-as-a-Service (2022, background): “This line of research, along with the work of parameter-efficient tuning (Houlsby et al., 2019; Liu et al., 2021a; Li and Liang, 2021; Lester et al., 2021; Sun et al., 2021a; Liu et al., 2021b; Hu et al., 2021a; He et al., 2021), demonstrate that”
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
- Pre-trained Vision and Language Transformers are Few-Shot Incremental Learners (2024, background): “However, since prefix tuning has a limitation of updating B-Prompt as mentioned in [4, 5] which is the slow adaptation speed of learnable prompts, B-Prompt cannot properly learn domain-specific knowledge and can be ignored by the fine-tuned layers.”
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “The first category is to introduce task-specific adapters [25, 24, 51, 50].”