Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning

By Zhaojiang Lin, Andrea Madotto, Pascale Fung. 2020

196 citations recorded by the source corpus.

Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pretrained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “Adapter-tuning has promising performance on natural language understanding and generation benchmarks, attaining comparable performance with fine-tuning while adding only around 2–4% task-specific parameters (Houlsby et al., 2019; Lin et al., 2020).”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “At the other end of the spectrum, adapter methods [1, 15] that insert trainable transformations at different layers of the pretrained model require more parameters than the aforementioned approaches but are more memory-efficient and obtain performance comparable to full fine-tuning [1, 16].”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, neutral): “…and Liang, 2021), Bit-ﬁt (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), LoRA (Hu et al., 2021), UniﬁedSKG (Xie et al., 2022) and Adapter-based models (Rebufﬁ et al., 2017; Houlsby et al., 2019; Lin et al., 2020; He et al., 2021; Pfeiffer et al., 2021).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Various PEFT techniques have been explored, including prompt tuning [35, 30, 39, 38, 50, 72], Low-Rank Adaptation (LoRA) [23, 69, 20], and adapters [22, 48, 37, 9, 55].”
Plug-and-Play Conversational Models (2020, methodology): “Finally, residual adapters (Houlsby et al., 2019) has been used to learn multiple language generation tasks (Lin et al., 2020) without ﬁne-tuning the original models’ parameters.”
MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems (2020, background): “Generative pre-trained unidirectional LMs (e.g., GPT2) are effective in language generation tasks (Radford et al., 2019; Hosseini-Asl et al., 2020; Peng et al., 2020; Lin et al., 2020).”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “Residual Adapter [14] further improves efficiency by inserting the adapter module only after the feed-forward and layer normalization.”
Personalize Segment Anything Model with One Shot (2023, background): “Adapters (Houlsby et al., 2019; Pfeiffer et al., 2020; Lin et al., 2020; Chen et al., 2022) are designed to be inserted between layers of the original transformer, introducing lightweight MLPs for feature transformation.”