UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

By Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, Madian Khabsa. 2021

235 citations recorded by the source corpus.

Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, different PELT methods may perform rather differently on the same task, making it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks. In light of model diversity and the difficulty of model selection, we propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. On the GLUE benchmark, UniPELT consistently achieves 1 4% gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups. Moreover, UniPELT generally surpasses the upper bound that takes the best performance of all its submodules used individually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, background): “[13, 34] combine such approaches to propose a uniﬁed parameter-efﬁcient training framework.”
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, background): “There are two categories of solutions: ﬁrst, model compression (Jafari et al., 2021; Chen et al., 2021); second, parameter-efﬁcient tuning (PET) (Houlsby et al., 2019a; Karimi Mahabadi et al., 2021; Mao et al., 2021).”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, background): “Parameter-efﬁcient transfer learning (PETL) [31, 23, 39, 40, 34, 51, 58, 17, 24, 18, 42, 60, 14, 52, 59] is a recent research direction for online or multi-task learning.”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “UniPELT [96] proposed a hybrid framework that combines Adapter [155], Preﬁx-tuning [157], and LoRA [154] as experts.”
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, methodology): “We first show that SFT LM (either encoder-or decoder-based) always tends to acquire excessively redundant delta parameters.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “For instance, UniPELT [133] integrates LoRA, prefix-tuning, and adapters into each Trans-former block.”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, unknown): “Recent work has also shown that combining different types of parameter-efficient finetuning methods, including adapters, can lead to better performance (Mao et al., 2022; He et al., 2022).”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “Afterwards, similar to previous works (Ding et al., 2022; He et al., 2021a; Mao et al., 2021), we categorize these models into three categories based on how the parameters are chosen.”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “UniPELT [59] incorporates sequential adapter, prefix-tuning, and LoRA via a gating mechanism.”