LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

By Yi-Lin Sung, Jaemin Cho, Mohit Bansal. Neural Information Processing Systems, 2022

329 citations recorded by the source corpus.

Fine-tuning large pre-trained models on downstream tasks has been adopted in a variety of domains recently. However, it is costly to update the entire parameter set of large pre-trained models. Although recently proposed parameter-efficient transfer learning (PETL) techniques allow updating a small subset of parameters (e.g. only using 2% of parameters) inside a pre-trained backbone network for a new task, they only reduce the training memory requirement by up to 30%. This is because the gradient computation for the trainable parameters still requires backpropagation through the large pre-trained backbone model. To address this, we propose Ladder Side-Tuning (LST), a new PETL technique that can reduce training memory requirements by more substantial amounts. Unlike existing parameter-efficient methods that insert additional parameters inside backbone networks, we train a ladder side network, a small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from backbone networks and makes predictions. LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections. We evaluate our method with various models (T5 and CLIP-T5) on both NLP (GLUE) and vision-and-language (VQA, GQA, NLVR2 , MSCOCO) tasks. LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2.7x more memory savings). Moreover, LST achieves higher accuracy than Adapter and LoRA in a low-memory regime. To further show the advantage of this better memory efficiency, we also apply LST to larger T5 models, attaining better GLUE performance than full fine-tuning and other PETL methods. The accuracy-efficiency trade-off also holds on VL tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/960d40497717ad22a7ebb84db238fa2415fc89cc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Jiaming Han
6. Pan Lu
7. Ee-Peng Lim
8. Lidong Bing

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Peng Gao
3. Yu Qiao
4. Renrui Zhang
5. Pan Lu
6. Xiaogang Wang
7. Jiaming Han
8. Luc Van Gool

Citations in the corpus

• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “For example, prevalent PEFT techniques such as adapters and LoRA can only reduce memory usage to approximately 70% compared to full model fine-tuning according to some literatures [167], [88].”
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “Many successful PEFT approaches [10, 17, 22, 28, 31, 59, 60, 71, 73] have been proposed for adopting popular pre-trained models such as BERT [11], GPT [4, 53, 54], ViT [13], CLIP [52], and Stable Diffusion [55] to various downstream tasks.”
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023, methodology): “Another approach, called Ladder Side-Tuning (Sung et al., 2022), involves training a lightweight ladder side network.”
• BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, background): “, 2022) and ladder side-tuning (Sung et al., 2022) can adapt BLOOM as well as the adapterbased strategies explored in our experimental setting.”
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “LST (Ladder Side-Tuning) [29] trains a ladder side network in conjunction with the pretrained network and takes intermediate activations as input via shortcut connections, known as ladders, from pretrained network.”
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, unknown): “Some other methods, e.g., black-box tuning [56], ladder side-tuning [57], sparse structure search[26], and Scaling&Shifting [38] Prompt Design.”